2025 to now · AI engineer
Document-AI platform
Classifies and splits scanned German occupational pension paperwork. Multi-tenant, Azure, queue-driven.
A few hundred scanned pages arrive as a single PDF. The pipeline has to work out where each document ends, decide which of 54 types it is, and pull the fields out of it. Then it has to do that for several tenants who are not allowed to see each other’s data.
I own the AI half and most of the backend underneath it: the DSPy pipeline, the evaluation harness, the cost telemetry, the Azure Functions that run the whole thing, and the Postgres that pushes live updates to the front end through LISTEN/NOTIFY instead of a Redis nobody wanted to operate.
Most of the job is making a non-deterministic system safe enough to deploy on a Friday. That means an evaluation harness whose tests still hold when the model answers differently twice in a row.
- 54 document types, 438 tests running in CI.
- Cut inference cost 5.7× per document by designing around a cheaper model instead of buying a better one.
- Shipped the cheaper pipeline over a more accurate ensemble that cost 40% more and added 170 seconds of latency per document.
- Contained a DSPy major-version break and a compromised litellm release inside an hour each, before either reached a user.
Python · DSPy · GPT-4o vision · Azure Functions · FastAPI · PostgreSQL · Docker · GitHub Actions
The expensive part of a document-AI platform is rarely the classification. Ours was page stream segmentation: deciding where one document ends and the next begins inside a scanned batch of a few hundred pages. Get a boundary wrong and every page after it inherits the wrong type.
We were running GPT-4o vision on it. It worked. It also dominated the cost of every document we processed.
What failed first
A better model. We benchmarked GPT-5 against GPT-4o on our actual task. It was not better for us. Not worse either, just not worth a migration on this workload.
A better prompt. I ran DSPy's optimizers, BootstrapFewShot and MIPROv2 and GEPA, for days. All three scored 0% improvement over the prompt I had written by hand. That result was annoying and useful: the prompt was not the bottleneck, so I stopped spending time there.
More context. Intuition says a vision model deciding a boundary should see more of the surrounding pages. Accuracy dropped roughly 20% when I gave it more. The model was not short on information. It was drowning in it.
What worked
Stop trying to make an expensive model smarter. Take a model that costs about 30 times less and run it about 3 times harder.
The pipeline compensates for a weaker model instead of paying for a stronger one:
Narrow the window. One batch of context per call, not the full document history. Less input, better decisions, cheaper calls.
Sample the same boundary more than once. Vision models are not deterministic even at temperature 0.0. The same document, same model, same prompt gave us 61 boundaries on one run and 90 on the next. If the output varies, one sample is a coin flip and three samples are a measurement.
Never let a fallback invent an answer. GPT-4o silently dropped pages from structured-output batches roughly 13% of the time. Our original fallback guessed the missing page's type from its neighbour, and that single guess poisoned 94 pages of downstream classification. A missing page now raises.
Net result: 5.7 times cheaper per document, with accuracy holding.
The architecture we deliberately did not ship
We also built a dual-model ensemble. On paper it was the better system, worth about 5.7% more accuracy.
It also cost 40% more and added 170 seconds of latency per document. We kept the cheaper one. For this workload, a user waiting three extra minutes is a worse product than a system that is a few points less accurate on a step a human already reviews.
That ensemble's original 88 F1 was a measurement artifact. It came from combining two good runs offline. The version actually deployed scored below the single-model baseline, because a runtime optimization added to speed up local runs had quietly gutted the recall the ensemble depended on. A docstring had conflated the two runs, so the regression hid in plain sight for weeks.
Score the config you ship, in the same run. Your optimistic number probably came from a setup that is not in production.
What I would tell someone starting this
Check your ground truth before you blame the model. We spent two weeks chasing an accuracy problem that turned out to be wrong labels. The model was right and our eval was broken.
Check whether academia already solved it. Page stream segmentation has published architectures going back to 2018. I found that out later than I should have.
Cost has to be in the objective function. Obvious until you find it missing from your own harness. I have since seen a benchmark suite built explicitly to make an LLM app cheaper, which recorded cost in its metrics and gave it zero weight in the score.
The pipeline is usually the lever. Every expensive thing we tried was an attempt to buy accuracy. The thing that worked was designing around a cheaper model's weaknesses.