2026 · Personal project
Supportly
A retrieval-augmented email assistant that drafts replies from your own past threads.
Gmail in, retrieval over your own history, a draft out. DSPy runs the multi-stage pipeline on Llama 3 70B through Groq, Qdrant holds the vectors, FastAPI and Next.js hold everything else.
It is in the portfolio for how it failed. Three weeks of fluent, confident drafts built on embeddings that were random numbers will change how you write tests.
- Ran three weeks on random-number embeddings before anyone noticed, because the drafts still read as confident.
- That failure is the reason the project now asserts on retrieval quality instead of on response shape.
Python · DSPy · Qdrant · Groq · Llama 3 70B · Gmail API · FastAPI · Next.js
It ran for three weeks on embeddings that were random numbers.
How that happens
The embedding call was wrapped in a try/except that fell back to a zero-or-random vector when the provider errored, so the pipeline would "stay up". The provider was erroring. Nothing crashed, nothing alerted, and the queue drained normally.
Retrieval still returned five neighbours every time, because a nearest-neighbour search over noise still has a nearest neighbour. The drafts still read as confident and well-formed, because the language model was fine. Only the evidence it was given was garbage.
Every test I had asserted on the shape of the response. Not one asserted that the retrieved documents had anything to do with the query.
What I changed
A fallback that invents data is worse than a crash. The except branch now re-raises. If embeddings are down, the feature is down, and I find out in minutes instead of weeks.
Assert on retrieval, not on response shape. A fixed query with a known-correct document has to return that document in the top k. That test would have caught this on day one.
Sanity-check the vectors themselves. Random vectors have a distinctive signature: cosine similarities cluster near zero and the spread across neighbours is tiny. That is one cheap assertion at write time.
The uncomfortable part is how good the output looked. A language model given irrelevant context does not produce obvious nonsense. It produces a fluent, plausible, completely ungrounded answer, which is much harder to notice than an error page.