Every AI feature begins the same way: a prompt that works beautifully in a playground at 11pm. The gap between that moment and a feature real users rely on is where most AI products quietly die. This is a map of that gap, drawn from shipping LLM-backed features into production.
Design for the failure, not the demo
A model call can be slow, wrong, refused, or down — and your interface must be honest about all four. The features that survived had one thing in common: the AI was an enhancement to a path that worked without it. Autocomplete that falls back to templates. Summaries that fall back to excerpts. When the model call is the only path, every model failure is an outage.
Streaming is not a nicety; it is the difference between perceived responsiveness and abandonment. Time-to-first-token is your real latency metric. Users forgive a slow completion that starts instantly far more than a fast one that arrives all at once after four seconds of spinner.
Evaluation before iteration
You cannot improve what you evaluate by vibes. Before tuning anything, we froze a set of fifty real inputs with graded expected outputs — not perfect answers, but rubrics. Every prompt change ran against the set. Most 'obvious improvements' regressed something; the eval set caught what enthusiasm missed.
The uncomfortable discovery: prompt changes mattered less than context quality. Retrieval improvements moved our scores three times more than any rewording. If you have a week, spend four days on what the model sees and one on what you say to it.
What actually moved eval scores
Retrieval / context quality vs. prompt wording