Offline metrics look clean. CTR is flat. Conversion is down. The problem isn't the model. It's what the model is ranking against. This pattern shows up across recommendation engine audits: The team ships a new model version. Offline retrieval score improves. A/B test shows neutral-to-positive CTR. Three months later, conversion is still flat. In 20 audits, 18 teams answered with a model name. Two answered with a pipeline diagram. The two that answered with a pipeline diagram had fixed the problem. Why offline metrics lie Offline retrieval metrics measure how well your model ranks items against a historical behavior sample. They cannot measure two things: Whether the behavioral signals feeding the model are fresh Whether those signals belong to the right user Both failures are silent. The model scores look correct. The production output is fiction.…