Most teams pick LLM providers based on demos and vibes. Here's the evaluation framework that separates good choices from expensive ones. When teams evaluate LLM providers, they almost always do it wrong. They run a prompt, compare the outputs, pick the one that sounds best, and move on. Three months later they're dealing with inconsistent behavior, unexpected cost spikes, or mysterious accuracy drops they can't explain. The problem isn't the evaluation — it's that they're measuring the wrong things. Output quality in a controlled test is not the same as output quality in production. What matters is what happens over time, at scale, under variance. Here's what to actually measure. The 5 Metrics That Matter Metric What It Tells You Target Range Accuracy Consistency Does the model perform the same on identical inputs over time? CV < 5% across daily runs Latency p95 What's your 95th percentile response time? < 2s for most tasks Cost per Eval What's your evaluation cost per test run?…