Book: LLM Observability Pocket Guide: Picking the Right Tracing & Evals Tools for Your Team Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You are running a 70B model on two H100s for an internal coding assistant. The product team wants the median response under two seconds. The model is sitting at 4.1s. You already swapped to FP8 quantization. You already turned on continuous batching. The GPU memory is comfortable. The bottleneck is the one thing quantization does not fix: every output token is one full forward pass through 70 billion parameters, in series, one after the next. A 200-token answer is 200 of those passes back to back. Speculative decoding is the trick that breaks the series. What it actually does The idea, from Leviathan et al., 2023 , is to split the work in two.…