Attention doesn't actually look at all words. That single insight breaks open the most misunderstood mechanism in modern AI. Every time GPT-4 finishes your sentence, Claude writes code, or Gemini generates an image caption, the same eight-step computation runs billions of times—and most developers have no idea what's happening inside it. This article walks through the exact math, the real implementation tricks, and the one optimization that made today's 200K-token context windows possible. Key Facts Most People Don't Know The original 2017 Transformer used 8 parallel attention heads in each layer, but GPT-3 uses 96 heads per layer with each head operating on only 128 dimensions instead of the full 12,288. Scaled dot-product attention divides by the square root of the key dimension (√dk) specifically because without it, dot products grow large in magnitude pushing softmax into regions with extremely small gradients below 0.0001.…