When researchers scale a language model — more parameters, more layers, wider hidden dimensions — there's an implicit assumption: a bigger model can represent more things. More expressiveness, more knowledge, better predictions. Mostly this is true. But there's a structural ceiling that scaling alone can't raise, and it sits right at the final layer of the network. It's called the softmax bottleneck. Understanding it explains why some models hit a performance wall that raw compute can't fix, and why certain architectural choices (mixture of experts, output factorisation, mixture of softmaxes) exist beyond just increasing model size. What the Softmax Bottleneck Actually Is At the final step of a language model, you need to produce a probability distribution over every token in the vocabulary — typically 30,000 to 200,000 tokens. The model does this by taking the hidden state vector h (dimension d ), multiplying by an output embedding matrix W (shape d × V , where V is vocabulary size), and applying softmax.…