Sequence Processing, POS Tagging, and the Context Problem By the end of this post, the fixed-window language model will be a closed chapter, and the recurrent neural network will make sense to you structurally: one new weight matrix, one thread of hidden state running through time. You'll see why n-gram and feedforward language models hit a wall on context. You'll meet the three families of sequence problems (labeling, classification, sequence-to-sequence) and see how each one maps to a different RNN setup. You'll understand the two-pass training algorithm (backpropagation through time) and why it starts leaking signal once sequences get long. Then we'll apply all of it: RNN language models, autoregressive generation (what ChatGPT does under the hood), and encoder-decoder for machine translation. This is the missing link between feedforward LMs and transformers. You already know why n-gram and feedforward LMs are limited — they see only the last n words.…