Greetings all! Quick context: this is part of an ongoing series where I'm building Rainbow DQN one component at a time on Snake and measuring what each piece actually does. The first post covered the encoding, the second covered a memory optimisation. This one is about the finding I've been teasing: which Rainbow component hurts performance on Snake. The answer is Prioritised Experience Replay (PER). Removing it from Rainbow DQN didn't just match performance. It set a new world record of 153 on a 20×20 grid, smashing the previous record of 134 set by full Rainbow (with PER), and nearly 2.5× the best published peer-reviewed result of 62 (Sebastianelli et al., 2021). The component that Hessel et al. (2018) ranked as one of Rainbow's two most important pieces actively hurts on some games such as snake. What Is PER? (And Why Does Everyone Use It?) Prioritised Experience Replay changes how an agent samples from its replay buffer.…