Post-training doesn't erase what models know internally; it changes how they convert internal representations into answers. Small amounts of labeled training data can flip model behavior, showing the readout mechanism is malleable and task-dependent.
This paper investigates whether post-training changes what language models represent or how they read out those representations. Using linear probes and signal detection theory, the author finds that post-training often degrades model answers despite probes showing good internal representations—suggesting post-training changes the readout mechanism rather than the underlying knowledge.