In-context learning works through a mechanistic circuit of identifiable components (memory storage, key matching, gating) that you can trace causally from input to output, and this same circuit appears consistently across model sizes and architectures.
Researchers reverse-engineered how transformers perform in-context learning by identifying the specific circuit of attention heads and signals that enable few-shot classification.