If you're using SAEs to understand what features neural networks learn, ConSDL's per-sample optimization makes those discoveries reproducible and reliable, solving a major problem where different training runs find completely different dictionaries.
This paper addresses inconsistency in sparse autoencoders (SAEs) used to interpret AI models by proposing Consistent Sparse Dictionary Learning (ConSDL). Instead of using a single linear encoder, ConSDL optimizes sparse codes per sample, achieving much more consistent and reproducible dictionary discovery across different training runs—improving consistency by up to 35.9% over existing methods.