Data attribution for LMs works much better when you focus on the most informative tokens within documents using negative log-odds gradients instead of averaging attribution scores across entire documents—a simple fix that improves performance by 35-72 percentage points.
This paper diagnoses why data attribution methods for language models fail by creating a controlled benchmark with synthetic training data.