High agreement between LLMs and human annotators doesn't guarantee the model understands the construct being measured—you need to test whether the model follows the theory's logic or just correlates with surface features.
This paper tests whether Portugal's AMALIA language model can reliably annotate moral concepts by comparing its agreement with human coders against its actual understanding of the underlying construct.