Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization

Marco Colombo, Balaji A Jayaram, Sowmyakumari Selvaraj, Sumit Tokkar, Animesh Kumar et al.|November 6, 2026arXiv

Key Takeaway

You can assess LLM-generated code summary quality by measuring inter-model agreement and using an LLM-as-judge, eliminating the need for expensive human reference summaries.

Summary

This paper proposes a method to evaluate whether AI-generated code summaries are correct without needing human reference summaries. It uses multiple AI models to generate summaries, has another AI model compare them pairwise, and combines these judgments with a calibrated rule that rewards agreement and penalizes only confirmed disagreements.

evaluation applications

Key Terms

llm-as-a-judge inter-model-agreement reference-free-evaluation calibrated-aggregation