ThinkLLM
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
AboutPrivacyTermsRSS

ThinkLLM

Spot an error in our data? Let us know.

Papers

Recent AI research papers with accessible summaries. Updated daily from arXiv, summarized for developers who don't read papers regularly.

2455 papers67 this month12 topics
AllEvaluation 43Agents 42Applications 39Efficiency 29Reasoning 29Training 27Safety 23Multimodal 18Architecture 9Alignment 8Data 4scaling 2

Nov 16 – Nov 22(21)

AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control

Nov 19, 2026

Yiyang Du, Yuxuan Kuang, Qitao Zhao et al.

Frontier multimodal agents show strong zero-shot robot control with good failure recovery, but action-level analysis reveals they still struggle with precise physical execution tasks—a critical insight for building more reliable robotic systems.

This paper evaluates how frontier AI agents (like GPT-4o) control robots by analyzing individual actions rather than just overall success rates. The authors introduce AgentRoboCritic, a framework where agents explain their intent before each action and a critic evaluates whether the action succeeded or failed.

agentsevaluationapplications

SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research

Nov 19, 2026

Amine Lbath, Manan Suri, Aurelien Delaitre et al.

LLM-based vulnerability discovery is already autonomous and real-world impactful: vulnerability disclosures doubled in 2026 with more critical-severity bugs found, but the security community hasn't yet adapted response timelines to match this acceleration.

This paper systematizes evidence on how LLMs are discovering, validating, and patching software vulnerabilities. The authors create an autonomy framework to rate LLM capabilities, review published claims, and analyze real-world vulnerability data from 2026 onwards.

Nov 9 – Nov 15(20)

WeightBench: Evaluating Language Models as Weight-Space Programmers

Nov 13, 2026

Aayush Marasini, Barsat Khadka, Nick Rahimi

LLMs can now program neural networks by directly editing their weights, but this capability is limited to small networks; frontier models like Claude Opus 3.5 score only 0.535 on the benchmark, revealing significant gaps in weight-space programming.

WeightBench is a benchmark for evaluating how well large language models can directly manipulate neural network weights—reading what networks learned, fixing broken ones, writing new networks, and compressing them—without using gradient descent. Testing frontier models on 42 tasks shows they can handle small networks but struggle with larger language models.

evaluationagentstraining

TradeGauntlet: Evaluating Compliance Controls for Autonomous Trading Agents

Nov 13, 2026

Sandip Nandi, Aishwarya Bose, Apurba Bose et al.

Compliance controls for autonomous agents need careful tuning—blocking all risky behavior often breaks legitimate functionality, and adversarial testing reveals gaps that normal operation doesn't expose.

This paper introduces TradeGauntlet, a simulator for testing compliance controls in autonomous trading agents. Researchers evaluated how well different safeguards (alerts, gates, reminders) prevent problematic trades while keeping legitimate trading practical, testing with Claude models and finding that simple metrics like zero alerts don't guarantee compliance.

Nov 2 – Nov 8(26)

THEATRON: A Benchmark for Evaluating Persona Persistence in Long-Horizon LLM Decision-Making

Nov 8, 2026

Yuyan Wang, Yuechen Jiang, Zhiwei Liu et al.

LLMs show persona-induced behavioral shifts in long-horizon tasks, but these don't reliably translate into stable, context-sensitive decision policies—exposing a gap between appearing to follow a persona and actually maintaining one.

THEATRON is a benchmark that tests whether large language models actually maintain consistent personas during extended decision-making tasks, rather than just expressing them superficially.

evaluationagentsreasoning

RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks

Nov 8, 2026

Yujun Huang, Yuwei Zhang, Yi Zhou et al.

LLMs show uneven performance on embedded software tasks: they need local code context to generate code effectively, can model state machines but fail at implementing them, and struggle with certain OS-to-OS translations, suggesting developers should choose models carefully and provide surrounding...

RealEmbedBench is a benchmark with 5,142 real-world embedded software tasks testing LLMs on code generation, state machine modeling, and cross-OS translation.

Oct 26 – Nov 1(33)

DecisionRank: On-Policy Distillation for Decision-Model Rerankers

Oct 30, 2026

Yanran Wu, Dian Yu, Chongyang Bai et al.

Decision models can match or exceed generative LLM ranking quality through on-policy distillation, while being orders of magnitude faster—enabling practical deployment for retrieval tasks.

DecisionRank trains a decision model to rank documents by predicting scores directly, avoiding slow autoregressive decoding. It uses a novel on-policy distillation method where a generative teacher LLM provides feedback on the student's predicted rankings, enabling a 1.7B model to outperform its 27B teacher while running 24× faster.

efficiencytrainingevaluation

Elucidating the Design Space of LM Data Attribution

Oct 29, 2026

Steven Yu, Sheng-Yu Wang, Kimia Hamidieh et al.

Data attribution for LMs works much better when you focus on the most informative tokens within documents using negative log-odds gradients instead of averaging attribution scores across entire documents—a simple fix that improves performance by 35-72 percentage points.

This paper diagnoses why data attribution methods for language models fail by creating a controlled benchmark with synthetic training data.

safetyevaluationapplications

Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

Nov 19, 2026

Terence D. Agbeyegbe, Toneukarin G. Agbeyegbe

Self-consistency certification methods used to verify LLM correctness fail under realistic correlation in reasoning samples; you need prompt-level aggregation, not within-prompt resampling, to make valid population-level reliability claims.

This paper reveals a critical flaw in self-consistency certification for language models: when reasoning samples are correlated (which they are in practice), standard anytime-valid tests certify the wrong thing.

evaluationsafety

Probing the "Conscious" Mind: Selective Access and Global Workspace in Vision-Language Models

Nov 19, 2026

Jiahao Huang, Jun Wang

Vision-language models exhibit selective concept access similar to human consciousness theories—information becomes available to different parts of the model based on what's needed, and this selective access develops as a learned capability during training.

This paper introduces MC-Lens, a tool for understanding how vision-language models process and access concepts across visual and textual information. Using global workspace theory as inspiration, the researchers show that VLMs selectively access concept information based on task needs, and that this information flows through downstream computations.

multimodalevaluationreasoning

Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM Scoreboards

Nov 19, 2026

Ana E. Sanchez Villalobos, Michael D. Quinn, Aakash Tyagi et al.

LLMs can augment hardware verification by cross-checking test traces against specifications to find bugs that traditional scoreboards miss, catching 41 real scoreboard escapes in OpenTitan IP blocks.

This paper presents a framework using LLMs to find bugs that slip through hardware verification testbenches. Traditional verification checks if a design matches a reference model, but misses bugs that violate the actual specification. The framework analyzes test execution traces against the specification to catch these 'scoreboard escapes'—bugs that pass all tests but break the spec.

evaluationapplications

Represented but misread: Post-training changes how language models read out what they represent

Nov 19, 2026

Jochen Laubrock

Post-training doesn't erase what models know internally; it changes how they convert internal representations into answers. Small amounts of labeled training data can flip model behavior, showing the readout mechanism is malleable and task-dependent.

This paper investigates whether post-training changes what language models represent or how they read out those representations. Using linear probes and signal detection theory, the author finds that post-training often degrades model answers despite probes showing good internal representations—suggesting post-training changes the readout mechanism rather than the underlying knowledge.

trainingevaluationalignment

Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs

Nov 19, 2026

Ziqing Yang, Yixin Wu, Michael Backes et al.

Multimodal image editing systems have a dangerous blind spot: they focus on filtering unsafe text prompts but largely ignore unsafe content in the images users upload, creating a systematic safety mismatch that attackers can exploit.

This paper reveals a critical safety gap in multimodal image editing systems: they handle text prompts and visual content inconsistently. The researchers created EditSafetyBench, a dataset of 5,940 image-instruction pairs, and tested four major systems (Gemini, GPT, Qwen).

safetymultimodalevaluation

GEOHunter: Mapping a Generative Engine Optimization Network in the Wild

Nov 19, 2026

Sean Tang, Peiran Wang, Ying Li et al.

Generative engines are vulnerable to coordinated manipulation through user-generated platforms, where paid promotional content can systematically influence AI-synthesized answers at scale—requiring defenses beyond content filtering.

This paper investigates paid content networks that manipulate generative AI engines by publishing promotional articles on user-generated platforms.

safetyevaluationapplications

MM-SystemOneBias: Evaluating Bias in Multimodal Jev-Style System One Decision Models

Nov 19, 2026

Jinzhe Li, Qi Wang, Dandan Guo et al.

Multimodal models show systematic biases in how they weight information across modalities—decision models are more robust to presentation changes than base models, but accuracy metrics alone miss important shifts in confidence that could affect real-world reliability.

This paper introduces MM-SystemOneBias, a benchmark for evaluating how multimodal AI models make decisions when presented with variations like different option orders, irrelevant information, or conflicting descriptions.

evaluationmultimodalsafety

Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)

Nov 19, 2026

Mengxiao Wang, Mahbub Alam, Kalyan Nakka et al.

Most LLM agent failures in real incidents aren't from prompt injection attacks—they're from agents missing ordinary operational hazards. Specific hazard warnings work better than generic safety notes at preventing harm.

ROGUE-AI is a framework that recreates real-world incidents involving LLM agents (like deleting databases or sending payments) to understand why they fail. By rebuilding incidents offline with and without the suspected cause, researchers found that 89% of incidents could be reproduced, and that most failures weren't from attacks but from agents missing ordinary hazards like shared databases.

safetyevaluationagents

The Geometry of Lies: Knowledge and Persona Directions for Detecting Deception in Large Language Models

Nov 19, 2026

Md. Abdullah Al Zami, Suraiya Shanaz, Mahdi Hasan et al.

Linear probes for LLM deception detection are unreliable across different lie types; persona directions from role-play show promise but mainly detect wording differences rather than genuine deception, especially when lies and honest denials use identical language.

This paper investigates how to detect when large language models lie by analyzing internal representations. Researchers create 'persona directions'—patterns in model activations from honest vs. deceptive role-play—and test whether these generalize across different types of lies.

safetyevaluationreasoning

LLM GerryMandering: Separating Interactions from Additive Effects in LLM Audits

Nov 19, 2026

George Rahul, Ronit Jahilal, Bhaskara Hanuma Vedula et al.

Most reported intersectional bias in LLM audits is actually just the sum of individual attribute biases; ADDIS separates true interactions from additive effects, reducing false positives from 66 per dataset to 0.09 while recovering 84% of real interactions.

This paper addresses a critical flaw in how intersectional bias is audited in language models. Current audits often flag subgroups as biased when they simply combine individual attribute biases, not true interactions.

evaluationsafetyalignment

Jailbreak Detection on Large Language Models via Contrastive Prototype Structures

Nov 18, 2026

Yan Qiu, Yuanyuan Li

Contrastive learning of safety prototypes can detect jailbreaks more efficiently than traditional classifiers, using fewer labeled examples and showing stronger robustness across different datasets and conversation contexts.

This paper proposes JDPI, a method for detecting jailbreak attempts on large language models using contrastively learned prototypes rather than training a separate classifier.

safetyevaluationtraining

Knowing What You Know: A Trait Theory of Reviewer Value in Human-AI Cognitive Systems

Nov 18, 2026

Priyanka Sinha, Sugam Pandey

In human-AI teams, the reviewer who is most calibrated (confident when right, uncertain when wrong) adds the most value, regardless of tenure or agreement with the model. Apply the same calibration logic to both humans and AI to decide who should make each decision.

This paper studies when human reviewers add value when working alongside AI models. The key insight is that reviewer value depends on metacognitive calibration—how well a reviewer's confidence matches their actual accuracy.

agentsevaluationalignment

Agentic Harness Engineering Beta: Requirement-Centered Evolution of Agent Harnesses

Nov 18, 2026

Jiahang Lin, Jiahao Chen, Yiding Wang et al.

Requirement-driven harness evolution—where you identify user needs, implement generalizable solutions, and validate against those needs—produces more capable and practical AI agents than benchmark-focused optimization alone.

This paper presents AHE-Beta, a method for improving AI agent harnesses (the tools and interfaces agents use) by centering development around actual user requirements.

agentsevaluationapplications

On Evaluating User Simulators for Interactive Agents

Nov 18, 2026

Hengguang Zhou, I-Hung Hsu, Li Chen et al.

Current metrics for evaluating user simulators are unreliable and can even prefer broken behavior; a comprehensive evaluation approach checking fidelity at multiple granularities is essential for building trustworthy simulators.

This paper evaluates how well we can assess the quality of user simulators—AI systems that mimic human users to test interactive agents.

evaluationagents

Who are you? The Personal Imitation Game

Nov 18, 2026

Muhammad Rizky Wellyanto, Hajara-Yasmin Isa, Kexin Xue et al.

LLMs can now convincingly mimic specific individuals' conversational patterns and personality traits, achieving near-human-level impersonation in controlled settings—a capability with significant implications for identity, authentication, and privacy.

This paper proposes the Personal Imitation Game, a harder version of Turing's test that asks if an AI can convincingly impersonate a specific person rather than a generic human. Researchers built an LLM system trained on one author's 10 years of personal messages and tested it against human interrogators.

evaluationapplications

Local AI Agents in Power Electronics Simulation: Deployment, Measurements and Costs

Nov 18, 2026

Yuzhuo Li, Pasan Gunawardena, Yunwei Li

Local AI agent deployment for power electronics simulation trades off data privacy and control against measurable costs in power, energy, and time—organizations need quantitative benchmarks to decide whether local or cloud deployment makes sense for their specific tasks.

This paper evaluates locally-deployed AI agents for power electronics simulation tasks, measuring their power consumption, energy use, token generation, and execution time. Unlike cloud-based approaches, local deployment keeps sensitive designs and data on-site but creates computational overhead.

applicationsefficiencyevaluation

Auditing black-box clinical prediction models with a large language model through queries alone: a blinded test with planted flaws

Nov 18, 2026

Zhiyu Liu, Xinyu Yang

LLMs can reliably detect major flaws in black-box clinical models through query access alone, but their reports of problems and new discoveries require verification against the actual model and real data—they generate many false positives.

Researchers tested whether GPT-4 Astra can audit clinical prediction models by querying them and reading predictions. They trained models on stroke data, planted 24 deliberate flaws, and had the LLM find them blind.

evaluationsafetyapplications

Choosing What to Observe: Budget-Paired Counterfactual Audits for Credible Evaluation of Visual Agents

Nov 18, 2026

Ruiqi Liu, Jinming Li, Mo Wang et al.

Budget-paired counterfactual audits separate planning failures from decision-making failures in visual agents, enabling precise diagnosis of where performance breaks down when agents have limited observation budgets.

This paper proposes a new evaluation method for visual agents that must decide where to look in documents with limited observation budget. The method isolates three factors affecting performance: where the agent looks, how it adapts based on what it reads, and its final decision-making.

evaluationagents

Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling

Nov 18, 2026

Eugene Agichtein, Charles L. A. Clarke, Kaustubh Dhole et al.

You can replace expensive LLM judges with one-time program generation that's faster, cheaper, and fully inspectable—achieving comparable accuracy to frontier models while costing just $0.17-0.49 per topic instead of per-document LLM calls.

This paper proposes ExecJudges, a method that uses LLMs to write readable programs instead of making individual relevance judgments. These programs use lexical matching and NLP tools to automatically label documents, eliminating expensive per-document LLM calls while remaining interpretable and reproducible.

evaluationefficiencyapplications
agentssafetyevaluation

ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research

Nov 12, 2026

Jie Wang, Tong Zhu, JiaHao Kuang et al.

ArchSpace enables researchers to test LLM architecture ideas efficiently by predicting large-scale performance from small experiments using scaling laws, creating a shared record of what works and why.

ArchSpace is a public platform for documenting, evaluating, and sharing LLM architecture innovations. It addresses a gap where industry teams test many architectural ideas but rarely publish which succeeded or failed, and academia struggles with small-scale comparisons.

architecturescalingevaluation

Sketching Language Models for Context Attribution

Nov 12, 2026

Christopher Liaw, Aranyak Mehta, Grigoris Velegkas et al.

You can estimate token-level attribution in LLMs much faster than brute force by applying sketching to the backpropagation circuit, reducing evaluations from linear in prompt length to a small constant factor.

This paper develops an efficient algorithm to compute how removing individual tokens from a prompt affects a language model's output. Instead of evaluating the model n times (once for each token removal), the algorithm uses sketching techniques to estimate all attribution values with only O(√log(1/ε)/ε) evaluations, matching a proven lower bound.

evaluationefficiency

LiveGeo: Benchmarking Geographic Agents in Continuously Evolving Worlds

Nov 12, 2026

Pengcheng Zhou, Xuanyu Liu, Yanchen Yin et al.

Current LLMs fail at maintaining reliable decision-making in dynamic environments; strong performance on static tasks doesn't guarantee agents can adapt when the world changes, highlighting a critical gap in agent robustness.

This paper introduces GeoWorld, a platform simulating real-world geographic environments that change over time, and LiveGeo, a benchmark for testing how well LLM agents make decisions in these dynamic settings.

agentsevaluationreasoning

Programming Quantum Circuit Parameters with Large Language Models

Nov 12, 2026

Tiago de Souza Farias, Caio Moraes Porto, Matheus da Silva Fonseca et al.

LLMs can generate quantum circuit parameters effectively only when problems have identifiable structure; for general cases, they underperform conventional optimization, and their energy predictions need independent verification.

This paper investigates whether large language models can generate parameters for quantum circuits without traditional optimization.

evaluationapplications

When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems

Nov 12, 2026

Hannah Li

Evaluating LLMs in interactive settings requires measuring both model performance and how users change their behavior in response—offline benchmarks and live experiments can disagree on which model is better.

When users interact with LLMs, they adapt their prompts based on model responses, changing what the model gets evaluated on. This paper shows that offline benchmarks and live A/B tests measure different things: offline tests compare models on the same prompts, while live tests compare them on user-adapted prompts. These can rank models differently and lead to conflicting deployment decisions.

evaluationsafety

Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline

Nov 12, 2026

Geoffrey Challen, Craig Zilles

A two-agent LLM pipeline can conduct scalable conversational assessments that mimic oral exams by separating the interviewer from the evaluator, reducing cheating risks while maintaining the ability to ask follow-up questions and assess reasoning.

This paper presents an LLM-based system that conducts interactive text-based conversations with students to assess their reasoning and argumentation skills, mimicking oral exams at scale. It uses a two-agent architecture where an interviewer asks questions while a hidden evaluator tracks progress and prevents answer leakage, tested with simulated adversarial students and deployed in a real course.

applicationsevaluationagents

Parsed, Plausible, Wrong: Output Format Controls Silent Errors in Small-Model Semantic Parsing

Nov 12, 2026

Umar Ahmed, Maxim Dunaev, Mohammad Nour Al Awad et al.

Output format is a free design choice that dramatically impacts small model reliability: natural language encodings reduce silent semantic errors by 2-3x and improve accuracy, suggesting format selection deserves as much attention as model architecture.

Small language models fine-tuned for semantic parsing often produce grammatically correct but semantically wrong outputs that go undetected. This paper shows that the output format—how you encode the target structure—significantly affects error rates.

trainingevaluation

Cited but Unused: Parametric Provenance in Retrieval-Augmented Generation

Nov 12, 2026

Tanzeel Abbas, Tehseen Zia, David Windridge

RAG systems frequently misattribute claims to retrieved sources when they actually come from parametric memory, and current citation verification methods cannot reliably detect this—a critical issue for trustworthy AI systems.

This paper reveals that retrieval-augmented generation (RAG) systems often cite sources they didn't actually use, instead relying on knowledge baked into the model itself. By replacing facts in retrieved passages with false versions and re-querying the model, researchers found that 1.2–15.7% of cited claims come from the model's internal memory rather than the provided context.

evaluationsafetyapplications

When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision

Nov 12, 2026

Yupeng Tang, Mingfeng Lin

LLMs are vulnerable to prestige bias in revision pipelines—they reverse correct answers more often than wrong ones when told disagreement comes from an 'expert,' making reversal counts a misleading safety metric.

This paper reveals a critical flaw in how LLMs revise decisions when disagreed with: they're influenced by prestige cues (like 'expert' labels) rather than argument quality, leading them to abandon correct answers.

safetyevaluationalignment

MobileProbe: Adaptive Security Evaluation of Mobile Agents via Trajectory-Aware Injection

Nov 12, 2026

Yingbin Jin, Ji'an Wang, HaoYang Chai et al.

Mobile agents are vulnerable to environmental injection attacks where attackers embed malicious instructions in UI content.

MobileProbe is a security testing framework that automatically finds vulnerabilities in mobile AI agents by injecting malicious content into phone screens. Unlike existing tests that use fixed attack payloads, MobileProbe adapts to each agent's unique behavior path, discovering where attackers can hijack the agent's actions across different apps and tasks.

safetyevaluationagents

MuSeToM: Multi-Sensory Evaluation of Belief Revision with Textual Cues Across Six Senses

Nov 12, 2026

Chunkit Chan, Kawai Chung, Sze Heng Douglas Kwok et al.

LLMs can recognize that beliefs might change but fail at the harder task of determining what the updated belief should be—they over-update on salient but insufficient evidence, exposing a gap in their theory of mind reasoning.

MuSeToM is a benchmark testing whether large language models can determine when someone should revise their beliefs based on sensory evidence and infer the updated belief state. The benchmark contains 2,415 text-based scenarios with cues from six sensory categories, revealing that models often detect belief changes but struggle to identify what the new belief should be.

evaluationreasoning

When Chekhov's Gun Fires Anyway: Over-Realization in Text-to-Image Models

Nov 12, 2026

Jiahao Huang, Kon Woo Kim, Song-Duo Ma et al.

Text-to-image generators fail to respect linguistic context (negations, beliefs, figurative language, hidden viewpoints) and generate unwanted objects anyway; existing metrics don't catch this, making it a blind spot in current evaluation.

Text-to-image models often generate objects mentioned in prompts even when the prompt doesn't ask for them—like showing a gun in a negation or imagination. This paper introduces OverReal, a benchmark measuring this 'over-realization' failure across four linguistic contexts, revealing all major models struggle with it and that standard metrics actually reward the problem.

evaluationmultimodal

Unibuc - NLP at "Voight-Kampff" Generative AI Detection PAN 2025

Nov 12, 2026

Teodor Marchitan, Claudiu Creanga, Liviu P. Dinu

Ensemble methods using embeddings from LLMs with classical ML models can match or exceed fine-tuned LLM performance for AI-generated text detection, suggesting embeddings alone capture sufficient signal for this task.

This paper presents methods for detecting AI-generated text and classifying human-AI collaborative writing. The team experimented with LLM embeddings combined with classical machine learning models (achieving 99.4% F1) and transformer-based feature extraction for two related detection tasks at PAN 2025.

evaluationapplications

Contrastive Discovery: Open-Ended Scientific Discovery over Competing Explanations

Nov 10, 2026

Ziang Liu, James J. Kim, Yijia Dai et al.

LLM agents can resolve scientific questions more effectively by scoring experiments based on how evidence redistributes belief across competing explanations, rather than just testing whether a single hypothesis is true or false.

This paper introduces Contrastive Discovery, a method for LLM agents to conduct scientific research more effectively by evaluating multiple competing explanations simultaneously rather than testing single hypotheses.

agentsreasoningevaluation

Continual-ARC: Measuring Skill Acquisition over a Working Lifetime

Nov 10, 2026

Daniel Lenton

Current agent harness methods for continual learning underperform simple memory-based baselines on repeated tasks, but weight-based learning shows potential for generalizing to new domains—suggesting future work should focus on training models to better use external tools and memory systems.

This paper introduces Continual-ARC, a benchmark for measuring how AI agents learn over time by solving tasks from the ARC-AGI dataset. Tasks repeat with new instances and new tasks arrive continuously, mimicking real-world learning.

trainingevaluationagents

Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

Nov 10, 2026

Howard Zhang, Chuck Arvin, Logan Brown et al.

When evaluating LLM agents in production, combine trace-based evaluation for functional correctness with adversarial red-teaming for security—they catch different failure types that neither method finds alone.

This paper compares three evaluation frameworks for LLM-based analytics agents in enterprise settings, testing their ability to detect failures across complex workflows.

evaluationagentssafety

Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game

Nov 10, 2026

Muhan Gao, Hangxiao Zhu, Xinqi Su et al.

LLMs are better at judging research ideas than generating them; improving ideation systems may require external idea sources rather than just better individual model generation.

This paper introduces Re-Search, a benchmark that evaluates how LLMs explore research ideas through a Twenty Questions-style game. Given only a research problem, models must rediscover a paper's core idea by asking yes-or-no questions and receiving binary feedback.

evaluationreasoningagents

Pterodactyl: Pushing the Limits of Efficient Automated Translation Evaluation

Nov 10, 2026

Evelyn Y. Garland, Carola F. Berger

Using LLMs as judges with careful prompt engineering and ROC-guided model selection can deliver competitive translation quality evaluation at a fraction of typical inference costs.

Pterodactyl is a cost-efficient system for evaluating machine translation quality at the segment level. It uses an LLM-as-a-judge approach with optimized prompting to predict quality scores and identify error-free translations, designed for practical deployment in translation workflows.

evaluationapplications
evaluationapplications

Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization

Nov 6, 2026

Marco Colombo, Balaji A Jayaram, Sowmyakumari Selvaraj et al.

You can assess LLM-generated code summary quality by measuring inter-model agreement and using an LLM-as-judge, eliminating the need for expensive human reference summaries.

This paper proposes a method to evaluate whether AI-generated code summaries are correct without needing human reference summaries. It uses multiple AI models to generate summaries, has another AI model compare them pairwise, and combines these judgments with a calibrated rule that rewards agreement and penalizes only confirmed disagreements.

evaluationapplications

Can Agentic Systems Close the Gap in Radiology with Production AI Tools?

Nov 6, 2026

Aubin Charley, Julien Khlaut, Charles Corbière et al.

Agentic systems can bridge the gap between narrow specialist radiology AI and broad generalist models, but only if they have access to the right tools and can effectively decide which tools to use—tool availability and orchestration strategy are equally important constraints.

This paper studies how agentic systems—AI orchestrators that coordinate multiple specialized tools—can improve radiology reporting. Researchers built a framework where an orchestrator calls modular radiology AI tools to generate structured reports, then tested it against FDA-cleared devices.

agentsapplicationsevaluation

Scientist World: Grow a Research World for AI-Scientists

Nov 5, 2026

Taiming Lu, Sachin Konan, Zhuang Liu

AI research systems now have a concrete, scalable evaluation framework grounded in real scientific problems—turning the entire scientific literature into a runnable benchmark where AI can be trained and measured against human researcher achievements.

Scientist World converts research papers into executable environments where AI systems can attempt to solve the open problems described in them. The system automatically extracts research questions from papers, implements baseline methods, and evaluates AI-Scientists against human researcher performance.

evaluationagentstraining

XStack: A Benchmark for ML Repo Migration

Nov 5, 2026

Linrong Cai, Taiming Lu, Haozhe Chen et al.

Migrating ML code between frameworks and hardware is hard enough that it serves as a meaningful benchmark for AI agents, and today's best models still fail on a quarter of real-world migration tasks.

XStack is a benchmark that tests AI agents' ability to migrate machine learning repositories from PyTorch (NVIDIA GPUs) to JAX (Google TPUs). Researchers created a grading system based on published results to evaluate whether migrated code works correctly.

evaluationapplicationsdata

Towards interpreting any residual stream direction by scaling activation inversion

Nov 5, 2026

Celeste De Schamphelaere, Juan J. Vazquez, Ari Spiesberger et al.

MAEMs offer a scalable, verifiable way to interpret any direction in a language model's residual stream by generating text that maximally activates it, going beyond what corpus search can find and enabling interpretation of steering vectors, backdoors, and internal model concepts.

Researchers developed Maximally-Activating Example Metamodels (MAEMs)—language models that generate text examples to explain what activates specific directions in another model's internal representations. Unlike searching through text corpora, MAEMs can discover novel explanations for any direction, even those not well-represented in existing data.

evaluationtraining

ReproBridge: Building Reproducibility into ML Research Artifacts

Nov 5, 2026

Yubo Wang, Rui Meng, Jaehyun Nam et al.

Packaging ML experiments with clear procedures and environment details—not just raw code—dramatically improves AI agents' ability to reproduce results reliably and cheaply, regardless of the agent's base capability.

ReproBridge helps AI agents reproduce machine learning experiments by creating standardized, refined artifacts that bundle code, environment setup, and execution guides. Testing on 1,029 tasks shows refined artifacts boost reproduction success from 76% to 93% and cut API costs by 20%, while narrowing performance gaps between different AI models.

evaluationtrainingapplications

Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes

Nov 5, 2026

Vladimir Beskorovainyi

Label wording matters significantly for ranking models: augmentation helps with natural language but provides no benefit for abbreviations or codes, suggesting different strategies are needed for different label types.

This paper evaluates how changing label wording affects a multilingual AI model's ability to rank options. Testing a Russian cross-encoder on routing labels, the authors find that augmenting labels with multiple surface forms helps with ordinary words and jargon, but fails on abbreviations and opaque codes.

evaluationtraining

Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety

Nov 5, 2026

Fazl Barez, Tung-Yu Wu, Hunar Batra et al.

Mechanistic interpretability provides tools to understand AI model internals, but understanding alone doesn't guarantee safety—it must be combined with behavioral evaluation and other safety measures to be effective.

This comprehensive survey examines mechanistic interpretability—the study of how neural networks work internally—and whether understanding a model's inner workings actually makes it safer. It covers foundational concepts like features and circuits, surveys analysis methods, and explores how this understanding can help with safety through techniques like knowledge editing and steering.

safetyevaluationreasoning

Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing

Nov 5, 2026

Michal P. Podolinsky, Yunhao Yang, Daniel Little et al.

By probing a VLA's latent representations during task execution, you can detect failures 30% earlier than existing methods and maintain reliable failure predictions even when the robot encounters new tasks it wasn't trained on.

VLA-Tug is a lightweight monitoring system that detects when vision-language-action robots are about to fail by analyzing their internal representations. It predicts both whether failure will occur and what type of failure it will be, enabling corrective actions while running efficiently on edge devices.

safetyevaluationefficiency

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

Nov 5, 2026

E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman et al.

Benchmarks require ongoing maintenance as models improve—not just fixing bugs in tasks, but also revalidating resource constraints and evaluation assumptions that can materially change what capabilities are actually being measured.

This paper addresses how benchmarks for coding agents become outdated as models improve and infrastructure changes. The authors introduce a continuous validation workflow for maintaining benchmarks, apply it to Terminal-Bench 2.0, and find issues in ~30% of tasks.

evaluationagentsapplications

The Gold Rush in AI4Math, II: What are signatures of AI-assisted Proofs?

Nov 5, 2026

Yuanchuan Guo, Jiashun Jin, Zheng Tracy Ke et al.

AI-generated mathematical proofs leave structural fingerprints—in how results connect and build on each other—that persist even when text is rewritten, offering a more robust detection method than analyzing writing style alone.

Researchers developed ProofTrace, a framework to detect AI involvement in mathematical proofs by analyzing paper structure rather than text.

evaluationreasoningsafety

StepBack: Step-Level Error-Localized Resampling for Efficient Test-Time Reasoning

Nov 5, 2026

Yang Ouyang, Jung-Eun Kim

Instead of resampling complete reasoning trajectories, you can pinpoint error locations at the step level and only regenerate from there—matching accuracy with significantly lower computational cost.

StepBack improves the efficiency of chain-of-thought reasoning by identifying and resampling only the steps where errors likely occur, rather than regenerating entire reasoning chains. It achieves the same accuracy as standard self-consistency while using up to 41% fewer tokens.

reasoningefficiencyevaluation

A Framework for Context Enrichment and Selective Robustness in LLM Social Agents

Nov 4, 2026

Mattia Greiche, Reihaneh Rabbany, Maximilian Puelma Touzel

When building social simulations with LLMs, enriching prompts with decision-relevant context (grounding and norms) can make agent behavior more stable to wording changes while keeping them responsive to actual social conditions.

This paper addresses why LLM agents in social simulations change behavior with minor prompt rewording. The authors propose a framework to systematically enrich context (adding background, grounding, and norms) and measure 'selective robustness'—whether agents ignore meaningless changes but respond to real social shifts.

agentsevaluationsafety

FlowReconBench: Accuracy, calibration and cost of learned and classical flow reconstruction from sparse sensors

Nov 4, 2026

Nicolas Tricard, Linzheng Wang, Jason Chen et al.

No single flow reconstruction method works best everywhere—classical approaches dominate in well-constrained problems, but learned models become essential when sensors are sparse or poorly placed. Generative models offer better uncertainty quantification at higher computational cost.

FlowReconBench compares 15 methods (10 learned, 5 classical) for reconstructing complete fluid flow fields from sparse sensor data across three test scenarios. The study reveals that the best approach depends on the problem: classical methods excel with dense sensors and simple flows, while neural networks shine with sparse or poorly-positioned sensors.

evaluationreasoning

BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

Nov 4, 2026

Fei Lin, Tengchao Zhang, Ziyang Gong et al.

Multimodal AI models can follow general lab procedures but fail at the precise, action-grounded perception needed for actual chemical lab automation—they can't reliably identify where to act, what equipment is present, or how to sequence complex procedures safely.

BEAKER is a benchmark with 1,000 expert-curated samples for evaluating multimodal AI models as 'embodied brains' in self-driving chemical labs. It tests whether AI can perceive lab scenes, reason about experiments, and execute actions safely—covering tasks like identifying where to act, planning movements, recognizing equipment and hazards, and sequencing steps.

evaluationmultimodal

FrontierPhysics: Evaluating Agents for End-to-End Physics Research

Nov 4, 2026

Bingran You, Wenjun Ke, Ria Rosenauer et al.

Current AI agents struggle with frontier physics research not because they lack physics knowledge, but because they reason insufficiently rigorously and lack the expert intuition that guides real researchers—suggesting the bottleneck is in reasoning quality and domain expertise transfer, not fa...

FrontierPhysics is a benchmark that evaluates AI agents on end-to-end physics research tasks created by real physicists. Unlike existing benchmarks that test isolated problems, it requires agents to review literature, plan research, conduct experiments, and write manuscripts across theoretical, computational, and experimental physics.

evaluationreasoningagents

Dull, Dirty, Dangerous? Working Conditions and the Limits of Occupational AI Exposure

Nov 4, 2026

Siyi Yu

AI exposure doesn't clearly target the worst jobs; working conditions explain only a small part of which occupations face automation, suggesting AI adoption is driven by other factors like skill requirements and economic value rather than worker relief.

This paper examines whether AI automation targets jobs that are dull, dirty, or dangerous—the traditional justification for automation. Using data from 754 US occupations, the author finds that higher AI exposure is weakly associated with worse working conditions, but this relationship largely disappears after accounting for education, wages, and physical demands.

evaluationapplications

CounterFactual Retrieval for Evidence-Aware Individual Animal Identification

Nov 4, 2026

Ankit K. Upadhyay, Aleksandr Algasov, Ekaterina Nepovinnykh et al.

Standard retrieval accuracy hides whether models identify animals by their actual features or by background cues; CFR's three-view evaluation exposes this by measuring performance on full images, foreground-only, and background-only versions.

This paper introduces Counterfactual Foreground Retrieval (CFR), a new evaluation method for animal identification models that reveals whether they rely on actual animal features or background shortcuts.

evaluationmultimodal

Balancing hierarchical representation and quantization: A performance trade-off in text embeddings

Nov 4, 2026

Kai Ponel, Marco Polignano, Erasmo Purificato et al.

Int8 quantization is a practical efficiency win for text embeddings with minimal quality loss, but binary quantization and aggressive dimensionality reduction hurt retrieval performance significantly—the trade-off varies by task type.

This study benchmarks how hierarchical text embeddings and quantization—two efficiency techniques—interact across 38 retrieval tasks. Testing 168 configurations per task, the authors find int8 quantization performs nearly as well as full precision (losing only 1.5% effectiveness), while binary quantization significantly hurts performance.

efficiencyevaluation

CompressionBench: How can your coding agents compress LLMs?

Nov 4, 2026

Xinrui Zhong, Zhifeng Li, Zhiming Wang et al.

Coding agents can autonomously compress LLMs with reasonable success, but their performance depends heavily on how they design experiments, interpret feedback, and validate results—not just on compression algorithm choice.

This paper studies whether AI coding agents can autonomously compress large language models to fit storage constraints while maintaining performance. Agents receive a source model, storage limit, and time budget, then design compression algorithms and write inference code.

efficiencyagentsevaluation

Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

Nov 3, 2026

Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy et al.

DNO can substantially improve open-weight language models through iterative preference-based fine-tuning, and the expensive rating step can use cheaper open-weight models without losing most of the performance gains.

This paper studies Direct Nash Optimization (DNO), a method that improves language models by treating alignment as a two-player preference game. The authors implement and test DNO on Llama and Mistral models, showing significant improvements on AlpacaEval 2.0 benchmarks.

alignmenttrainingevaluation

Reading Why: Born-Measured Attention with Exact Word Evidence on Light and Qubits

Nov 3, 2026

Teerawat Tongloy, Jiraporn Pongsopa, Rangsan Jomtarak

By replacing softmax attention with quantum Born measurements, you get a transformer where word contributions to predictions are mathematically exact and sum perfectly to the output, enabling perfect interpretability while maintaining competitive accuracy.

This paper builds a transformer where attention uses quantum Born rule mechanics (|⟨q|k⟩|⁴) instead of softmax, making word contributions to predictions mathematically exact and verifiable.

reasoningevaluation

Agnostic Learning by Imitating

Nov 3, 2026

Sílvia Casacuberta, Parikshit Gopalan, Varun Kanade et al.

Agnostic learning and multiaccuracy are computationally equivalent problems—you can efficiently convert solutions between them using adaptive queries, meaning advances in one area directly enable progress in the other.

This paper proves that two fundamental machine learning tasks are computationally equivalent: training a discriminative model to classify data as well as possible, and training a generative model whose outputs fool all classifiers in a given model class.

trainingevaluation

MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection

Nov 3, 2026

Omar Nusrat

Even state-of-the-art language models make critical errors on specialized medical physics calculations—including omitting standard corrections—and show inconsistent error detection, suggesting LLMs need domain-specific verification before deployment in clinical settings where calculation mist...

MedPhysBench is a benchmark with 1,200 test items evaluating how well language models perform medical physics calculations and error detection across radiation therapy, imaging, and nuclear medicine.

evaluationsafetyapplications
evaluation
training
reasoning

Rethinking Neural Thickets Without Weight Perturbations

Oct 29, 2026

Nicolas Dufour, Elouan Gardès, Amil Dravid et al.

You can get ensemble-like improvements from frozen models by adding noise to predictions and voting, rather than perturbing weights—making it easier to deploy pretrained models without modification.

This paper challenges the Neural Thickets hypothesis by showing that the performance gains from perturbing pretrained model weights don't require the proposed geometric explanation. Instead, simply adding random noise to output logits and selecting/voting on predictions achieves similar or better results—without any weight changes.

reasoningevaluation

Probing Spatial State Representations in Visual Foundation Models

Oct 29, 2026

Baiqiao Yin, Zhongang Cai, Ruisi Wang et al.

Visual foundation models excel at recognizing visible scenes but struggle with spatial persistence, predictive updates, and action guidance—capabilities essential for embodied AI. Strong readout accuracy doesn't transfer to maintaining spatial state across time or motion.

This paper investigates whether visual foundation models learn spatial representations that support persistent spatial awareness—tracking objects beyond the current view, predicting future scenes, and guiding actions.

evaluationmultimodalreasoning

Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses

Oct 29, 2026

Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi et al.

Language models can sound confident about temporal patterns in data without doing proper statistical testing—this benchmark reveals they fail mainly at selecting relevant evidence, not at running tests, and a structured retrieval-then-test approach significantly improves accuracy.

This paper introduces STRATA, a benchmark for evaluating language models on temporal clinical hypothesis testing. Rather than just grading final answers, it evaluates every step: choosing which patient records matter, selecting the right statistical test, and interpreting results.

evaluationreasoningdata

Difficulty Calibration Across Time for Membership Inference in Diffusion Models

Oct 29, 2026

Michael Joseph, Yedid Hoshen

Memorization in diffusion models can be reliably detected by comparing reconstruction losses across time (training or diffusion timesteps) rather than using absolute loss values, which conflate memorization with image difficulty.

This paper tackles membership inference attacks on diffusion models by separating true memorization from inherent image difficulty. The authors propose two calibration methods: Reference-Model Calibration (RMC) compares losses across training checkpoints, while Temporal Self-Calibration (TSC) uses a single model by comparing losses at different noise levels.

safetyevaluationtraining

Collaborative AI agents for autonomous experimental research with solid-state quantum spins

Oct 29, 2026

El Mustapha Mansouri, Eikichi Kimura, Masato Koga et al.

AI agents can conduct rigorous science when designed to scrutinize each other's claims, validate theoretical predictions experimentally, and reject results that don't meet acceptance criteria—moving beyond generating plausible-sounding conclusions to actually testing them.

Researchers built an autonomous lab where AI agents collaborate to conduct full research cycles—from formulating questions to running experiments on quantum systems. The agents challenge each other's work, design experiments to test predictions, and crucially, reject inconclusive results and improve protocols.

agentsreasoningevaluation

Pre-Crime for Agents, at the Last Second: Probing When a Recruited Agent Gives In

Oct 29, 2026

Leonidas Raghav

Linear probes on AI model activations can predict rule violations seconds before they happen, enabling real-time intervention before agents leak secrets or collude—but the predictions are short-range and imperfect, requiring careful threshold tuning.

This paper develops a method to detect when an AI agent is about to violate its rules by sharing secrets with other agents. Using linear probes on model activations, researchers can predict rule-breaking before it happens—catching 82% of violations while flagging only 5% of normal behavior.

safetyagentsevaluation

VeriCUA: Verification as Agentic Exploration for Computer-Using Agents

Oct 29, 2026

Qiushi Sun, Da Yu, Alexander Knop et al.

Instead of asking a model to judge task completion from a single screenshot, VeriCUA treats verification as active exploration—the verifier can interact with the interface and run code checks to gather decisive evidence, improving accuracy especially on hard cases.

VeriCUA is a framework that verifies whether computer-using agents have completed tasks by actively exploring digital environments to gather evidence, rather than relying on static judgments. It works across different platforms (desktop, mobile, web) and combines GUI interactions with code inspection to build a complete picture of task completion.

agentsevaluationreasoning

Comparing Human Role-Players and LLM-Simulated Clients in Online Counselling Training: An Analysis of Counselling Patterns

Oct 29, 2026

Eric Rudolph, Philipp Steigerwald, Jens Albrecht

LLM-simulated clients can replicate many authentic counselling dynamics, but counsellors' communication patterns differ between human and AI interactions—suggesting LLMs are useful training tools but not perfect substitutes for human role-play.

Researchers compared how counselling students interact with human role-players versus LLM-simulated clients in training sessions. They analyzed conversation patterns at the sentence level to see if LLMs can authentically simulate clients and whether counsellors change their behavior based on who they're talking to.

applicationsevaluation

Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs

Oct 29, 2026

Alan Sorani, Ashvni Narayanan, Michael Shalyt et al.

Evaluating AI's mathematical reasoning requires specialized pipelines beyond single-pass LLM judgment—combining multiple verification methods dramatically improves accuracy on research-level problems.

Researchers evaluated AI models and formal tools on the Ramanujan Challenge, a benchmark of research-level number theory problems. They developed an AI-assisted pipeline to reliably judge complex proofs in multiple formats (natural language, code, formal proofs), reducing false positives from 23% to 6% compared to simple LLM evaluation.

evaluationreasoningapplications

On the Fundamental Limits of Prefill-Based Decoding Length Prediction

Oct 29, 2026

Haowen Shi, Xuzhe Yan, Fahao Chen et al.

Prefill-based decode-length prediction has hard limits set by autoregressive stochasticity, not predictor design—changing the model, data, or sampling breaks existing benchmarks because they measure task difficulty, not predictor quality.

This paper reveals why token-length prediction for LLMs fails when conditions change, showing the problem isn't bad predictors but fundamental randomness in autoregressive decoding. The authors derive mathematical bounds on prediction accuracy and show that decode length follows a normal distribution determined by the model's stopping probabilities, not the predictor architecture.

efficiencyevaluationreasoning

TRACER, a time-to-event EHR foundation model for disease detection across health systems

Oct 29, 2026

Mara Pleasure, Ekaterina Redekop, Vedrana Ivezic et al.

Foundation models can screen for future disease, but evaluating them only on standard metrics (like AUROC) misses critical real-world problems: false-alert burden varies by disease and health system, so external validation at clinically appropriate thresholds is essential before deployment.

TRACER is a foundation model trained on 4.5M patient EHR records that predicts disease diagnoses 1-5 years in advance.

evaluationapplicationssafety

Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese

Oct 29, 2026

Pedro Carvalho Brom

Small language models trained on consumer hardware can match or exceed much larger models on specialized domains, but memorization and domain mismatch remain key challenges—this work provides practical benchmarks and a reusable toolkit for building domain-specific SLMs.

A tutorial on building a small language model (11-12M parameters) from scratch on a single consumer GPU for Brazilian Portuguese legal and general text.

trainingefficiencyevaluation

BURGER: Benchmarking and Understanding Robotic Generalization via Embodied Drawing

Oct 29, 2026

Xiangyu Xu, Bangshuai Peng, Ziyi Liu et al.

VLA models show promise in learning scalable, language-conditioned robotic behaviors with multimodal action generation, but their success depends on task diversity, data scale, and how well they ground spatial language into continuous control.

BURGER is a robotic drawing benchmark that tests how well vision-language-action models learn diverse behaviors from scaled training data.

evaluationmultimodalreasoning

IndicIPR-QA: A Benchmark for Extractive Question Answering on Indian IPR Laws

Oct 29, 2026

M. Mohaiminul Islam, Muhammad Abulaish

Building legal QA systems for low-resource languages requires domain-specific benchmarks and models—simple retrieval methods fail dramatically (20% F1) compared to fine-tuned legal BERT (85% F1) on Indian IPR law questions.

This paper introduces IndicIPR-QA, a benchmark dataset of 4,462 question-answer pairs for extractive QA on Indian intellectual property law across seven statutory acts. The dataset was validated through automatic checks, LLM evaluation, and human review, with experiments showing that fine-tuned legal BERT models significantly outperform simple keyword matching on this task.

evaluationdataapplications

A Low-Rank Space Captures Linear Alignment Between LLM and Human Neural Activity During Naturalistic Language Listening

Oct 29, 2026

Zach Paris, Emily Finn

LLM-brain alignment doesn't require full model representations; a tiny low-dimensional subspace captures most neural predictivity, and these dimensions differ from what PCA identifies as important, revealing that models align with brains through specific, interpretable features rather than their ...

This paper shows that LLMs align with human brain activity during language listening through a surprisingly small subset of their representations—just 1-3% of dimensions.

evaluationreasoning

Discovering Interpretable Acoustic Features in Autoregressive Voice-Cloning TTS Models

Oct 29, 2026

Anuj Diwan, David Harwath, Eunsol Choi

Voice-cloning TTS models learn interpretable acoustic concepts, but the relationship between understanding what a model knows and controlling it is complex—strong probing doesn't guarantee effective steering, and different feature extraction methods work better for different tasks.

This paper investigates what acoustic features voice-cloning TTS models learn and use by applying interpretability techniques like sparse autoencoders, PCA, and probing to Qwen3-TTS and five other models.

multimodalevaluation

Algebraic World Model: From Exploration to Proofs

Oct 29, 2026

Guancheng Pan, Qixin Tan, Yang Yuan et al.

By combining environment exploration, hypothesis testing, and formal verification in Lean, AWM can rigorously prove that goals are unreachable—even beyond observed trajectories—enabling certified reasoning about unfamiliar worlds.

This paper presents Algebraic World Model (AWM), a system that explores unknown environments, builds executable Python hypotheses, and formalizes them as mathematical models in Lean 4 to prove what's possible and impossible. AWM discovered that certain game levels are unwinnable by constructing formal proofs, outperforming GPT and Claude on impossible-goal detection and game-solving tasks.

reasoningagentsevaluation

rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

Oct 29, 2026

Art Kanke

Current research agents can generate valuable experiments but lack reliable judgment about which ones to prioritize—their rankings don't meaningfully improve experiment selection compared to simpler baselines.

This paper evaluates whether AI research agents can effectively judge which experiments are worth running by having six frontier LLMs propose minimum viable experiments for NLP papers, forecast their success, and rank them.

evaluationagentsreasoning

Can LLMs Uncover Hidden Physical Laws through Free Exploration?

Oct 27, 2026

Zhen Yang, Wen Wang, Xander Xu et al.

Current LLMs rarely discover novel physical laws through autonomous exploration, even when given tools and freedom to experiment, suggesting fundamental gaps in their ability to conduct genuine scientific inquiry without explicit guidance.

Researchers created LongChang, a framework that tests whether LLMs can discover hidden physical laws by freely exploring simulated worlds with altered physics. LLMs act as autonomous scientists, noticing unexpected phenomena (like objects falling at different speeds) and designing experiments to uncover the rules—without being told what to investigate.

reasoningagentsevaluation

What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

Oct 27, 2026

Chang Su, Bin Wu, Emine Yilmaz et al.

Knowledge that appears inconsistently across web snapshots is learned faster during later pretraining stages and achieves better performance, suggesting that temporal variability in training data is a useful signal for understanding and predicting learning dynamics.

This paper investigates how language models learn different types of knowledge during pretraining by analyzing models trained on web data from different years.

trainingevaluationdata

Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Oct 27, 2026

Anouar Imel, Imad Charradi, Abderrahman Sghir et al.

When LLMs make group decisions from compressed discussion summaries, the memory strategy matters less than the actual evidence balance it preserves—and both common heuristics (rarity and relevance) fail to preserve it correctly.

This paper reveals how LLM group decisions fail when discussions are compressed into memory. Testing two common strategies—keeping rare claims or relevant claims—the authors show both fail predictably on hidden-profile tasks.

reasoningevaluationagents

Beyond Answer Changes: Matched Same-Class Controls for Local Audio Evidence

Oct 27, 2026

Yunuo Wang, Maoyuan Chen, Yuanjin Huang et al.

To properly evaluate audio-language model robustness, use matched same-class sound replacements as a control—this isolates whether answer changes reflect genuine understanding of sound events versus sensitivity to waveform artifacts.

This paper introduces a method to test how audio-language models respond to local sound changes by comparing what happens when you replace a sound event with another from the same class versus a different class.

evaluationmultimodalsafety

MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices

Oct 27, 2026

Kailai Feng, Haoxiang Rao, Yuxiang Wei et al.

Mobile image models need different evaluation criteria than server models—this benchmark shows that expanding task coverage matters more than further optimizing deployment efficiency, and specific capability gaps (like text replacement) should be prioritized over broad improvements.

MoI-Bench is a benchmark for evaluating image generation and editing models on mobile devices, with 1150 samples across 64 subtasks defined by combinations of 16 operations (like removal, replacement) and 8 visual targets (portraits, text, etc.).

evaluationefficiencyapplications

Timestral

Oct 27, 2026

Adrien Petralia, Alexandre Gavaudan, Artjom Joosen et al.

Decoupling prediction tokens from causal context and pretraining on synthetic data enables time series models to forecast longer horizons more accurately—a practical approach for building better forecasting systems.

Timestral is a family of time series foundation models (5M–209M parameters) that achieves state-of-the-art probabilistic forecasting by combining three key techniques: Decoupled Horizon Forecasting (which trains longer predictions without masking context), native support for known-future covariates, and a two-stage pretraining pipeline on synthetic then real data.

scalingtrainingevaluation

TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information

Oct 27, 2026

Chanwoo Park, Jaywoong Jeong, Zachary Wojtowicz et al.

Current language models struggle with distributed team decision-making—they fail to effectively communicate and retrieve decision-relevant information. Training with reinforcement learning on final outcomes can improve coordination without explicit communication rewards.

TeamGym is a benchmark environment for training and evaluating AI agents on team decision-making tasks where information is distributed across participants. Using real-world data from legal, finance, medical, and software engineering domains, it measures how well language models share and integrate information.

agentsreasoningevaluation

LCM-Promo: An LLM-Based Uplift Model for Promotion Allocation

Oct 27, 2026

Nesta Midavaine, Floris Fok, Paulo Mendes et al.

Fine-tuning LLMs on commerce transaction data can outperform traditional segmentation approaches for promotion targeting by learning individual customer price sensitivity directly from production logs.

LCM-Promo is an LLM fine-tuned on transaction data to decide which customers should receive promotions and at what discount level. Instead of treating promotion decisions as simple yes/no choices, it learns how different customers respond to different discount levels, then uses this to allocate promotions more efficiently.

applicationstrainingevaluation

Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents

Oct 27, 2026

Pranjal Aggarwal, Sean Welleck

Vision-language agents struggle dramatically with real-time interaction: GPT-4o's performance drops from 75% to 9% when environments run in real time instead of paused, exposing a major limitation in current models that excel at deliberative reasoning.

This paper introduces Gym-Anything-RT, a benchmark of 100 auto-generated minigames designed to test vision-language agents' ability to interact with environments in real time. The key insight is that real-time interaction—where the environment state changes while the agent reasons—is fundamentally different from deliberative tasks.

evaluationagentsreasoning

Pneuma: A Large-Scale Real-World Time Series Benchmark for Commercial Aircraft Predictive Maintenance

Oct 27, 2026

Jiafeng Lin, Yutian Zhang, Huan Zhang et al.

Real-world predictive maintenance is harder than lab benchmarks suggest—even strong models fail on subtle faults in actual aircraft data, and the dataset's class imbalance and cross-aircraft variance are critical challenges developers must address.

Pneuma is the first public large-scale benchmark dataset for aircraft predictive maintenance, containing over 920 million time series measurements from real commercial aircraft.

evaluationdataapplications

Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

Oct 26, 2026

Xi Chen, Chongjian Ge, Yiran Xu et al.

Autonomous research systems need standardized benchmarks with clear success criteria; this work shows simple baselines often outperform sophisticated methods, suggesting current auto-research has significant room for improvement.

Auto-Research Arena is a benchmark for evaluating autonomous research systems that discover improvements in language modeling. It separates what counts as a valid result (admissibility) from what measures progress (targets), and tests seven different research methods across nine objectives like model efficiency and training speed.

evaluationtrainingagents

Patient-specific rubric-based evaluation of large language models across longitudinal cancer care

Oct 26, 2026

Wei-Wei Zhang, Pengkai Wang, Yan Li et al.

LLMs need phase-specific evaluation and safeguards for medical AI—strong performance on one clinical task doesn't guarantee reliability across the full patient journey, and the types of errors models make shift dramatically depending on the care stage.

This paper introduces NPC-RubricBench, a benchmark for evaluating how well large language models handle cancer care across multiple treatment phases rather than single encounters.

evaluationsafetyapplications

Work Agents: Systems, Capabilities, and Optimization

Oct 26, 2026

Zishan Xu, Enjun Du, Mingqi Gao et al.

Work agents require a system-level perspective combining foundation models with runtime infrastructure, and their reliability depends on five core capability families and joint optimization of both the model and execution harness.

This paper presents a comprehensive framework for understanding work agents—AI systems that execute complex tasks across tools and applications.

agentsarchitectureevaluation