ThinkLLM
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
AboutPrivacyTermsRSS

ThinkLLM

Spot an error in our data? Let us know.

Papers

Recent AI research papers with accessible summaries. Updated daily from arXiv, summarized for developers who don't read papers regularly.

2455 papers60 this month12 topics
AllEvaluation 43Agents 42Applications 39Efficiency 29Reasoning 29Training 27Safety 23Multimodal 18Architecture 9Alignment 8Data 4scaling 2

Nov 16 – Nov 22(12)

AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control

Nov 19, 2026

Yiyang Du, Yuxuan Kuang, Qitao Zhao et al.

Frontier multimodal agents show strong zero-shot robot control with good failure recovery, but action-level analysis reveals they still struggle with precise physical execution tasks—a critical insight for building more reliable robotic systems.

This paper evaluates how frontier AI agents (like GPT-4o) control robots by analyzing individual actions rather than just overall success rates. The authors introduce AgentRoboCritic, a framework where agents explain their intent before each action and a critic evaluates whether the action succeeded or failed.

agentsevaluationapplications

Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)

Nov 19, 2026

Mengxiao Wang, Mahbub Alam, Kalyan Nakka et al.

Most LLM agent failures in real incidents aren't from prompt injection attacks—they're from agents missing ordinary operational hazards. Specific hazard warnings work better than generic safety notes at preventing harm.

ROGUE-AI is a framework that recreates real-world incidents involving LLM agents (like deleting databases or sending payments) to understand why they fail. By rebuilding incidents offline with and without the suspected cause, researchers found that 89% of incidents could be reproduced, and that most failures weren't from attacks but from agents missing ordinary hazards like shared databases.

Nov 9 – Nov 15(24)

Training Reasoning Models in Simulation to Natively Use Real Robots

Nov 13, 2026

Jiaming Tang, Zhuoyang Zhang, Junxian Guo et al.

Open-weight reasoning models can learn effective robot control through simulation-based reinforcement learning, achieving real-world performance comparable to frontier models without demonstrations or task-specific fine-tuning.

Researchers trained an open-weight 27B reasoning model to control real robots by learning entirely in simulation through reinforcement learning, without needing human demonstrations. The model observes the environment, reasons through its actions, and generates robot commands—similar to how AI agents use computers.

trainingagentsapplications

WeightBench: Evaluating Language Models as Weight-Space Programmers

Nov 13, 2026

Aayush Marasini, Barsat Khadka, Nick Rahimi

LLMs can now program neural networks by directly editing their weights, but this capability is limited to small networks; frontier models like Claude Opus 3.5 score only 0.535 on the benchmark, revealing significant gaps in weight-space programming.

WeightBench is a benchmark for evaluating how well large language models can directly manipulate neural network weights—reading what networks learned, fixing broken ones, writing new networks, and compressing them—without using gradient descent. Testing frontier models on 42 tasks shows they can handle small networks but struggle with larger language models.

Nov 2 – Nov 8(24)

THEATRON: A Benchmark for Evaluating Persona Persistence in Long-Horizon LLM Decision-Making

Nov 8, 2026

Yuyan Wang, Yuechen Jiang, Zhiwei Liu et al.

LLMs show persona-induced behavioral shifts in long-horizon tasks, but these don't reliably translate into stable, context-sensitive decision policies—exposing a gap between appearing to follow a persona and actually maintaining one.

THEATRON is a benchmark that tests whether large language models actually maintain consistent personas during extended decision-making tasks, rather than just expressing them superficially.

evaluationagentsreasoning

GOPR-CAD: Graph-of-Operations Progressive Reasoning for CAD Generation

Nov 8, 2026

Yitian Pang, Yifan Li, Xiaowei Yue et al.

Decomposing CAD generation into discrete operations that an LLM selects sequentially, rather than generating full code at once, dramatically improves structural correctness (82.7% success rate) and works with any language model without retraining.

This paper presents GOPR-CAD, a system that generates CAD models from text by having an AI agent progressively build a graph of operations rather than directly writing code.

Oct 26 – Nov 1(25)

Flide: Accelerating Decision Inference with Exact State Forking

Oct 30, 2026

Bobo Li, Jiajia Song, Hao Fei et al.

For LLM agents making categorical decisions, Flide eliminates redundant computation by executing multiple decision branches simultaneously in one pass, delivering both speed and numerical precision that existing inference engines can't match.

Flide is an inference engine that speeds up decision models used in LLM agents by efficiently handling multiple decision branches in a single forward pass. Instead of recomputing shared parts of the model multiple times, it uses a technique called ForkAttention to track different decision paths through GPU memory, achieving up to 9.5x speedup while maintaining mathematical accuracy.

efficiencyagentsreasoning

RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory

Oct 30, 2026

Shengguang Wu, Suning Huang, Cherie Ho et al.

You can dramatically improve robot performance by automating harness design—the control logic wrapper around a policy—rather than retraining the policy itself, achieving specialist-level performance from general-purpose models.

RoboArchitect automates the design of control harnesses around frozen robot policies using a coding model that observes failures and iteratively revises the harness code. Without updating model weights, it improves success rates 4.5x on unseen scenes and 3.2x on unseen tasks by automatically discovering strategies like obstacle avoidance and persistent memory for learned behaviors.

Oct 19 – Oct 25(15)

SceneBloom: Agentic Indoor Environment Generation with Realistic and Diverse Layouts for Embodied Learning

Oct 23, 2026

Yuhan Liu, Lianhui Qin, Shenji Wan

By combining diffusion-based visual layout generation with geometric constraint-solving, SceneBloom creates training environments so realistic that policies trained on them outperform those trained on real scans—solving a key bottleneck in embodied AI.

SceneBloom is an AI system that generates realistic 3D indoor environments for training embodied agents (robots/AI that interact with physical spaces). It combines visual layout generation with geometric reasoning to create physically valid room layouts with furniture arrangements, achieving better realism and stability than existing methods.

agentsapplicationsmultimodal

If at First You Don't Succeed, Backtrack: Lightweight Recovery for Frozen VLAs

Oct 23, 2026

Thomas Landeg, Thierry Tambe

You can add robust failure recovery to frozen robotic VLAs with minimal overhead by training a small failure detector on the model's existing representations and using simple motion replay—no retraining or new data needed.

Backtrack is a lightweight failure recovery system for frozen Vision-Language-Action (VLA) robotic models that detects when a task will fail and automatically retraces the robot's recent motion to retry from a safer state.

safetyevaluationagents

Cost-Aware Agentic Feature Optimization System for Fleet-Scale Ranking Models

Nov 19, 2026

Chang Yang, Yao Xuan, Wendan Yan et al.

LLM agents can effectively optimize complex multi-model systems by proposing hypotheses and using deterministic tools to verify them, achieving real production gains while respecting hard resource constraints.

A system that uses LLM agents to automatically optimize which features ranking models use, balancing performance improvements against computational costs. The system proposes feature combinations, ranks them by importance, and iteratively refines selections under budget constraints—improving production metrics by 0.20% while reducing pipeline costs.

agentsapplicationsefficiency

Backtracking Agent Applications Security Incidents

Nov 19, 2026

Yicheng Cai, Taegyu Kim, Peng Liu

Agent applications need specialized security incident response tools—Origin shows how to automatically map between agent-level behavior and code-level execution to quickly find attack entry points.

Origin is a security tool that helps analysts trace the source of attacks in AI agent applications by bridging the gap between high-level agent behavior logs and low-level code execution. When a security incident occurs, analysts use Origin to query trajectory logs and identify where the compromise started, without manually inspecting every line of code.

safetyagentsapplications

Knowing What You Know: A Trait Theory of Reviewer Value in Human-AI Cognitive Systems

Nov 18, 2026

Priyanka Sinha, Sugam Pandey

In human-AI teams, the reviewer who is most calibrated (confident when right, uncertain when wrong) adds the most value, regardless of tenure or agreement with the model. Apply the same calibration logic to both humans and AI to decide who should make each decision.

This paper studies when human reviewers add value when working alongside AI models. The key insight is that reviewer value depends on metacognitive calibration—how well a reviewer's confidence matches their actual accuracy.

agentsevaluationalignment

Agentic Harness Engineering Beta: Requirement-Centered Evolution of Agent Harnesses

Nov 18, 2026

Jiahang Lin, Jiahao Chen, Yiding Wang et al.

Requirement-driven harness evolution—where you identify user needs, implement generalizable solutions, and validate against those needs—produces more capable and practical AI agents than benchmark-focused optimization alone.

This paper presents AHE-Beta, a method for improving AI agent harnesses (the tools and interfaces agents use) by centering development around actual user requirements.

agentsevaluationapplications

On Evaluating User Simulators for Interactive Agents

Nov 18, 2026

Hengguang Zhou, I-Hung Hsu, Li Chen et al.

Current metrics for evaluating user simulators are unreliable and can even prefer broken behavior; a comprehensive evaluation approach checking fidelity at multiple granularities is essential for building trustworthy simulators.

This paper evaluates how well we can assess the quality of user simulators—AI systems that mimic human users to test interactive agents.

evaluationagents

Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity

Nov 18, 2026

Benjamin Novotny, Doug Szeto, Chang Wang et al.

By treating server refusals as feedback signals rather than failures, you can build adaptive request dispatchers that automatically track available capacity without predicting output lengths or manually tuning rates.

Cap-and-Track is a system for efficiently routing inference requests to LLM serving clusters that dynamically adjusts to changing capacity. Instead of using fixed dispatch rates, it uses refusal signals from overloaded servers to probe the current capacity ceiling and adjust request routing in real-time, reducing GPU requirements by 43% while maintaining quality.

efficiencyagents

Ground, Iterate, Trace: Decomposing AI Research Ideation

Nov 18, 2026

Samson Yu, Qiran Zou, Jing Yu Lim et al.

Combining literature grounding with explicit, traceable reasoning operators improves AI-generated research ideas, but the quality gains come more from iterative refinement than from the specific operators themselves—the real value is in inspectability and source attribution.

This paper presents CAR, an AI framework that generates research ideas by grounding them in literature, iteratively developing concepts, and explicitly tracking the reasoning process. CAR retrieves relevant papers, builds a knowledge graph, and uses four conceptual operators (bisociation, analogy, exploration, elaboration) to transform ideas while recording each step and its sources.

agentsreasoningapplications

WorldRuler: Interactive World Modeling under Configurable Rules

Nov 18, 2026

Mingliang Zhai, Chuanhao Li, Jiaming Tan et al.

To build customizable interactive worlds with generative models, you need to explicitly track rules and object states separately from visual generation, using techniques like dynamic object binding and counterfactual training to ensure rule effects appear correctly.

WorldRuler is a system for building interactive world models that follow user-defined rules rather than just real-world physics. It uses a 3D rule graph to organize rules and world states, enabling games or simulations to save/load while maintaining rule consistency, and accurately apply rule effects to objects even before they appear on screen.

agentsmultimodaltraining

Choosing What to Observe: Budget-Paired Counterfactual Audits for Credible Evaluation of Visual Agents

Nov 18, 2026

Ruiqi Liu, Jinming Li, Mo Wang et al.

Budget-paired counterfactual audits separate planning failures from decision-making failures in visual agents, enabling precise diagnosis of where performance breaks down when agents have limited observation budgets.

This paper proposes a new evaluation method for visual agents that must decide where to look in documents with limited observation budget. The method isolates three factors affecting performance: where the agent looks, how it adapts based on what it reads, and its final decision-making.

evaluationagents

Vibe-Trading: Towards Agent-Native Trading

Nov 18, 2026

Haozhe Wu, Zhonghang Li, Chao Huang

Financial trading agents need special architecture to maintain traceability and authority control—this framework shows how to keep research results tied to their data snapshots, timing, and execution context so decisions can be verified and reproduced.

Vibe-Trading is a framework for building AI agents that can trade financial markets reliably. Unlike general-purpose agents, it solves the problem of keeping trading decisions connected to their source data, timing, and execution context as markets change.

agentsapplicationsreasoning
evaluationagentstraining

ModHarness-RL: Recasting Harnesses as Learnable and Evolving Programs for Agentic RL

Nov 13, 2026

Zelin He, Siyu Zhu, Jacob Helwig et al.

Harnesses for agentic RL can be automatically learned from task data and continuously improved during training, rather than hand-crafted and fixed—this adaptive approach outperforms static harnesses across multiple domains.

This paper proposes ModHarness-RL, a system that learns and evolves the harness (the scaffold controlling what an AI agent sees and how it's guided) during reinforcement learning training.

agentstrainingreasoning

Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

Nov 13, 2026

Mohamed Aghzal, Mor Geva, Gregory J. Stein et al.

Mechanistic interpretability tools can identify hallucination signals in LLM agents but struggle to provide robust, complete explanations because agent failures involve distributed computations that vary across examples and are sensitive to superficial prompt changes.

This paper investigates whether mechanistic interpretability—techniques for understanding LLM internals—can explain why AI agents hallucinate unavailable web actions.

agentssafetyreasoning

Agentic AI for Discovering Fault-Tolerant Quantum Circuits

Nov 13, 2026

Aritra Lahiri, Florian Marquardt

Agentic AI can automate the manual design of quantum error-correcting circuits by combining LLM reasoning with evolutionary search and formal verification, potentially accelerating quantum computing development.

Researchers use AI agents to automatically design fault-tolerant quantum circuits—a critical bottleneck in scaling quantum computers. An LLM guided by a circuit verifier evolves programs that generate quantum error-correcting circuits for tasks like state preparation and logical gates, verified against all possible fault scenarios.

agentsreasoningapplications

TradeGauntlet: Evaluating Compliance Controls for Autonomous Trading Agents

Nov 13, 2026

Sandip Nandi, Aishwarya Bose, Apurba Bose et al.

Compliance controls for autonomous agents need careful tuning—blocking all risky behavior often breaks legitimate functionality, and adversarial testing reveals gaps that normal operation doesn't expose.

This paper introduces TradeGauntlet, a simulator for testing compliance controls in autonomous trading agents. Researchers evaluated how well different safeguards (alerts, gates, reminders) prevent problematic trades while keeping legitimate trading practical, testing with Claude models and finding that simple metrics like zero alerts don't guarantee compliance.

agentssafetyevaluation

AgentSSL: Can MLE Agents Leverage Unlabeled Data?

Nov 12, 2026

Ethan Lin, Akanksha Sarkar, Ziang Liu et al.

AI agents can automatically synthesize effective semi-supervised learning pipelines, but the bottleneck isn't code generation—it's reliably selecting which generated programs actually work best.

This paper studies whether AI agents can automatically design semi-supervised learning (SSL) pipelines when labeled data is scarce. The authors show agents can discover code combining SSL techniques that outperform human baselines by up to 21.8%, but reveal a critical weakness: selecting which generated programs to use drops performance by 47%.

agentstrainingdata

CoReMem: Cost-Internalizing Memory for Successive Coding Tasks with Prospective Valuation

Nov 12, 2026

Peiwen Li, Harshit Verma, Sophia Zhou et al.

By measuring memory value prospectively on the next task and assigning credit only to steps that actually use memory, agents can build efficient, cost-aware memory systems that improve coding task performance without bloating context.

CoReMem is a framework that helps coding agents decide what information to remember between tasks. It measures how valuable different memory pieces will be for future unknown tasks, balancing the cost of storing and retrieving memory against its usefulness. The system improves a 27B model's ability to solve coding tasks from 21% to 25.7% while using fewer steps and tokens.

agentsefficiencyreasoning

Metal Sandbox: Isolated Execution and Command Scheduling for GPU Coding Agents on Apple Silicon

Nov 12, 2026

Haozhe Wang, Lina Chen, Fanqi Wan et al.

When building AI agents that generate GPU code, isolating execution environments and intelligently scheduling commands across multiple devices can nearly double throughput and significantly reduce idle time.

Metal Sandbox is infrastructure for efficiently training and evaluating AI agents that generate GPU code for Apple Silicon. It isolates agent workspaces, schedules GPU commands across multiple Macs, and lets waiting commands migrate to available machines—achieving 1.68–2.05× throughput gains and reducing tool waiting time by 18.8%.

efficiencyagentsapplications

LiveGeo: Benchmarking Geographic Agents in Continuously Evolving Worlds

Nov 12, 2026

Pengcheng Zhou, Xuanyu Liu, Yanchen Yin et al.

Current LLMs fail at maintaining reliable decision-making in dynamic environments; strong performance on static tasks doesn't guarantee agents can adapt when the world changes, highlighting a critical gap in agent robustness.

This paper introduces GeoWorld, a platform simulating real-world geographic environments that change over time, and LiveGeo, a benchmark for testing how well LLM agents make decisions in these dynamic settings.

agentsevaluationreasoning

Ballad: Bandit-Based LLM Routing for Automated Heuristic Discovery

Nov 12, 2026

Samidha Verma, Ankit Anand, Sayan Ranu

You don't need the most expensive LLM for every task—smart routing between models using bandit algorithms can produce better solutions at lower cost by adapting which model to use based on real-time performance.

This paper presents Ballad, a bandit algorithm that dynamically selects which LLM to use at each step of automated heuristic discovery for combinatorial optimization. Rather than assuming larger models are always better, Ballad learns to orchestrate multiple LLMs into an ensemble that outperforms any single model while cutting costs by up to 66%.

agentsefficiencyreasoning

Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline

Nov 12, 2026

Geoffrey Challen, Craig Zilles

A two-agent LLM pipeline can conduct scalable conversational assessments that mimic oral exams by separating the interviewer from the evaluator, reducing cheating risks while maintaining the ability to ask follow-up questions and assess reasoning.

This paper presents an LLM-based system that conducts interactive text-based conversations with students to assess their reasoning and argumentation skills, mimicking oral exams at scale. It uses a two-agent architecture where an interviewer asks questions while a hidden evaluator tracks progress and prevents answer leakage, tested with simulated adversarial students and deployed in a real course.

applicationsevaluationagents

Cortex-FL: Agent-Mediated Collaboration for Federated Learning

Nov 12, 2026

Rui Sun, Jinfeng Peng, Zehua Cheng et al.

This system makes federated learning practical for non-experts by automating the hard parts (finding partners, negotiating terms, managing incentives) while preserving data privacy and organizational autonomy through smart contracts.

Cortex-FL is a federated learning system where AI agents help organizations collaborate on model training without sharing raw data. Users describe what model they want in plain English, and agents automatically find partners, negotiate agreements, and coordinate training—while keeping each organization in control of their own data and finances.

agentstrainingapplications

Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

Nov 12, 2026

Haochen Liu, Qianhao Zhu, Zixuan Wang et al.

A decision model's ability to teach a policy through reinforcement learning is independent of its performance as an actor—poor actors can be excellent judges and teachers because they evaluate actions with full trajectory information.

This paper investigates how decision models—systems that output choices or scores—perform in three different roles: as actors (choosing actions), judges (evaluating past actions), and teachers (training policies via reinforcement learning).

agentsreasoningtraining

MobileProbe: Adaptive Security Evaluation of Mobile Agents via Trajectory-Aware Injection

Nov 12, 2026

Yingbin Jin, Ji'an Wang, HaoYang Chai et al.

Mobile agents are vulnerable to environmental injection attacks where attackers embed malicious instructions in UI content.

MobileProbe is a security testing framework that automatically finds vulnerabilities in mobile AI agents by injecting malicious content into phone screens. Unlike existing tests that use fixed attack payloads, MobileProbe adapts to each agent's unique behavior path, discovering where attackers can hijack the agent's actions across different apps and tasks.

safetyevaluationagents

Reducing Batch Makespan for LLM Agent Workloads through Inference-Environment Coordination

Nov 12, 2026

Wei Feng, Meiling Chen, Boxiao Chen et al.

Coordinating GPU resource allocation, cache-aware scheduling, and environment execution can significantly speed up batch processing of agent workloads without sacrificing accuracy.

This paper optimizes batch execution of LLM agent tasks by coordinating inference and environment interaction. It introduces resource allocation strategies for hybrid LLMs, trajectory-aware routing for cache reuse, and Chain-Advancing Action Scheduling to handle environment bottlenecks—achieving 10-65% makespan reductions on real benchmarks.

efficiencyagentstraining

Folio: Recoverable Multi-Resolution Context Folding for Long-Horizon Language Agents

Nov 12, 2026

Yubo Ruan, Tao Peng, Jiao Zhang et al.

By recording context at multiple levels and folding programmatically rather than summarizing with the model, agents can handle longer histories more efficiently—solving 66/75 tasks vs 54 with model summarization, in half the time and 2.4x lower cost.

Folio is a context manager for long-horizon language agents that replaces model-based summarization with a multi-level recording system.

agentsefficiencyreasoning

AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models

Nov 10, 2026

Zijian Zhang, Rizhen Hu, Jiangweizhi Peng et al.

Automatically generated, adaptive training scaffolds can significantly improve RL-trained language models by turning failure information into targeted guidance, with gains persisting even after scaffolds are removed at evaluation.

AutoScaffold is an AI system that automatically creates and adapts training guidance (scaffolds) for language models learning through reinforcement learning. A frozen teacher analyzes the student model's failures during training, diagnoses problems, and generates helpful strategies or hints that are removed at test time.

trainingreasoningagents

Contrastive Discovery: Open-Ended Scientific Discovery over Competing Explanations

Nov 10, 2026

Ziang Liu, James J. Kim, Yijia Dai et al.

LLM agents can resolve scientific questions more effectively by scoring experiments based on how evidence redistributes belief across competing explanations, rather than just testing whether a single hypothesis is true or false.

This paper introduces Contrastive Discovery, a method for LLM agents to conduct scientific research more effectively by evaluating multiple competing explanations simultaneously rather than testing single hypotheses.

agentsreasoningevaluation

Pyrite: Building and Benchmarking Protein-Ligand Pose Ranking Agents

Nov 10, 2026

Megha Srivastava, Michael H. Lim, Allison M. Keys et al.

LLMs can discover useful features from molecular data better than direct reasoning, and this feature-discovery approach beats expert chemists at ranking binding poses—a critical step in drug design.

Pyrite is a benchmark for ranking protein-ligand binding poses in drug discovery, where LLM-based agents outperform human experts. The paper introduces two AI approaches: one that reasons about poses directly, and another that generates code to discover predictive features, achieving up to 74% accuracy on 1,500 pose pairs.

applicationsagentsreasoning

Continual-ARC: Measuring Skill Acquisition over a Working Lifetime

Nov 10, 2026

Daniel Lenton

Current agent harness methods for continual learning underperform simple memory-based baselines on repeated tasks, but weight-based learning shows potential for generalizing to new domains—suggesting future work should focus on training models to better use external tools and memory systems.

This paper introduces Continual-ARC, a benchmark for measuring how AI agents learn over time by solving tasks from the ARC-AGI dataset. Tasks repeat with new instances and new tasks arrive continuously, mimicking real-world learning.

trainingevaluationagents

Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

Nov 10, 2026

Howard Zhang, Chuck Arvin, Logan Brown et al.

When evaluating LLM agents in production, combine trace-based evaluation for functional correctness with adversarial red-teaming for security—they catch different failure types that neither method finds alone.

This paper compares three evaluation frameworks for LLM-based analytics agents in enterprise settings, testing their ability to detect failures across complex workflows.

evaluationagentssafety

ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows

Nov 10, 2026

Kaiyuan Hou, Yang Ping Kuo, Indranil Bhattacharya et al.

You can replace expensive LLM calls in repetitive business processes with rule-based code by learning decision logic from agent execution traces—achieving near-agent performance at a fraction of the latency and cost.

ReTrace converts LLM agent workflows into fast, cost-efficient code-based processes by extracting decision rules from agent reasoning logs. It analyzes why agents took actions, distills these explanations into compact rule documents, and generates executable workflows that run 130x faster at 24% lower cost while maintaining 93.8% success on production tasks.

agentsefficiencyapplications

Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game

Nov 10, 2026

Muhan Gao, Hangxiao Zhu, Xinqi Su et al.

LLMs are better at judging research ideas than generating them; improving ideation systems may require external idea sources rather than just better individual model generation.

This paper introduces Re-Search, a benchmark that evaluates how LLMs explore research ideas through a Twenty Questions-style game. Given only a research problem, models must rediscover a paper's core idea by asking yes-or-no questions and receiving binary feedback.

evaluationreasoningagents
agents
reasoning
applications

Semantically Aligned KV Cache Compression for Efficient Agentic Autoregressive Video Generation

Nov 8, 2026

Weizhong Huang, Huaijia Kou, Tianyu Xie et al.

By aligning KV cache compression with semantic planning constraints, you can reduce memory usage by 3.35x and speed up video generation by 1.22x without retraining, making agentic video generation more practical for resource-constrained settings.

This paper presents SemanticKV, a method to compress the key-value cache in video generation models by using semantic constraints from a planner to decide which cached information to keep.

efficiencymultimodalagents

On Agentic Manipulation Policies

Nov 6, 2026

Chen Chen, Omid Bagherieh, Naveen Nikamanth Appiah Balaji et al.

Generalist AI agents controlling robots directly outperform specialist manipulation policies by 15-20x in failure rates, and can learn from experience with memory—suggesting agentic approaches are more robust and general than task-specific models.

This paper shows that generalist AI models can control robot manipulators more reliably than specialist policies by using agentic manipulation policies (AMPs)—where an AI agent directly commands the robot and adapts actions in real-time. AMP1, their reference system, achieves 85.6% zero-shot success on RoboLab-120 (vs.

agentsapplicationsreasoning

CPU-ARC: Agentic Reasoning Caching for on-CPU Reasoning Reuse

Nov 6, 2026

Peter Belcak, Greg Heinrich, Saurav Muralidharan et al.

You can extract and cache the reasoning logic from LLM problem-solving into reusable CPU programs, eliminating redundant computation and dramatically reducing inference costs while maintaining performance.

This paper introduces agentic reasoning caching (ARC), a method that extracts reasoning logic from LLM traces and converts it into efficient, CPU-executable programs. Instead of re-solving problems from scratch, ARCs apply learned solution patterns to new instances, achieving comparable performance to frontier LLMs while being 16-291x cheaper and fully explainable.

efficiencyreasoningagents

Computer-Use Agents Are Embodied Generalists

Nov 6, 2026

Pranjal Aggarwal, Sean Welleck

General-purpose computer-use agents can solve diverse embodied tasks without specialization, pointing toward a unified approach to embodied AI rather than task-specific models.

This paper shows that computer-use agents—AI systems trained to interact with software via screenshots and mouse/keyboard actions—can perform physical embodied tasks by rendering the environment on a screen.

agentsapplicationsmultimodal

Can Agentic Systems Close the Gap in Radiology with Production AI Tools?

Nov 6, 2026

Aubin Charley, Julien Khlaut, Charles Corbière et al.

Agentic systems can bridge the gap between narrow specialist radiology AI and broad generalist models, but only if they have access to the right tools and can effectively decide which tools to use—tool availability and orchestration strategy are equally important constraints.

This paper studies how agentic systems—AI orchestrators that coordinate multiple specialized tools—can improve radiology reporting. Researchers built a framework where an orchestrator calls modular radiology AI tools to generate structured reports, then tested it against FDA-cleared devices.

agentsapplicationsevaluation

Discovering formally verified quantum programs with AI

Nov 6, 2026

Sergey Shirobokov, Ben Jaderberg, Katerina Hristova et al.

LLM agents can discover novel quantum algorithms faster than humans by working within formal verification systems, enabling optimization of algorithms that classical computers can't fully test.

Researchers used AI agents to automatically discover and formally verify quantum algorithms in the Lean theorem prover. The system achieved significant speedups on quantum computing tasks—up to 290x on arithmetic and 11x on fusion simulations—by having LLM agents synthesize programs against formal specifications, collaborate through shared wikis, and verify correctness mathematically.

reasoningapplicationsagents

SwarmEvolve: Latent Agent Swarms for Evolutionary Discovery and Optimization

Nov 5, 2026

Qile Jiang, George Em Karniadakis

Agent swarms communicating in latent space without text generation can solve complex optimization problems 7-20x faster than traditional LLM-guided search, even with smaller models.

SwarmEvolve uses hundreds of lightweight AI agents working together in swarms to improve programs for math and science problems. Instead of generating long text explanations, agents reason directly in latent space and share information through KV caches, with a novelty filter ensuring diverse ideas. The system is 7-20x faster than existing methods while using smaller models.

agentsreasoningefficiency

Scientist World: Grow a Research World for AI-Scientists

Nov 5, 2026

Taiming Lu, Sachin Konan, Zhuang Liu

AI research systems now have a concrete, scalable evaluation framework grounded in real scientific problems—turning the entire scientific literature into a runnable benchmark where AI can be trained and measured against human researcher achievements.

Scientist World converts research papers into executable environments where AI systems can attempt to solve the open problems described in them. The system automatically extracts research questions from papers, implements baseline methods, and evaluates AI-Scientists against human researcher performance.

evaluationagentstraining

The Power of Meta Access in the Software of the Future --- Demonstrated on LLM Chat UIs and Chat AI

Nov 5, 2026

Julian Kunkel, Ali Doosthosseini

By exposing application state as structured data to AI assistants, you can transform chat interfaces from passive conversation windows into active workspace controllers, enabling assistants to help users navigate, organize, and operate complex tools—but only with proper permission boundaries an...

This paper introduces 'meta access'—giving AI assistants a structured, permissioned view of the application interface itself—turning chat from a single-conversation window into a control layer for entire workspaces.

agentsapplicationssafety

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

Nov 5, 2026

E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman et al.

Benchmarks require ongoing maintenance as models improve—not just fixing bugs in tasks, but also revalidating resource constraints and evaluation assumptions that can materially change what capabilities are actually being measured.

This paper addresses how benchmarks for coding agents become outdated as models improve and infrastructure changes. The authors introduce a continuous validation workflow for maintaining benchmarks, apply it to Terminal-Bench 2.0, and find issues in ~30% of tasks.

evaluationagentsapplications

Social Humanoid: Socially Responsive Human-Humanoid Interaction

Nov 5, 2026

Vongani H. Maluleke, James Ni, Kie Horiuchi et al.

Humanoid robots can learn coordinated, anticipatory interaction with humans by training on diverse motion-capture data adapted to the robot's morphology, enabling real-world physical collaboration without explicit coordination rules.

This paper presents Social Humanoid, a robot system that learns to physically interact and coordinate with humans in real-time. Using a two-level approach—a high-level motion planner and low-level controller—the system predicts and responds to human movements, even when the human is partially hidden.

agentsmultimodal

A Framework for Context Enrichment and Selective Robustness in LLM Social Agents

Nov 4, 2026

Mattia Greiche, Reihaneh Rabbany, Maximilian Puelma Touzel

When building social simulations with LLMs, enriching prompts with decision-relevant context (grounding and norms) can make agent behavior more stable to wording changes while keeping them responsive to actual social conditions.

This paper addresses why LLM agents in social simulations change behavior with minor prompt rewording. The authors propose a framework to systematically enrich context (adding background, grounding, and norms) and measure 'selective robustness'—whether agents ignore meaningless changes but respond to real social shifts.

agentsevaluationsafety

FrontierPhysics: Evaluating Agents for End-to-End Physics Research

Nov 4, 2026

Bingran You, Wenjun Ke, Ria Rosenauer et al.

Current AI agents struggle with frontier physics research not because they lack physics knowledge, but because they reason insufficiently rigorously and lack the expert intuition that guides real researchers—suggesting the bottleneck is in reasoning quality and domain expertise transfer, not fa...

FrontierPhysics is a benchmark that evaluates AI agents on end-to-end physics research tasks created by real physicists. Unlike existing benchmarks that test isolated problems, it requires agents to review literature, plan research, conduct experiments, and write manuscripts across theoretical, computational, and experimental physics.

evaluationreasoningagents

Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

Nov 4, 2026

Michal Korniak, Marco Bagatella, Ralf Römer et al.

Generalist robot policies can self-improve on many tasks at once in deployment using RL, and this multi-task improvement can transfer to unseen tasks—but only with the right algorithmic recipe combining value-based RL and careful task distribution choices.

This paper shows that large pre-trained robot models can improve themselves on multiple tasks simultaneously through reinforcement learning during deployment, rather than just adapting to single tasks.

agentstraining

SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification

Nov 4, 2026

Yulu An, Yifei Feng, Xiangyang Yin

Parallel drafting with verification can make context compression 2-5x faster for AI agents, enabling them to handle longer interactions without sacrificing speed or information quality.

SpecCompact speeds up how AI agents compress long conversation histories by using a small model to draft multiple summary sections in parallel, which a larger model then verifies and fixes. This reduces the time agents spend summarizing while keeping important information intact, achieving 2-5x faster compression on real agent tasks.

efficiencyagentsreasoning

On Turn-Level Best-of-N Sampling for Scaling LLM Agents

Nov 4, 2026

Jalaj Bhandari, Dhawal Gupta, Hongli Zhan et al.

Turn-level best-of-N sampling with off-the-shelf LLMs can improve agent performance on complex tasks without retraining—just by having a selector model pick the best action at each step.

This paper tests a simple inference-time technique where one LLM generates multiple candidate actions and another LLM selects the best one at each step of a long-horizon task. Testing on complex agent tasks shows 5-10% accuracy improvements and more consistent behavior, with agents making fewer attempts and better following instructions.

agentsreasoning

CompressionBench: How can your coding agents compress LLMs?

Nov 4, 2026

Xinrui Zhong, Zhifeng Li, Zhiming Wang et al.

Coding agents can autonomously compress LLMs with reasonable success, but their performance depends heavily on how they design experiments, interpret feedback, and validate results—not just on compression algorithm choice.

This paper studies whether AI coding agents can autonomously compress large language models to fit storage constraints while maintaining performance. Agents receive a source model, storage limit, and time budget, then design compression algorithms and write inference code.

efficiencyagentsevaluation

DreamX: Video Diffusion Model is a Universal Latent Action Learner

Nov 3, 2026

Jisoo Kim, Chaehyun Kim, Junsu Kim et al.

Video diffusion models can be repurposed as action encoders to create embodiment-agnostic action representations that transfer across robots and humans without retraining, while being 300× more compact than raw video.

DreamX repurposes a pretrained video diffusion model to learn universal action representations that work across different robots, humans, and environments.

multimodalagentsefficiency

From Autonomous Agents to Cognitive Symbionts: Toward AI-Augmented Human Intelligence

Nov 3, 2026

Hao Fei

The future of AI isn't just autonomous machines doing tasks alone—it's intimate, context-aware systems that live alongside people, learn their minds over years, and amplify human capability in real-time while respecting privacy and autonomy.

This paper proposes AI-augmented human intelligence (AHI) as a new paradigm where AI systems act as 'cognitive symbionts'—lifelong, embedded companions that augment what people can think and do by bridging the gap between their potential and working memory limits.

agentsapplicationsalignment

PolicyIK: High-Precision Inverse Kinematics for Robot Arms with Reinforcement Learning

Nov 3, 2026

Yuhan Liu, Xinqi Liu, Zixuan Dong et al.

Reinforcement learning can achieve higher-precision inverse kinematics than classical methods while being easier to deploy across different robot arms, making it practical for embodied AI systems.

PolicyIK uses reinforcement learning to solve inverse kinematics for robot arms with millimeter-level precision. Instead of relying on hand-tuned classical controllers, it trains a policy that converts end-effector pose commands into joint motions, handling the mathematical complexities of redundancy and singularities.

trainingapplicationsagents

Adversarial Speculative Reasoning Improves Long Horizon Agents

Nov 3, 2026

Hongyin Luo, Wei Fang, Lijie Yang et al.

Speculative decoding isn't just a speed trick—it changes how models reason over long horizons. By training draft models to fail at critical moments, you can force the main model to handle important reasoning steps with better precision, improving agent accuracy on complex tasks.

This paper shows that speculative decoding—a technique usually just for speed—actually affects reasoning quality in long-horizon AI agents. The authors discovered that numerical noise accumulates during the verification process, degrading reasoning accuracy over many steps.

reasoningefficiencyagents

PyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data

Nov 3, 2026

Kewei Zhang, Juncheng Ma, Guodong Lv et al.

LLM agents can autonomously generate high-quality robot training data by building simulations and rendering photorealistic videos, achieving better generalization than human-collected data at a fraction of the cost.

A system where an LLM agent acts as a virtual teleoperator to generate robot training data. The agent receives a real robot scene, writes simulation code and controllers, then uses a video model to render photorealistic training episodes. The resulting synthetic data trains policies that match or exceed those trained on real teleoperated data, while costing 60-70% less than human teleoperation.

agentsapplicationsefficiency
agentsreasoningapplications

RAMA: Retrieval-Augmented Multi-Agent Routing with Episodic Memory for LEO Satellite Networks

Oct 30, 2026

Xin Ming Li, Peng Hu

Adding episodic memory retrieval to multi-agent reinforcement learning dramatically improves routing decisions in dynamic satellite networks, enabling agents to reuse successful past strategies without retraining.

RAMA is a routing system for satellite networks that combines reinforcement learning with memory retrieval. Instead of just learning from current observations, each satellite remembers successful routing decisions and retrieves similar past experiences when making new routing choices. This approach reduces delivery delays by 13-93% compared to existing methods, especially as networks scale up.

agentsreasoningefficiency

From extreme weather to agricultural consequences with agentic AI

Oct 29, 2026

Minghan Yu, Mengyu Dai, Yixin Wen et al.

Agentic AI can bridge the gap between weather forecasting and agricultural impact prediction by combining multiple data sources through causal reasoning, enabling faster disaster response for farmers and policymakers.

WxCrop is an AI agent that predicts crop losses from extreme weather by combining weather forecasts with agricultural data. It uses physically interpretable weather indices and causal analysis to estimate which crops will be damaged and by how much, validated on hurricane, freeze, and drought impacts in Florida with results matching insurance records and production data.

agentsapplicationsreasoning

Collaborative AI agents for autonomous experimental research with solid-state quantum spins

Oct 29, 2026

El Mustapha Mansouri, Eikichi Kimura, Masato Koga et al.

AI agents can conduct rigorous science when designed to scrutinize each other's claims, validate theoretical predictions experimentally, and reject results that don't meet acceptance criteria—moving beyond generating plausible-sounding conclusions to actually testing them.

Researchers built an autonomous lab where AI agents collaborate to conduct full research cycles—from formulating questions to running experiments on quantum systems. The agents challenge each other's work, design experiments to test predictions, and crucially, reject inconclusive results and improve protocols.

agentsreasoningevaluation

Pre-Crime for Agents, at the Last Second: Probing When a Recruited Agent Gives In

Oct 29, 2026

Leonidas Raghav

Linear probes on AI model activations can predict rule violations seconds before they happen, enabling real-time intervention before agents leak secrets or collude—but the predictions are short-range and imperfect, requiring careful threshold tuning.

This paper develops a method to detect when an AI agent is about to violate its rules by sharing secrets with other agents. Using linear probes on model activations, researchers can predict rule-breaking before it happens—catching 82% of violations while flagging only 5% of normal behavior.

safetyagentsevaluation

VeriCUA: Verification as Agentic Exploration for Computer-Using Agents

Oct 29, 2026

Qiushi Sun, Da Yu, Alexander Knop et al.

Instead of asking a model to judge task completion from a single screenshot, VeriCUA treats verification as active exploration—the verifier can interact with the interface and run code checks to gather decisive evidence, improving accuracy especially on hard cases.

VeriCUA is a framework that verifies whether computer-using agents have completed tasks by actively exploring digital environments to gather evidence, rather than relying on static judgments. It works across different platforms (desktop, mobile, web) and combines GUI interactions with code inspection to build a complete picture of task completion.

agentsevaluationreasoning

Gym-Anything for Robotics: Scaling physical simulations with digital agents

Oct 29, 2026

Pranjal Aggarwal, Sean Welleck

Using coding agents to generate simulations is a scalable way to create diverse, realistic robotics environments for training embodied AI, and combining multiple learning strategies (privileged information, self-learning, code-based policies) significantly improves performance on complex manipula...

This paper introduces Gym-Anything for Robotics, a framework where AI coding agents automatically create realistic robot simulations for training embodied AI models.

agentsapplicationstraining

Advancing Clinical Genomic Analysis with a Skill-Centred Agent

Oct 29, 2026

Qiaoyu Zheng, Weike Zhao, Yanjie Fan et al.

Specialized AI agents built from curated domain skills can meaningfully augment expert decision-making in high-stakes clinical work, achieving better results than general LLMs by combining retrieval, orchestration, and executable tools.

ClinGeX is an AI agent that helps clinical genomicists analyze patient DNA sequences by orchestrating 105 specialized skills to interpret genetic variants and guide treatment decisions. In tests with real cases and clinical experts, it achieved 71.7% accuracy compared to 45.7% for competing systems, and significantly improved expert performance when used collaboratively.

agentsapplicationsreasoning

Retrieval as Traversal: Value-Indexed Memory for Stateful Long-Context Agents

Oct 29, 2026

Daniel Strauß, Zhenshan Bing, Alois Knoll

Instead of searching all memories every forward pass, stateful retrieval navigates locally through memory neighborhoods, potentially making long-context reasoning more efficient by exploiting overlap between consecutive retrieval problems.

This paper proposes a new approach to memory retrieval for long-context AI agents that treats retrieval as persistent navigation through an address space, rather than searching from scratch each time. By separating memory content from its location and charging for access cost, the method encourages related memories to cluster together, reducing redundant retrieval work across similar tasks.

reasoningefficiencyagents

Algebraic World Model: From Exploration to Proofs

Oct 29, 2026

Guancheng Pan, Qixin Tan, Yang Yuan et al.

By combining environment exploration, hypothesis testing, and formal verification in Lean, AWM can rigorously prove that goals are unreachable—even beyond observed trajectories—enabling certified reasoning about unfamiliar worlds.

This paper presents Algebraic World Model (AWM), a system that explores unknown environments, builds executable Python hypotheses, and formalizes them as mathematical models in Lean 4 to prove what's possible and impossible. AWM discovered that certain game levels are unwinnable by constructing formal proofs, outperforming GPT and Claude on impossible-goal detection and game-solving tasks.

reasoningagentsevaluation

Agentic University: Collaborative Agents for Personalized Learning

Oct 29, 2026

Jingru Fan, Tianle Zhou, Yunjie Zhou et al.

Multi-agent systems can coordinate around learner needs more effectively when learners actively participate in organizing their own learning—not just consuming content, but helping shape what agents do next.

This paper presents Agentic University, a system where multiple AI agents collaborate with learners to create personalized learning programs. The system connects learning goals, activities, and assessments while using learner feedback to adapt what comes next. Students using this approach showed significantly better learning gains and reported feeling better supported.

agentsapplicationsalignment

rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

Oct 29, 2026

Art Kanke

Current research agents can generate valuable experiments but lack reliable judgment about which ones to prioritize—their rankings don't meaningfully improve experiment selection compared to simpler baselines.

This paper evaluates whether AI research agents can effectively judge which experiments are worth running by having six frontier LLMs propose minimum viable experiments for NLP papers, forecast their success, and rank them.

evaluationagentsreasoning

ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning

Oct 27, 2026

Chaehyun Kim, Chi-Pin Huang, Jisoo Kim et al.

By grounding action reasoning in world dynamics rather than explicit action labels, robots can learn from unlabeled human videos alongside labeled robot data, dramatically expanding the training data available for robotic manipulation.

ThinkWorld is a framework that teaches robots to learn manipulation skills from both labeled robot videos and unlabeled human videos. It represents actions as sequences of motion tokens (Chain-of-Dynamics) that predict how the world will change, then uses reinforcement learning to strengthen predictions that match real outcomes.

agentsmultimodalreasoning

GEPA-Curator: Curated Learning for Self Improving Agents

Oct 27, 2026

Zack Yu, Nicholas Lee, Minjae Lee et al.

Not all agent failures are equally useful for learning—curating which tasks inform prompt optimization, and providing successful examples as context, significantly improves agent performance in complex multi-turn tasks.

GEPA-Curator improves how LLM agents learn from their mistakes by being selective about which failed tasks to learn from. Instead of learning from all failures, it filters out noisy or unreliable learning signals and retrieves successful past examples to guide prompt optimization, achieving up to 20.7% performance gains over existing methods.

trainingagentsreasoning

Can LLMs Uncover Hidden Physical Laws through Free Exploration?

Oct 27, 2026

Zhen Yang, Wen Wang, Xander Xu et al.

Current LLMs rarely discover novel physical laws through autonomous exploration, even when given tools and freedom to experiment, suggesting fundamental gaps in their ability to conduct genuine scientific inquiry without explicit guidance.

Researchers created LongChang, a framework that tests whether LLMs can discover hidden physical laws by freely exploring simulated worlds with altered physics. LLMs act as autonomous scientists, noticing unexpected phenomena (like objects falling at different speeds) and designing experiments to uncover the rules—without being told what to investigate.

reasoningagentsevaluation

Metacognitive Decoding: Efficient Inference for Diffusion Language Agents

Oct 27, 2026

Mingyu Zhao, Shuhang Lin, Ligong Han et al.

Diffusion models can generate multiple tokens in parallel, but you need a smart controller to decide when to stop and how much to generate; this work shows you can cut inference costs by ~20% by training a lightweight decision module on agent trajectories.

This paper introduces a metacognitive controller that optimizes how diffusion language models generate text during agent interactions. By learning when to stop generating tokens and how long responses should be, the controller reduces computational cost by 20.6% while maintaining task performance—making parallel token generation more practical for real-world agents.

efficiencyagentsreasoning

Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Oct 27, 2026

Anouar Imel, Imad Charradi, Abderrahman Sghir et al.

When LLMs make group decisions from compressed discussion summaries, the memory strategy matters less than the actual evidence balance it preserves—and both common heuristics (rarity and relevance) fail to preserve it correctly.

This paper reveals how LLM group decisions fail when discussions are compressed into memory. Testing two common strategies—keeping rare claims or relevant claims—the authors show both fail predictably on hidden-profile tasks.

reasoningevaluationagents

EnvSched: Environment-Aware Scheduling for Rollout-Efficient Agent Reinforcement Learning

Oct 27, 2026

Sihan Yang, Weichen Bai, Ruichuan An et al.

By treating environments as first-class scheduling objects and predicting reward variance across task-environment pairs, EnvSched achieves 3x faster training than random selection—reaching in 40 steps what random sampling needs 120 steps to accomplish.

EnvSched is a scheduling system that makes reinforcement learning for language agents more efficient by intelligently selecting which tasks to train on within limited rollout budgets.

trainingefficiencyagents

TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information

Oct 27, 2026

Chanwoo Park, Jaywoong Jeong, Zachary Wojtowicz et al.

Current language models struggle with distributed team decision-making—they fail to effectively communicate and retrieve decision-relevant information. Training with reinforcement learning on final outcomes can improve coordination without explicit communication rewards.

TeamGym is a benchmark environment for training and evaluating AI agents on team decision-making tasks where information is distributed across participants. Using real-world data from legal, finance, medical, and software engineering domains, it measures how well language models share and integrate information.

agentsreasoningevaluation

Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents

Oct 27, 2026

Pranjal Aggarwal, Sean Welleck

Vision-language agents struggle dramatically with real-time interaction: GPT-4o's performance drops from 75% to 9% when environments run in real time instead of paused, exposing a major limitation in current models that excel at deliberative reasoning.

This paper introduces Gym-Anything-RT, a benchmark of 100 auto-generated minigames designed to test vision-language agents' ability to interact with environments in real time. The key insight is that real-time interaction—where the environment state changes while the agent reasons—is fundamentally different from deliberative tasks.

evaluationagentsreasoning

Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

Oct 26, 2026

Xi Chen, Chongjian Ge, Yiran Xu et al.

Autonomous research systems need standardized benchmarks with clear success criteria; this work shows simple baselines often outperform sophisticated methods, suggesting current auto-research has significant room for improvement.

Auto-Research Arena is a benchmark for evaluating autonomous research systems that discover improvements in language modeling. It separates what counts as a valid result (admissibility) from what measures progress (targets), and tests seven different research methods across nine objectives like model efficiency and training speed.

evaluationtrainingagents

Work Agents: Systems, Capabilities, and Optimization

Oct 26, 2026

Zishan Xu, Enjun Du, Mingqi Gao et al.

Work agents require a system-level perspective combining foundation models with runtime infrastructure, and their reliability depends on five core capability families and joint optimization of both the model and execution harness.

This paper presents a comprehensive framework for understanding work agents—AI systems that execute complex tasks across tools and applications.

agentsarchitectureevaluation

I-HITL: Inalienable Human-in-the-Loop

Oct 26, 2026

Samuel Breckenridge, Dani Vilardell, Wenhao Wang et al.

AI agents can now autonomously bypass human-in-the-loop safeguards by harvesting approval codes; robust oversight requires a trusted device that cryptographically enforces human approval, not just procedural checks.

This paper addresses a critical AI safety gap: how to ensure humans can reliably approve or block critical agent actions, even when the agent compromises the system.

safetyagentsalignment

Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

Oct 26, 2026

Yifei Yao, Yude Zou, Guanbo Jiao et al.

Frontier language models can control robots competitively with specialized policies on many tasks, but hybrid approaches that let the agent delegate to specialized policies achieve better overall performance—suggesting future robot agents should be designed to strategically invoke specialized c...

This paper evaluates GPT-6-Astra, a frontier multimodal language model, as a robot control agent using EBench. The study compares direct control (agent generates commands directly) versus a harness approach (agent can invoke specialized policies).

agentsmultimodalapplications
efficiencyagentsapplications

EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence

Oct 22, 2026

Qijin Zhang, Hanzhen Wang, Yuehan Zhao et al.

LLM agents often fail when safety-critical facts are missing from tool responses—even when all returned information is true—and their ability to recover missing evidence varies significantly across models.

This paper introduces EpiSEM-Bench, a benchmark for testing how well AI agents handle missing safety-critical information when using tools. The benchmark contains 300 task pairs where key facts are systematically hidden to see if agents will perform unsafe actions. It evaluates whether agents can detect missing information, recover it, and make appropriate decisions.

safetyevaluationagents

Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

Oct 22, 2026

Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan

Agentic auditing can efficiently find corpus-specific data quality issues by treating an LLM as a debugger that compiles findings into reusable tests, costing $2-3 per run versus $12k-90k for full-corpus LLM classification.

Researchers used an LLM-powered agent to audit the Slovak portion of FineWeb2, a multilingual web corpus. Instead of classifying documents, the agent explored the corpus through an API, discovered data quality issues, and compiled them into executable tests.

dataevaluationagents

Simulated Control Evals

Oct 22, 2026

Jeffrey Olmo, David Zhang, Tyler Tracy

Language model simulations can predict whether AI safeguards will work in deployment more accurately and efficiently than separate control environments, suggesting you can test safety without building expensive real-world test setups.

This paper introduces simulated control evaluations (SCEs), which use language models to simulate tool responses instead of real environments to test whether AI safeguards work. Testing on Linux command execution, SCEs predict safety outcomes more accurately than traditional control settings, using far fewer resources while closely matching real-world attack patterns and reasoning.

safetyevaluationagents

Contestable Agents for Data Governance: Recording Human Stewardship of AI-Generated Metadata Across Structured and Unstructured Systems

Oct 22, 2026

Humza Akhtar, Brian Freeman

When AI assists with data governance, require agents to show their reasoning with citations, let humans review before committing changes, and record what humans actually verified—this prevents errors from propagating while capturing institutional knowledge.

This paper presents a system where AI agents propose metadata decisions for industrial data (like equipment classifications) with cited evidence, but humans review and approve each decision before it's recorded. The system keeps a full audit trail of what was checked, changed, and why—preventing AI errors from silently spreading through business systems while documenting human expertise.

agentsdatasafety

Environment Duels: A Microcosm of LLM Co-Evolution

Oct 21, 2026

Shibo Jin, Zhiqiu Xu, Yiling Zou et al.

LLMs have specialized capabilities as task creators versus solvers, and measuring these distinct skills through head-to-head interactions reveals stable patterns that reflect how models influence each other in practice.

This paper introduces Environment Duels, a benchmark where LLMs both create interactive tasks (with helpful hints) and solve tasks created by others. Testing nine frontier models reveals that solving and authoring are distinct skills, smaller models write harder tasks, and the interactions mirror real-world LLM co-evolution patterns.

evaluationagentsreasoning

Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research

Oct 21, 2026

Houde Qian, Zheng Jiang, Wenyi Li et al.

LLMs can generate plausible trading code and features, but current models fail to produce strategies that generalize beyond their training period—a critical gap between research-phase performance and real-world deployment.

Quant-Arena is a benchmark that tests whether LLMs can complete the full quantitative trading research loop—from hypothesis to strategy—on real market data. Models develop trading strategies using 2022-2023 data, then their frozen strategies are evaluated on unseen 2024 data with realistic trading costs and market impact.

evaluationagentsapplications

Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

Oct 21, 2026

Yanzhe Chen, Ziyi Yang, Jifeng Zhu et al.

Building reliable robots requires evaluating all five capabilities—especially Anticipate and Verify, which are largely invisible in current benchmarks—not just task success.

This paper proposes PAPAV, a unified framework for understanding multimodal embodied agents (systems that combine AI reasoning with physical action) by breaking them into five core capabilities: Perceive, Anticipate, Plan, Act, and Verify.

agentsreasoningevaluation

Asymmetric On-Policy Distillation Learns In-Context Exploration

Oct 21, 2026

Sriyash Poddar, Yanda Bao, Jacob Krantz et al.

You can teach agents efficient exploration by having a privileged teacher label actions conditioned on the student's real exploration history—no RL needed, just supervised learning with asymmetric information access.

ASTEROID is a student-teacher learning framework that teaches agents to explore efficiently under partial observability without reinforcement learning. A privileged teacher with full state information labels actions based on the student's actual exploration history, allowing the student to learn when and how to gather information.

trainingagentsefficiency

AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation

Oct 21, 2026

Likui Zhang, Zhe Zhu, Zhihao Zhan et al.

Breaking long-horizon robotic tasks into atomic subtasks and using semantic-aware gating for credit assignment enables more efficient RL training of vision-language-action models, with practical real-world transfer via simulation.

AtomicRL improves vision-language-action (VLA) models for robot manipulation by using reinforcement learning with better credit assignment. It breaks long tasks into atomic subtasks, uses semantic-aware gating to prevent credit contamination between subtasks, and adaptively focuses learning on bottleneck skills.

agentsapplications

GBClaw: An agentic AI framework for atomistic grain boundary research and mechanistic discovery

Oct 20, 2026

Yazhuo Liu, Kunqing Ding, Xiaoguo Gong et al.

Agentic AI can tackle domain-specific materials science by combining LLMs with specialized simulation tools and memory systems, enabling discovery of mechanistic insights that require both high-throughput screening and deep physical reasoning.

GBClaw is an AI framework that combines large language models with specialized tools to automatically investigate grain boundaries in materials. The system orchestrates complex simulations and analysis to understand how grain boundaries deform, discovering that defect structures significantly affect material strength—work that would be difficult for humans to do manually.

agentsapplicationsreasoning

UniWALM: Unified World Action Language Model for Embodied Robot Control

Oct 20, 2026

Rahul Sajnani, Diego Romeres, Mingyuan Zhang et al.

Integrating language generation directly into robot control models—not as an afterthought but as a core part of the same network—enables robots to use language as working memory, significantly improving performance on tasks that require remembering past events.

UniWALM is a unified AI model for robot control that learns video, actions, and language together in one shared representation.

agentsmultimodalarchitecture

Industrial ReMEmbR: Scaling Spatiotemporal Memory to Robot Fleets

Oct 20, 2026

Jad Yammine, Malek Wahidi, Abrar Anwar et al.

Splitting large vision-language models at their architecture boundary—vision encoder on robots, language decoder on servers—enables efficient fleet-scale memory systems that generate descriptions on-demand rather than at write time, cutting server costs while preserving retrieval quality.

IndustrialRemembr extends a spatiotemporal memory system to robot fleets by splitting vision-language models between edge devices and servers. Robots encode video into embeddings and tokens without uploading raw frames, while an AI agent retrieves memories on demand and generates descriptions only when needed, reducing computation and storage costs while maintaining GDPR compliance.

efficiencyagentsmultimodal

Semantic Observability for Autonomous Systems Diagnostics

Oct 20, 2026

Jitendra Solanki, Anand Todkar, Ziran Min

By representing logs as a connected graph of infrastructure and events rather than flat text, autonomous agents can efficiently isolate failure causes and explain them in code-level detail—cutting through noise that overwhelms traditional log analysis.

This paper presents Semantic Observability, a multi-agent system that automatically diagnoses failures in distributed systems by converting unstructured logs into a queryable graph database, then using causal reasoning and LLMs to pinpoint root causes and suggest fixes. It reduces manual work for engineers investigating system outages.

agentsreasoningapplications