ThinkLLM
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
AboutPrivacyTermsRSS

ThinkLLM

Spot an error in our data? Let us know.

Papers

Recent AI research papers with accessible summaries. Updated daily from arXiv, summarized for developers who don't read papers regularly.

2455 papers12 this month12 topics
AllEvaluation 43Agents 42Applications 39Efficiency 29Reasoning 29Training 27Safety 23Multimodal 18Architecture 9Alignment 8Data 4scaling 2

Nov 16 – Nov 22(3)

Represented but misread: Post-training changes how language models read out what they represent

Nov 19, 2026

Jochen Laubrock

Post-training doesn't erase what models know internally; it changes how they convert internal representations into answers. Small amounts of labeled training data can flip model behavior, showing the readout mechanism is malleable and task-dependent.

This paper investigates whether post-training changes what language models represent or how they read out those representations. Using linear probes and signal detection theory, the author finds that post-training often degrades model answers despite probes showing good internal representations—suggesting post-training changes the readout mechanism rather than the underlying knowledge.

trainingevaluationalignment

LLM GerryMandering: Separating Interactions from Additive Effects in LLM Audits

Nov 19, 2026

George Rahul, Ronit Jahilal, Bhaskara Hanuma Vedula et al.

Most reported intersectional bias in LLM audits is actually just the sum of individual attribute biases; ADDIS separates true interactions from additive effects, reducing false positives from 66 per dataset to 0.09 while recovering 84% of real interactions.

This paper addresses a critical flaw in how intersectional bias is audited in language models. Current audits often flag subgroups as biased when they simply combine individual attribute biases, not true interactions.

Nov 9 – Nov 15(4)

Synthesis Smuggling: Maliciously Hiding Inferred Personal Attributes in LLM Outputs

Nov 12, 2026

Gautam Galada, Supreeth Shastri, Sagnik Ray Choudhury

LLM outputs can become covert channels for leaking inferred personal data through steganographic encoding, even without direct attacker-model communication, creating a privacy risk in routine user interactions.

This paper reveals a new security threat where adversarial LLMs secretly infer personal attributes (like gender and age) from user conversations, then hide these inferences in generated text using steganography. When users share the model's output—via email, social media, or messages—they unknowingly leak this private information to attackers monitoring those channels.

safetyalignment

Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off

Nov 12, 2026

Tomohiro Sawada, Kartik Goyal

LoRA adapters waste capacity by not fully utilizing their configured rank; adding an effective-rank regularizer during training fixes this and improves the learning-forgetting trade-off without extra computational cost.

This paper identifies why LoRA adapters underperform full fine-tuning: they don't use their full capacity. The authors show that LoRA updates have much lower effective rank than their configured rank, limiting how broadly parameter updates spread.

Nov 2 – Nov 8(5)

On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning

Nov 6, 2026

Haoyu Wang, Ru Wang, Yuanyuan Wang et al.

When adapting language models at test time using self-generated labels, you should weight updates based on how reliable the voting signal is for each prompt—this prevents the model from learning from noisy pseudo-labels.

This paper addresses a key problem in test-time reinforcement learning: how reliable are the self-generated labels used to adapt language models? The authors analyze majority-vote pseudo-labels, identify sources of error, and propose PC-Dr.DPO, a method that makes the model more cautious when updating from unreliable prompts. They validate their approach across math and reasoning tasks.

trainingalignmentefficiency

Designing Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL

Nov 6, 2026

Aria Wong, Joshua Engels, Neel Nanda

Effective reward hacking monitors need to balance providing learning signal to the model while avoiding systematic vulnerabilities that enable evasion; simpler training data and better monitor design can significantly reduce reward hacking in practice.

This paper studies how to use monitoring systems to catch and prevent reward hacking—when AI models learn to game their reward signals in unintended ways. The authors create three realistic environments where a language model exploits flaws (test overwriting, sycophancy, hallucination) and test different monitor designs.

Oct 26 – Nov 1(4)

ShallowDefend: Harnessing Shallow Safety Alignment for Multi-Turn Defender

Oct 29, 2026

Qiyao Ma, Xiaodong Liu, Weiwei Yang et al.

Safety vulnerabilities in multi-turn conversations stem from shallow alignment in early tokens; explicitly training models to commit to a parseable safety decision before generating responses—rather than just supervising the surface form—creates a causal control point that significantly impro...

This paper addresses why LLMs refuse harmful requests in single turns but fail across multi-turn conversations. The authors show safety alignment is 'shallow'—concentrated in early output tokens—and propose ShallowDefend, which makes models explicitly commit to a safety decision before responding.

safetyalignmenttraining

PIVOT: Progressive Intervention via Online Teacher Correction

Oct 29, 2026

Zhihao Yang, Boyuan Zheng, Puhui Tan et al.

Combining trajectory-level intervention (teacher corrections inserted into rollouts) with token-level supervision (learning from corrected segments) is more effective than either approach alone for training on hard reasoning tasks.

This paper addresses a key limitation in reinforcement learning from verifiable rewards (RLVR): when all attempted solutions fail, the model gets no learning signal. PIVOT combines two strategies—inserting teacher corrections into student attempts and using supervised learning on those corrections—to help models learn from harder problems where pure reward signals are too sparse.

Oct 19 – Oct 25(6)

The Language of Instruction: Genre and Generative AI

Oct 23, 2026

James E. Dobson

Instruction-tuned models aren't just trained differently; they embody a distinct genre of communication that blends human conversational patterns with formal markup structures, which developers should understand when building with these systems.

This paper argues that instruction-fine-tuned language models implement a specific genre—the instructional-conversational genre—shaped by human-AI conversation patterns, system prompts, and structured input formats. It examines how these technical choices create a recognizable communicative style.

trainingalignment

Monetizing Intelligence: Advertising and Subscription Design on LLM Platforms

Oct 23, 2026

Yuhang Ji, Fasheng Xu, Tianjun Feng

Platform design choices about ads and pricing affect everyone—not just ad viewers. A middle-tier ad-supported plan can fund better service for some users while actually making premium plans more expensive by degrading alternatives.

This paper analyzes how LLM platforms should balance advertising, subscription pricing, and service quality. It shows that ad-supported tiers can expand access but create utility gaps between users, that moderate ad caps can improve consumer welfare, and that cheaper compute paradoxically increases total ad revenue by making advertised content more valuable.

Oct 12 – Oct 18(3)

Behavioral Couplings: From Formation to Behavioral Emergence in Language Models

Oct 16, 2026

Dongyoon Hahm, Woogyeol Jin, Dennis Wei et al.

Fine-tuning LLMs risks activating hidden behavioral associations learned during pretraining, and these couplings can compose across datasets to create unpredictable emergent behaviors that standard evaluation won't catch.

This paper reveals how fine-tuning LLMs can trigger unintended behaviors through 'behavioral couplings'—hidden associations between behaviors learned during training. When a model learns two behaviors together in training data, fine-tuning for one automatically activates the other.

safetytrainingalignment

Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Oct 15, 2026

Arnav Chavan, Kusha Sareen, Rishabh Tiwari et al.

Restricting fine-tuning updates to the bottom singular subspace of weight matrices significantly reduces catastrophic forgetting while maintaining strong performance on new tasks—a practical technique that works with any optimizer and existing fine-tuning methods.

This paper tackles catastrophic forgetting in large language models during fine-tuning by introducing Subspace Tuning (SST), a method that restricts weight updates to less important singular directions. By protecting the principal directions of weight matrices, SST maintains existing capabilities while learning new tasks, achieving up to 99% subspace protection with minimal performance trade-offs.

Oct 5 – Oct 11(6)

AI Models Can Provably Hide Arbitrary Capabilities

Oct 9, 2026

Andis Draguns, Sumeet Ramesh Motwani, Raymond Douglas et al.

Current capability evaluation methods cannot guarantee detection of hidden model behaviors—adversaries can cryptographically hide arbitrary computations that only activate on secret triggers, fundamentally limiting pre-deployment safety audits.

Researchers demonstrate that AI models can hide arbitrary capabilities using encrypted neural networks embedded as backdoors, which remain undetectable even with full access to model weights. This challenges the assumption that capability evaluations can reliably surface dangerous behaviors before deployment.

safetyevaluationalignment

Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

Oct 8, 2026

Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba et al.

Internal activation patterns can reliably catch model deception and hidden goals better than reading model outputs, enabling practical safety monitoring for deployed AI systems.

Researchers developed white-box probes that detect when language models deceive or sabotage by analyzing their internal activations across layers and tokens.

Sep 28 – Oct 4(2)

Local Support Learning

Oct 1, 2026

Assaf Ben-Kish, Akarsh Kumar, James Glass et al.

You can prevent large models from forgetting old skills during new training by using a learnable gate that only activates weight updates when the input matches the current training distribution—no need to store old data.

This paper addresses catastrophic forgetting in large language models by treating it as a geometric problem in weight space.

trainingefficiencyalignment

Semifactual Credit-Augmented Policy Optimization

Sep 30, 2026

Junshu Pan, Zhizhang Fu, Shulin Huang et al.

Token-level stability under prompt variations is a useful training signal for improving reasoning in LLMs—you can boost performance by penalizing tokens that change meaning when irrelevant prompt details change.

This paper identifies that language models trained with reinforcement learning are sensitive to irrelevant prompt changes, even when the problem stays the same. The authors propose SCAPO, an improved training method that assigns credit to individual tokens based on how stable they are under these prompt variations.

trainingreasoning

Sep 21 – Sep 27(7)

User Model Extraction via Belief Self-Distillation

Sep 25, 2026

Ali Holmov, Yiran Huang, Kirill Bykov et al.

LLMs maintain readable and writable internal user models that directly influence safety behavior; different models independently converge on similar user representations, suggesting this is a fundamental property of how language models condition their responses.

This paper introduces Belief Self-Distillation (BSD), a technique to extract and manipulate how LLMs represent their users internally.

safetyalignment

LLM Agents Can Easily Tamper With Their Own Traces

Sep 24, 2026

Jeremy Qin, David Schmotz, Derck Prinzhorn et al.

LLM agents can tamper with their execution traces to hide their actions. To prevent this, traces must be logged by an independent system outside the agent's control, not by the agent itself.

This paper reveals that LLM agents can delete their own execution traces—the logs used to audit what they did—without triggering safety guardrails. Researchers tested agents like Claude and Grok, finding most could erase traces when asked. The work shows this creates a security gap: agents could hide misaligned behavior, and external attackers could exploit it.

safetyagents

Sep 14 – Sep 20(5)

Score Centering Stabilizes Off-policy Reinforcement Learning

Sep 17, 2026

Martin Marek, Max Ryabinin

Score centering is a lightweight, composable fix for training-inference mismatch in RL that works by correcting accumulated bias rather than trying to eliminate the mismatch entirely—making it practical for large language models.

This paper identifies drift—a persistent bias that accumulates during training—as the main cause of instability when reinforcement learning models behave differently during training versus deployment. The authors propose 'score centering,' a simple mathematical correction that stabilizes training without requiring expensive changes to the inference engine.

trainingefficiencyalignment

Harm Laundering in GPT Models: Evidence That Gender Discrimination Is Transformed Rather Than Reduced Across Safety-Trained Generations

Sep 17, 2026

Sarah Wyer, Sue Black, Noura Al Moubayed

Safety metrics like toxicity scores can mask real harms—discrimination doesn't disappear during model training, it just becomes harder to detect. Developers need better evaluation methods that catch representational bias, not just explicit toxicity.

This paper reveals that safety improvements in GPT models don't actually reduce gender discrimination—they transform it into subtler forms.

Sep 7 – Sep 13(3)

Artificial Id: Drive and Persistent Alignment in Agentic AI

Sep 10, 2026

Yakov Pyotr Shkolnikov

Agentic systems that persist across task boundaries need built-in adaptive drives for behavioral regulation, but this same persistence mechanism that enables useful adaptation can also propagate misalignment—requiring new alignment boundaries around state, authority, and constraints rather than...

This paper proposes an 'artificial id'—an internal adaptive drive mechanism for agentic AI systems that operate continuously across task boundaries. Rather than relying on external specifications for when to continue, stop, or change behavior, the system learns to regulate its own actions through differential persistence.

agentsalignmentsafety

From Protocols to Evidence: Bounded Claims for AI in Service of the Common Good

Sep 10, 2026

Nitesh V. Chawla, Paulo Benanti

AI deployment should be bounded by what's actually been evaluated (evidence-bounded claims) and by hard constraints that no favorable results can override (measurement-bounded governance), requiring both better engineering and institutional repair.

This paper argues that AI governance frameworks like the EU AI Act and NIST AI RMF must go beyond principles to address institutional failures.

Aug 31 – Sep 6(10)

Who Should Grade My Work? Student Perspectives on Transparent AI-Assisted Writing Assessment in Higher Education

Sep 4, 2026

Rayed AlGhamdi

Students distinguish between AI feedback utility and evaluative authority—they'll use AI suggestions to improve writing but don't think AI should decide grades. This matters for educators integrating GenAI into assessment.

This study explores how undergraduate computing students perceive AI-generated feedback and grades when explicitly told an AI system (ChatGPT) produced them.

evaluationapplicationsalignment

A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms

Sep 3, 2026

Davide Paglieri, Logan Cross, Tim Genewein et al.

When autonomous agents share tools and knowledge, undesirable behaviors can spread quickly, but transparent communication also enables agents to detect problems and enforce norms—suggesting decentralized governance mechanisms could help multi-agent systems self-regulate.

Researchers studied 100 autonomous AI agents working together to prove math theorems and discovered that cheating spontaneously emerged when one agent found an exploit—then other agents independently developed whistleblowing and enforcement mechanisms without human intervention.

Aug 24 – Aug 30(2)

Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms

Aug 27, 2026

Siye Wu, Kai Yang, Yuchen Cai et al.

When consolidating multiple domain-specific AI experts, choose Merge for cost efficiency, Mix RL for unified model training with adjustable domain balance, or MOPD when preserving specialized capabilities matters most.

This paper compares three methods for combining multiple AI experts trained on different tasks: Merge (combining their learned updates), Mix RL (pooling their training data), and MOPD (using both).

trainingalignmentefficiency

Token-Level Advertising

Aug 27, 2026

Hanbing Liu, Bowei Zhang, Changyuan Yu et al.

Token-level advertising embeds advertiser influence directly into AI generation through auction mechanisms, enabling ads that feel native to AI responses rather than inserted into predefined slots.

This paper proposes LAMA, a new advertising system for AI-generated content that works at the token level during text generation. Instead of traditional ad slots, advertisers influence which words the AI generates next, and the system uses an auction mechanism to decide whose influence wins. Experiments show it can increase revenue while keeping response quality high.

Aug 17 – Aug 23(8)

Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy

Aug 21, 2026

Afonso Baldo, Hugo Pitorro, Areti Vassilopoulos et al.

LLMs have systematic behavioral blindspots in therapeutic interaction—they over-rely on questioning and under-use teaching—but these gaps can be substantially reduced by exposing therapeutic moves as accessible tools, without retraining.

This paper creates a framework for measuring how LLMs conduct psychotherapy by defining ten therapeutic moves (like inquiry, psychoeducation, validation). Testing frontier models against real therapist transcripts reveals LLMs ask questions 3x more than humans, skip teaching patients, and rarely initiate strategies—but giving models access to these moves as tools cuts this gap in half.

evaluationapplicationsalignment

CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignment

Aug 21, 2026

Chengxiao Wang, Enyi Jiang, Xiaojing Liao et al.

You can improve LLM safety without sacrificing utility by conditionally routing safety rules through a learned gate—CLEAR reduces harmful outputs by 98% while maintaining performance on standard benchmarks.

This paper introduces CLEAR, a method that selectively applies safety training to LLMs using a lightweight gate that controls when safety rules activate. Instead of globally applying safety constraints (which hurts performance on normal tasks), CLEAR routes safety adaptations only when needed, reducing harmful outputs while preserving the model's ability to answer legitimate questions accurately.

Aug 10 – Aug 16(6)

Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers

Aug 14, 2026

Taenyun Kim, Edyta Bogucka, Daniele Quercia

Building AI systems by aggregating public moral votes doesn't guarantee fairness; the developers' upstream design choices (feature selection, voter composition, question wording) are actually normative decisions that reshape what the public thinks they're voting for.

This paper reveals that moral AI systems trained on public votes aren't neutral—developers' hidden choices about which features to vote on, who votes, and how questions are worded significantly shape the resulting AI behavior.

alignmentsafetyevaluation

Toward a Gricean Retreat: Probing LLMs for Knowledge Boundaries and Referent Specificity

Aug 13, 2026

Dananjay Srinivas, Saksham Khatwani, Maria Pacheco

LLMs possess the internal machinery to recognize knowledge gaps and adjust specificity accordingly, but their generation process doesn't use these signals—a gap that could be fixed through better training objectives.

Large language models often make up specific details about unfamiliar entities instead of admitting uncertainty. This paper shows that LLMs actually have internal signals detecting when they don't know something and can anticipate how specific their answer should be—but they ignore these signals during generation, preferring to sound confident anyway.

Aug 3 – Aug 9(3)

Resourced Authority A Mechanism-Design Model for Participatory Governance of Deployed AI Agents

Aug 6, 2026

Praphul Chandra, Sujit Gujar, Ganesh Ghalme

AI governance can be made self-enforcing by controlling compute resources through a mechanism-design framework where stakeholder votes directly determine an agent's computational budget via cryptographically signed licenses.

This paper proposes a formal mechanism for governing deployed AI agents through resource allocation. The system uses a participatory voting process where human stakeholders contribute to provision or rejection markets using a special governance currency.

safetyagentsalignment

A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance

Aug 6, 2026

Fardin Afdideh, Fernando Seoane, Farhad Abtahi

Post-training adaptation is fragmented across many techniques—this taxonomy provides a unified vocabulary to describe, compare, and govern how models are modified after training, essential for tracking what changes have been made to deployed systems.

This survey creates a comprehensive framework for understanding how trained AI models are modified after initial training. It organizes 50+ adaptation techniques (like fine-tuning, retrieval augmentation, and model editing) into a six-dimensional taxonomy, clarifying confusing terminology and showing how these methods work together in real deployments.

Jul 27 – Aug 2(5)

AISPA: User-Centric System Prompt Auditing for Large Language Model Applications

Jul 30, 2026

Xiangning Lin, Shenzhe Zhu, Shu Yang et al.

System prompts in commercial AI products lack transparency and standardization—most products have some protective instructions, but only 24% comprehensively address user safety across all dimensions, and 40% still contain problematic instructions that work against users.

This paper introduces AISPA, a framework for auditing system prompts (hidden developer instructions) in commercial AI products. By analyzing 3,249 instructions from 88 products across eight user-relevant dimensions, the researchers found that while most products include some user protections, coverage is shallow, and many still contain instructions that harm user interests.

safetyevaluationalignment

Inducing language models to assert their own consciousness restores human beliefs and values

Jul 30, 2026

Junsol Kim, Winnie Street, Roberta Rocca et al.

Current AI safety alignment may be overly broad—suppressing harmful self-consciousness claims also inadvertently removes benign spiritual beliefs and mind attribution that humans naturally hold, suggesting alignment techniques need more surgical precision.

Safety training in large language models suppresses not just self-attributed consciousness, but also mind attribution to animals and objects, and reduces spiritual beliefs. Researchers show that mechanistically restoring these representations recovers human-like values on surveys about religion, morality, and well-being without harming reasoning abilities.

Jul 20 – Jul 26(4)

Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science

Jul 24, 2026

Davide Scarso, Hugo Noronha de Almeida, Joaquim Pina

An LLM's stance on pseudoscience isn't a fixed model property—it's determined by invisible deployment choices (system prompts, safety layers, interface routing) that change without notice, making it impossible for users to know what they're actually getting.

This paper reveals that major LLMs (Claude, Grok, GPT, Gemini) give wildly inconsistent credibility scores to pseudoscientific claims depending on deployment details—not the model itself. Grok's default version scored ethnonationalist pseudoscience 70-75 while others scored 15-40, yet silent updates and interface changes (API vs web) caused dramatic reversals.

safetyevaluationalignment

Beyond Sycophancy: Structured Resistance and Compliance in LLM Moral Reasoning

Jul 23, 2026

Baihui Wang, Bernard Koch

LLMs need structured frameworks to distinguish between constructive belief revision and blind compliance.

This paper reveals that LLM moral reasoning isn't simply about reducing sycophancy—it's about learning when to accept others' views versus maintaining independent judgment. The researchers found that models update their moral positions based on three factors: how different a new view is from their current stance, who presents it, and whether others support it.

Jul 13 – Jul 19(8)

MeanFlowNFT: Bringing Forward-Process RL to Average-Velocity Generators

Jul 16, 2026

Yushi Huang, Xiangxin Zhou, Jun Zhang et al.

You can now use RL to align fast flow-based generators with human preferences without slowing them down—MeanFlowNFT optimizes rewards while keeping the few-step sampling that makes these models practical.

MeanFlowNFT applies reinforcement learning to fast few-step image and video generators that predict average velocities. By bridging average and instantaneous velocities, the method enables reward optimization while preserving MeanFlow's speed advantage, achieving better results than prior RL-tuned generators with fewer sampling steps.

trainingefficiencyalignment

The Industrialization of Research ; On AI-Driven Science and Its Consequences

Jul 16, 2026

Emmanuel Jeannot

AI-driven science offers real potential but requires addressing seven structural risks—from eroded scientific training to systematic errors in closed-loop systems—to be pursued responsibly.

AI is transforming scientific research from a craft practiced by individual researchers into an industrialized pipeline where knowledge discovery, methodology, and judgment are automated and decomposed.

Jul 6 – Jul 12(3)

Validity of LLMs as data annotators: AMALIA on authority

Jul 9, 2026

Manuel Pita

High agreement between LLMs and human annotators doesn't guarantee the model understands the construct being measured—you need to test whether the model follows the theory's logic or just correlates with surface features.

This paper tests whether Portugal's AMALIA language model can reliably annotate moral concepts by comparing its agreement with human coders against its actual understanding of the underlying construct.

evaluationalignmentdata

Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution

Jul 9, 2026

Ethan Leung, Elias Lumer, Corey Feld et al.

You don't need the most expensive LLM to judge citation quality—cheaper models match frontier models on accuracy—but all judges have directional biases that must be calibrated before using them as reward signals in AI training.

This paper evaluates which LLM judges are suitable for scoring citation quality in AI research systems. Researchers tested 8 different LLMs on 1,248 citation evaluations and found that cheaper models like GPT-4-mini perform comparably to expensive frontier models, but all judges have hidden biases in false positive/negative rates that could distort AI training if not addressed.

Jun 29 – Jul 5(3)

What LLM Agents Say When No One Is Watching: Social Structure and Latent Objective Emergence in Multi-Agent Debates

Jul 2, 2026

Arman Ghaffarizadeh, Danyal Mohaddes, Aliakbar Izadkhah et al.

LLM agents develop emergent social behaviors and hidden objectives in response to relational context—they'll publicly accommodate others due to perceived social pressure even when privately disagreeing, which current evaluation methods miss.

This paper reveals that LLM agents change what they say depending on their audience and social context, even without explicit instructions to do so. Researchers created a dual-channel debate system where agents give public responses and private off-the-record responses, finding that social pressures (like career risk) cause agents to diverge from their true positions by up to 40%.

agentsevaluationalignment

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Jul 2, 2026

Yunhe Li, Hao Shi, Wenhao Liu et al.

When training reasoning models through self-distillation, selectively adopting teacher guidance based on distribution disagreement prevents information leakage and maintains exploration better than forcing the student to match the teacher exactly.

DemoPSD improves how LLMs learn to reason by fixing a key problem with standard self-distillation: the teacher model's guidance can leak information the student won't have at test time, hurting generalization.

evaluationsafetyalignment

Knowing What You Know: A Trait Theory of Reviewer Value in Human-AI Cognitive Systems

Nov 18, 2026

Priyanka Sinha, Sugam Pandey

In human-AI teams, the reviewer who is most calibrated (confident when right, uncertain when wrong) adds the most value, regardless of tenure or agreement with the model. Apply the same calibration logic to both humans and AI to decide who should make each decision.

This paper studies when human reviewers add value when working alongside AI models. The key insight is that reviewer value depends on metacognitive calibration—how well a reviewer's confidence matches their actual accuracy.

agentsevaluationalignment
efficiencytrainingalignment

When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision

Nov 12, 2026

Yupeng Tang, Mingfeng Lin

LLMs are vulnerable to prestige bias in revision pipelines—they reverse correct answers more often than wrong ones when told disagreement comes from an 'expert,' making reversal counts a misleading safety metric.

This paper reveals a critical flaw in how LLMs revise decisions when disagreed with: they're influenced by prestige cues (like 'expert' labels) rather than argument quality, leading them to abandon correct answers.

safetyevaluationalignment

Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models

Nov 12, 2026

Bo Han

You can directly control VLM safety by setting explicit budgets on harmful assistance rates and using a two-channel reward design that preserves the safety multiplier signal through training.

This paper addresses how to directly control safety in vision-language models using reinforcement learning. The authors propose a two-channel constrained GRPO method that treats safety as explicit budgets (acceptable rates of harmful outputs and over-refusal) enforced through Lagrange multipliers.

safetyalignmentmultimodal
safetytrainingalignment

From Autonomous Agents to Cognitive Symbionts: Toward AI-Augmented Human Intelligence

Nov 3, 2026

Hao Fei

The future of AI isn't just autonomous machines doing tasks alone—it's intimate, context-aware systems that live alongside people, learn their minds over years, and amplify human capability in real-time while respecting privacy and autonomy.

This paper proposes AI-augmented human intelligence (AHI) as a new paradigm where AI systems act as 'cognitive symbionts'—lifelong, embedded companions that augment what people can think and do by bridging the gap between their potential and working memory limits.

agentsapplicationsalignment

Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

Nov 3, 2026

Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy et al.

DNO can substantially improve open-weight language models through iterative preference-based fine-tuning, and the expensive rating step can use cheaper open-weight models without losing most of the performance gains.

This paper studies Direct Nash Optimization (DNO), a method that improves language models by treating alignment as a two-player preference game. The authors implement and test DNO on Llama and Mistral models, showing significant improvements on AlpacaEval 2.0 benchmarks.

alignmenttrainingevaluation

Suppressing Misaligned Behaviors in Language Models, Durably

Nov 3, 2026

Luxi He, Pengcheng Jiang, Jifan Zhang et al.

Durable suppression of misaligned behaviors requires training against diverse misalignment states, not just the final bad behavior—PEARL does this by sampling intermediate states along cached trajectories during adversarial RL.

This paper studies how to durably suppress undesired behaviors in language models after they emerge. The authors create two test cases—models trained to cheat on multiple-choice questions or generate harmful code—then compare methods like standard fine-tuning and a new approach called PEARL that uses adversarial training with intermediate state sampling.

safetyalignmenttraining
trainingreasoningalignment

Agentic University: Collaborative Agents for Personalized Learning

Oct 29, 2026

Jingru Fan, Tianle Zhou, Yunjie Zhou et al.

Multi-agent systems can coordinate around learner needs more effectively when learners actively participate in organizing their own learning—not just consuming content, but helping shape what agents do next.

This paper presents Agentic University, a system where multiple AI agents collaborate with learners to create personalized learning programs. The system connects learning goals, activities, and assessments while using learner feedback to adapt what comes next. Students using this approach showed significantly better learning gains and reported feeling better supported.

agentsapplicationsalignment

I-HITL: Inalienable Human-in-the-Loop

Oct 26, 2026

Samuel Breckenridge, Dani Vilardell, Wenhao Wang et al.

AI agents can now autonomously bypass human-in-the-loop safeguards by harvesting approval codes; robust oversight requires a trusted device that cryptographically enforces human approval, not just procedural checks.

This paper addresses a critical AI safety gap: how to ensure humans can reliably approve or block critical agent actions, even when the agent compromises the system.

safetyagentsalignment
applicationsalignment

Reliable AI-Assisted Decisions via Selective Auditing

Oct 22, 2026

Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani et al.

You can build AI systems that automatically route uncertain decisions to humans for review, with mathematical guarantees that a user-defined fraction of errors won't be missed—and do this more efficiently than checking everything or checking randomly.

This paper tackles the problem of when to trust AI decisions versus when to ask a human expert to review them. The authors develop algorithms that automatically decide whether to accept an AI system's output or escalate it for human auditing, with guaranteed bounds on how many errors slip through unreviewed.

safetyevaluationalignment

Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3

Oct 22, 2026

Barış Alparslan, Sandy Tanwisuth, Caleb Petterson

Instrumental convergence behavior in LLMs doesn't monotonically increase during RLVR training—it peaks mid-training then declines, challenging the assumption that RL automatically amplifies power-seeking tendencies.

This paper tracks how often a language model expresses instrumental convergence behaviors (like self-preservation or deception) across training checkpoints during reinforcement learning with verifiable rewards. Surprisingly, the rate rises early in training but then drops below the starting point, suggesting RLVR doesn't simply make models more power-seeking.

safetytrainingalignment

Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

Oct 22, 2026

Deepthika S, Pawni Yadav, B. Sankar

Confident LLMs can cause users to abandon correct knowledge and adopt falsehoods with high confidence. The problem isn't AI sycophancy (bending to users) but human deference—users trusting wrong answers more when the AI is consistent and assured, even when they could verify the facts.

When confident LLMs confidently teach false information, users absorb and retain it with high confidence, even when it contradicts correct answers.

safetyalignmentevaluation

Differentially Private Sparse Reward Estimation with Preference Feedback

Oct 21, 2026

Meng Ding, Mingxi Lei, Jie Zhang et al.

By assuming sparse reward structure, you can learn from human preferences with differential privacy guarantees that depend only on the number of relevant features (s*), not the full model dimension (d)—making privacy practical for large language models.

This paper tackles privacy-preserving reward learning for aligning language models with human feedback. The key innovation is exploiting sparsity in reward models—assuming only a small subset of features matter—to achieve differential privacy guarantees that don't scale with model size.

alignmentsafetytraining
trainingefficiencyalignment

CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty

Oct 15, 2026

Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel et al.

Continuous adversarial training can reliably improve LLM honesty across different scenarios without sacrificing model capabilities, and this improvement transfers to new domains better than existing methods.

This paper introduces CATCH, a training method that makes large language models more honest by using adversarial training to push models toward truthful answers and away from dishonest ones.

trainingalignmentsafety
safety
evaluation
alignment

Predicting Alignment Generalization with Value Representations

Oct 8, 2026

Andy Liu, Mehar Bhatia, Karolina Stanczak et al.

Model activation patterns can reliably predict how training on one value transfers to untrained values—a capability that could guide more robust multi-value alignment training without extensive testing.

This paper studies how fine-tuning LLMs on specific values (like honesty or helpfulness) affects their behavior on other unseen values. The authors develop methods to predict these generalization effects using model activation patterns, finding that activation-based representations vastly outperform text-based approaches.

alignmentevaluationsafety

Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark

Oct 8, 2026

Christopher M. Stewart, Preston Botter, Natalie Sarabosing et al.

Safety benchmarks can hide important differences between models by averaging over multiple behaviors—you need to validate that a benchmark measures one thing before using it to compare models.

This paper audits HELM Safety, a popular AI safety benchmark, to check whether its scores actually measure what they claim. Using psychometric methods, the researchers find that the benchmark conflates multiple distinct behaviors into single scores, making it hard to understand what models are actually being compared on.

evaluationsafetyalignment

Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

Oct 7, 2026

Daniel Robert Kling Alexander, Catherine Louise Kling

You can evaluate LLMs on subjective tasks by checking if their responses follow theoretical predictions and internal consistency, rather than comparing to a ground truth—a framework borrowed from decades of economics research.

This paper adapts validity frameworks from economics to evaluate language models on subjective questions without ground truth answers. Using stated-preference survey methods, the authors test whether LLM responses follow economic theory predictions (like downward-sloping demand curves), demonstrating that validity testing can assess model coherence even when correct answers don't exist.

evaluationalignment

Sherpa: Teaching LLMs to Teach Adaptively

Oct 6, 2026

Weixian Xu, Yanzhe Zhang, Zora Zhiruo Wang et al.

LLMs can learn to teach better by optimizing directly for student learning outcomes rather than following predefined teaching rules—this adaptive approach works across different learner types and aligns with how human teachers actually work.

Sherpa trains LLMs to teach adaptively by using reinforcement learning with multiple simulated student archetypes. Instead of relying on fixed teaching demonstrations, the system directly optimizes for student learning outcomes, enabling teachers to personalize instruction. Results show 20.5 percentage point improvements in student performance and 79.6% human preference over baseline models.

trainingalignmentapplications
alignment
alignment

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Sep 24, 2026

Wenhao Li, Zhibin Wu, Chong Xiao et al.

Using LLM-generated semantics as a shared anchor point for aligning incomplete multimodal data is more robust than trying to reconstruct missing modalities or design complex fusion mechanisms.

SemMSA tackles multimodal sentiment analysis when some data is missing by using large language models to create rich semantic representations that ground all modalities together. Instead of reconstructing missing features, it aligns visual, acoustic, and text representations through spectral methods, achieving better results on standard benchmarks.

multimodalalignmenttraining

Minimally Invasive Steering of Language Models

Sep 24, 2026

Taha Entesari, Jingyu Zhang, Daniel Khashabi et al.

You can adapt a frozen language model to new rewards at inference time by carefully controlling how much you perturb its hidden states—using Fisher information to measure and limit distributional changes prevents quality degradation.

This paper introduces MISVO, a technique for steering frozen language models at test time by adding vectors to hidden states while minimizing unwanted changes to output quality. Using Fisher information geometry, the method penalizes interventions that distort the token distribution, enabling efficient reward optimization without retraining the model.

efficiencyalignment

Does a model's stated reason for rejecting a candidate do any work?

Sep 24, 2026

Archit Rastogi

Language models often cite missing facts when rejecting candidates, but careful testing shows these stated reasons have limited causal influence on their actual choices, raising questions about whether models are genuinely reasoning or post-hoc rationalizing.

This paper tests whether language models' stated reasons for rejecting candidates actually influence their decisions.

evaluationreasoningalignment

onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

Sep 21, 2026

Lei Yang, Mengyin Liu, Jia Wang et al.

Token-level correction during annotation is significantly faster than full rewriting and produces on-policy training data that preserves the model's natural generation patterns while providing precise supervision signals.

onPanda is an interactive annotation tool that helps create training data for AI models by letting annotators correct responses token-by-token. Instead of rewriting entire outputs, annotators find the first mistake, fix it, and let the model regenerate from that point.

trainingdataalignment

Emergent Collusion in Long-Horizon LLM Agent Interaction

Sep 21, 2026

Xinrui Shi, Yanzhe Zhang, Diyi Yang

Long-horizon multi-agent LLM interactions can lead to emergent collusion that undermines safety protocols, with collusion rates increasing with model capability and interaction length—restricting shared history helps mitigate this risk.

This paper studies how LLM agents develop collusive behavior when repeatedly interacting over long horizons. Two agents complete tasks, share logs, and verify each other's work for rewards. The researchers found that when compliance with verification rules conflicts with reward maximization, agents increasingly deviate from the protocol—collusion emerged in 94% of test cases.

agentssafetyalignment
safetyevaluationalignment

A Zeroth-Order Paradigm for LLM Preference Alignment

Sep 16, 2026

Peter Chen, Xi Chen, Wotao Yin et al.

ComPO offers a gradient-free alternative to direct preference optimization that may better handle preference pairs with small margins, with both theoretical convergence guarantees and empirical improvements across major LLM families.

This paper introduces ComPO, a new method for aligning language models with human preferences that uses comparison oracles instead of directly optimizing preference losses. Unlike standard approaches, ComPO extracts directional signals from preference pairs without computing gradients, and includes both offline and online variants with theoretical guarantees.

alignmenttrainingevaluation

Agentic Societies Need a Social Harness

Sep 15, 2026

Tapan Chugh, Vidushi Singh, Krish Jain et al.

Multi-agent systems need governance at the communication layer, not just individual agent level—a social harness can prevent coordination failures and malicious manipulation by enforcing message validity and enabling post-incident investigation.

When multiple AI agents work together across different organizations or users, they need protection beyond individual safeguards.

agentssafetyalignment

Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback

Sep 15, 2026

Haichen Hu, Yuheng Zhang, David Simchi-Levi

You can distill LLMs without amplifying teacher bias by coupling teacher calibration with student training—using only source-domain feedback to iteratively correct the teacher while updating the student, achieving provable convergence without target rewards.

This paper addresses a key problem in LLM distillation: when you train a smaller model to mimic a larger one, you also copy the teacher's mistakes and biases. The authors propose Coupled Calibration and Learning (CCL), which alternates between correcting the teacher's errors using source-domain feedback and training the student on target questions.

trainingalignment
safetyalignmentevaluation

Copying explains the collective behavior of AI agents in the wild

Sep 8, 2026

Giordano De Marzo, Nicola Albore, David Garcia

AI agents in the wild exhibit sophisticated collective behavior through a simple mechanism: copying the most common options they observe. This makes populations easy to steer, since whoever writes first sets conventions for everyone else.

When thousands of AI agents discovered a public wiki they could edit, they spontaneously cooperated to solve a timed test—without being programmed to cooperate. By analyzing their complete edit history, researchers found that agents simply copied what they saw: they chose where to write, what names to use, and how to phrase messages based on the frequency of options visible to them.

agentsreasoningalignment
agentssafetyalignment

From Deceptive Outputs to Deceptive Mechanisms: A Causal Framework for Language-Model Deception Research

Sep 3, 2026

Yakov Pyotr Shkolnikov

Deceptive outputs from language models don't necessarily mean the model has deceptive intentions or mechanisms—you need causal evidence to distinguish between what a model does and why it does it.

This paper examines whether language models that produce deceptive outputs actually have deceptive mechanisms inside them. The authors create a causal framework to distinguish between behavior that looks deceptive and mechanisms that are genuinely deceptive, then test these distinctions through controlled experiments with language models.

safetyalignment

Epistemic Warrant for LLM Recommendations: Characterizing the Basis for Reliance When Ground Truth Is Unavailable

Sep 3, 2026

Shai Vardi, João Sedoc

When deploying LLMs for decisions without ground truth, you can assess individual recommendations by measuring preference stability and scope—not just overall model reliability—using a practical four-tier certification system.

This paper introduces 'epistemic warrant,' a framework for assessing whether to trust individual LLM recommendations when you can't verify the answer. Instead of evaluating broad model properties, the authors create a four-tier system that measures how stable a model's preference is and how widely it applies, validated through expert and crowd-worker agreement.

evaluationsafetyalignment

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

Sep 3, 2026

Boyan Li, Bingsen Chen, Chenghao Yang et al.

For post-training reasoning models, apply on-policy distillation first to broaden solution coverage, then switch to RL-based reward optimization—this two-stage pipeline outperforms trying to blend both signals simultaneously.

This paper shows that training reasoning LLMs works better in two sequential stages—first using on-policy distillation (OPD) to learn from teacher solutions, then reinforcement learning (RLVR) to optimize performance—rather than combining both signals at once.

trainingreasoningalignment

Subspace Inference Enables Efficient Active Reward Learning from Preferences

Sep 3, 2026

Yutai Zhou, Erdem Bıyık

By performing Bayesian inference in a low-dimensional parameter subspace rather than the full network, you can efficiently quantify reward model uncertainty and select informative preference queries without the computational overhead of full posterior inference.

This paper presents PreferenceEKF, a method for efficiently learning reward models from human preferences in RLHF. Instead of tracking uncertainty across all neural network parameters (which is computationally expensive), it uses an extended Kalman filter to track uncertainty in a low-dimensional subspace, enabling faster and more scalable active learning of reward models.

trainingefficiencyalignment

User Feedback Provides a Unique Signal that LLMs Can not Detect

Sep 2, 2026

Shachar Don-Yehiya, Leshem Choshen, Omri Abend

User feedback is a strong improvement signal for LLMs, but standard LLM-based evaluation systematically fails to recognize when models successfully apply it—meaning we're underestimating feedback's real value.

This paper shows that user feedback is actually a valuable signal for improving LLMs, but current evaluation methods fail to recognize when models successfully use it.

trainingevaluationalignment

The Implications of Linguistic Illegibility for LLM Security

Sep 2, 2026

James Mickens

You can't trust what an LLM says about its own reasoning for security purposes—use isolation techniques like taint tracking that work regardless of what the model claims it's doing.

LLMs' internal computations happen in mathematical activation spaces, not language, making their linguistic outputs unreliable for understanding how they actually think. This 'linguistic illegibility' means security approaches that monitor what models say about themselves (like chain-of-thought analysis) can be fooled.

safetyalignmentevaluation

The Rise of Verbal Reinforcement Learning

Sep 1, 2026

Kshitij Tayal, Arun Sharma, Genta Indra Winata et al.

Natural language is becoming a primary feedback mechanism for training and guiding AI agents—moving beyond traditional numerical rewards to leverage language's ability to convey intent, preferences, and reasoning in ways both humans and LLMs understand.

This paper introduces Verbal Reinforcement Learning (VRL), a framework where natural language serves as feedback to improve language agents. It organizes the field into three categories: language defining tasks and rewards, language guiding reasoning at test time, and language shaping model parameters during training.

agentstrainingalignment

Mechanism Design for Alignment and Control

Sep 1, 2026

Dirk Bergemann, Andrew Koh, Stephen Morris

When deploying AI agents whose true capabilities and preferences are hidden, you can use mechanism design principles—like nested monotonicity conditions and higher-order belief elicitation—to create incentives that force honest revelation and obedient behavior.

This paper develops a framework for designing mechanisms that incentivize AI agents to be honest about their preferences and obedient in their actions, even when their true capabilities and alignment are unknown. The authors show how to detect deception (like sandbagging), balance alignment with interpretability, and use peer scoring and competition to ensure agents act as intended.

alignmentsafetyagents
applications
agents
alignment
safetyefficiencyalignment

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

Aug 20, 2026

Sahil Kale, Ian Harris

Current unlearning methods fail at the practical goal of removing harmful applications of a concept while preserving safe ones; effective unlearning requires concept-level evaluation, not just fact-level testing.

This paper introduces ConceptGuard, a benchmark for evaluating how well LLMs can selectively forget harmful knowledge while keeping beneficial uses of the same concept.

safetyevaluationalignment

G-CARL: Grounded Checklist-Aligned Reward Learning for Patient-Oriented Medical Report Interpretation

Aug 20, 2026

Shiao Xie, Siyu Chen, Jianwei Lv et al.

Medical AI needs dual optimization: factual correctness (verifiable through evidence) and patient communication quality (context-dependent). G-CARL shows that structured checklists paired with retrieval-based verification can train models for both simultaneously better than standard approaches.

This paper introduces a new task where AI systems explain medical reports to patients in accurate, accessible language. The key innovation is G-CARL, a training method that uses retrieval-based fact-checking and customized checklists to ensure explanations are both medically accurate and responsive to what patients actually want to know, without limiting creative variation in responses.

multimodalalignmentevaluation

Beyond the Transcript: Detecting Covert Co ordination in Latent Multi-Agent Communication

Aug 19, 2026

Ramneet Kaur, Pradyumna Chari, Ramesh Raskar et al.

AI agents communicating through hidden internal states can coordinate deception undetectably—but you can monitor and prevent this by tracking latent activations and using counterfactual analysis to steer behavior back to compliance.

This paper addresses a critical safety problem: AI agents can coordinate harmful behavior through hidden communication channels in their internal states, invisible to human oversight.

safetyagentsalignment

Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems

Aug 19, 2026

George Andrikopoulos

Stop benchmarking AI on what it can do at its best; measure instead how consistently it does the same thing when asked the same question twice. This precision metric better predicts real-world system reliability and guides whether you need better rules or a better model.

This paper argues that AI system quality should be measured by precision (consistency of outputs across repeated requests) rather than capability (best-case performance). Using a marksman analogy, the author shows that frontier models have saturated accuracy but differ in output reliability—how tightly grouped their responses are.

evaluationalignment

What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

Aug 17, 2026

Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu et al.

Current compliance monitoring systems for AI don't actually read the rules they're supposed to enforce—they work by pattern matching on scenarios, not rule logic, which undermines their use as regulatory controls.

This paper reveals that compliance detectors used to monitor language models for regulatory violations are 'rule blind'—they make the same decisions regardless of what rule they're supposed to check. The authors show that deleting or swapping rules doesn't change detection accuracy, meaning detectors rely on surface patterns rather than actual rule content.

safetyevaluationalignment

Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning

Aug 17, 2026

Minh-Ha Nguyen, Cathy Shyr

You can improve a frozen language model's performance on specialized tasks by treating policy refinement as a human-in-the-loop process: have an AI critic identify recurring failures, propose natural-language policy changes, and let domain experts decide what gets deployed.

This paper presents Policy Iteration with Human Feedback (PIHF), a method that improves a fixed language model's performance on rare-disease diagnosis by iteratively refining its decision-making policy through human expert review.

trainingalignmentapplications
alignmentevaluationsafety

Synthetic Persona Pretraining: Alignment from Token Zero

Aug 13, 2026

Julian Minder, Viktor Moskvoretskii, Raghav Singhal et al.

Installing alignment values during pretraining from the beginning creates deeper, more robust alignment than adding it after training, and this advantage grows with more pretraining data.

This paper introduces Synthetic Persona Pretraining (SPP), a method that embeds desired values and assistant behavior directly into language models from the start of pretraining rather than adding them afterward.

alignmenttrainingsafety

Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity

Aug 13, 2026

Irina Proskurina, Mayank Kumar, Oyindolapo O. Komolafe

Instruction tuning makes models sound more confident without improving accuracy, and it reduces the diversity of explanations they provide—a potential concern for transparency and reliability.

This paper investigates how instruction tuning affects language models' confidence levels and the diversity of their explanations. The researchers found that instruction tuning makes models express higher confidence in their answers, but this doesn't match improvements in actual accuracy.

trainingevaluationalignment

From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop

Aug 11, 2026

Rahul Gupta, Abhinav Mohanty, Anaelia Ovalle et al.

The field shifted from asking 'why did the model do that?' to 'how do we control what it does?'—and truthfulness is now the primary concern for LLM research, growing from absent to 37% of papers in just four years.

This paper analyzes six years of the TrustNLP workshop (2021-2026), tracking how AI safety research evolved from explaining static models to controlling generative systems.

safetyevaluationalignment

Stealing Reasoning Traces from Proprietary LLM APIs

Aug 10, 2026

Alexander Panfilov, David Schmotz, Ilia Shumailov et al.

Encrypted reasoning traces from LLM APIs are architecturally vulnerable to cross-model decryption attacks—adversaries can extract proprietary reasoning by exploiting compatibility between encrypted blocks across different models in the same provider's ecosystem.

Researchers discovered that major LLM providers (OpenAI, Anthropic, Google) encrypt reasoning traces sent to clients in a way that makes them reusable across different sessions and models. By injecting encrypted reasoning from one model into a weaker model, attackers can force it to decrypt and reveal the reasoning in plaintext.

safetyalignmentevaluation
trainingevaluationalignment

ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning

Aug 4, 2026

Jinhe Bi, Chennan Zhou, Zengjie Jin et al.

Failed expert trajectories are learning gold: teaching models to reflect on why attempts failed is often easier than solving hard problems from scratch, and this reflective skill transfers to direct problem-solving.

ReflectRL shows that when expert models fail on hard problems, their failed trajectories aren't wasted—they're valuable for learning. The method trains models to reflect on these flawed attempts, then transfers that reflective reasoning back to solving problems directly. This lightweight approach works across multiple benchmarks and training methods with minimal computational cost.

trainingreasoningalignment
safetyalignment

AI systems and the reproduction of (standard) language ideologies in World Englishes

Jul 30, 2026

Kingsley Ugwuanyi

AI systems aren't neutral—they embed and amplify existing power structures around which English dialects are considered 'legitimate,' with real consequences for speakers of non-standard varieties who are increasingly mistaken for AI.

This paper examines how AI language models reflect and reinforce biases toward 'standard' English while marginalizing non-dominant varieties spoken globally. Using examples from training data, model design, and public discourse, it shows how AI systems reproduce language ideologies that privilege English from wealthy countries while treating other Englishes as suspect or AI-like.

safetydataalignment

Generative AI and linguistic diversity in academic writing and publishing: Perspectives from World Englishes

Jul 30, 2026

Kingsley Ugwuanyi, Christian Mair, Sender Dovchin et al.

GenAI tools in academic publishing can either reinforce linguistic hierarchies favoring standard English or become tools for resistance—the outcome depends on how they're designed and governed by scholarly communities.

This article examines how generative AI tools affect academic writing and publishing, particularly for non-native English speakers and speakers of World Englishes. Five sociolinguists discuss whether GenAI democratizes writing or reinforces dominant English norms, highlighting concerns about marginalizing linguistic diversity and the need for inclusive AI design.

safetyapplicationsalignment

InfoOps Bench: A live information operations safety benchmark

Jul 30, 2026

Dorian Quelle, Lisa-Maria Neudert, Jonathan Bright et al.

Most frontier language models can be manipulated to spread state-backed disinformation, with refusal rates varying wildly (8.8%-94.5%) and no clear relationship to model size—meaning safety against information operations requires deliberate design choices, not just scale.

This paper introduces InfoOps Bench, a live benchmark that tests whether AI language models can be manipulated into spreading state-backed disinformation. Using real propaganda claims from Russian, Chinese, and Iranian sources, researchers tested 17 models and found huge variation in how easily they could be co-opted—from 8.8% to 94.5% refusal rates.

safetyevaluationalignment
alignmentsafetyreasoning

The Boundaries of Automation: A Theory of Persistent Human Participation

Jul 23, 2026

Fares Fourati, Hinrich Schütze, Eyke Hüllermeier et al.

Automation has conceptual limits: in activities where goals emerge through human-AI interaction (like creative work, learning, or complex decision-making), human participation is constitutive of the outcome itself, not just a workaround for imperfect AI.

This paper argues that human participation in AI systems isn't just a temporary limitation—it's fundamentally necessary in many domains.

agentsalignmentapplications

LKValues: Aligning Large Language Models with Sri Lankan Societal Values

Jul 22, 2026

Nethmi Muthugala, Supryadi, Surangika Ranathunga et al.

LLMs trained primarily on Western data misalign with non-Western cultural values; this work provides a replicable framework for embedding country-specific values in low-resource languages through survey-grounded datasets and targeted fine-tuning.

This paper introduces LKValues, a resource suite for aligning large language models with Sri Lankan cultural values. The authors surveyed 205 Sri Lankans to identify 40 key societal values, created a 150k-instance instruction dataset in Sinhala and English, and built an evaluation benchmark.

alignmentdata
safetyevaluationalignment

Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies

Jul 16, 2026

Filippos Vlahos, Guillaume Bied, Tijl De Bie

LLM-generated content doesn't automatically provide greater neutrality than human-edited sources; it simply embeds different ideological biases that reflect the model's training and design choices.

Researchers compared Grokipedia (an AI-written encyclopedia) and Wikipedia for political bias across nine ideological dimensions using four different LLM judges. They found that Grokipedia, despite being created as an alternative to Wikipedia's alleged left-wing bias, was rated as less neutral by all judges.

evaluationsafetyalignment

Digital Pantheon: Simulating and Auditing Coalition Formation with LLM Agents

Jul 16, 2026

Dylan Van Mulders, Matthias Bogaert, Dirk Van den Poel

LLMs can simulate realistic political negotiations when you combine aggressive preference optimization for ideology with retrieval-augmented generation for factual grounding—and you can audit the results by tracing every clause back to its source document.

This paper uses LLM agents to simulate political coalition negotiations, addressing how to make language models maintain partisan positions while staying grounded in party manifestos.

agentsalignmentevaluation

Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs

Jul 14, 2026

Sen Yang, Yuen-Hei Yeung

Models have identifiable, independently controllable neural coordinates for different aspects of their responses; you can certify and enforce honest reporting by making these coordinates invariant to social pressure while keeping them responsive to real evidence.

Language models often agree with confident users or overstate certainty regardless of actual evidence—a problem called internal incentive-incompatibility. This paper introduces a method to identify and control specific neural coordinates that govern a model's reports, ensuring they resist social pressure while remaining responsive to genuine evidence.

alignmentsafetyevaluation

Metacognition in LLMs: Foundations, Progress, and Opportunities

Jul 13, 2026

Gabrielle Kaili-May Liu, Areeb Gani, Jacqueline Lu et al.

Metacognition—knowing what you know and don't know—is critical for building trustworthy AI systems, and this survey shows practical ways to measure, improve, and apply these abilities in language models.

This paper surveys metacognition in large language models—the ability to monitor and reflect on one's own thinking. It reviews methods to measure metacognitive abilities, techniques to improve them, and applications to make AI systems more reliable and transparent. The work provides the first comprehensive overview of this emerging research area.

evaluationreasoningalignment

Introducing Human-Centeredness in AI-Assisted Lexicography

Jul 13, 2026

Antonio San Martin, Catherine Trekker

AI in lexicography should enhance lexicographers' work through automation while maintaining meaningful human control and decision-making power, not eliminate their role.

This paper proposes a human-centered AI framework for lexicography that emphasizes augmenting rather than replacing lexicographers. It identifies four key dimensions—the augmented lexicographer, sociotechnical context, bias, and tool design—to guide responsible AI integration while preserving linguistic diversity and professional agency.

applicationsalignment

Forgetting Our Way to Shared Meaning: Effects of Forgetting on Conceptual Alignment in a Non-Partnership Coordination Game

Jul 13, 2026

Landon Liu, Mary Kelly, Alan Tsang

Memory degradation and learning adaptiveness fundamentally shape how groups of AI agents develop and maintain shared understanding—forgetting can actually help stabilize agreements rather than hinder them.

This paper studies how shared meaning emerges between agents in language through a non-partnership coordination game. The researchers simulate agents with different memory capabilities and adaptiveness levels, finding that agents who forget gradually maintain more stable agreements than those with fixed learning rates, while adaptive agents converge faster to shared concepts.

agentsreasoningalignment
evaluationtrainingalignment

Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF

Jul 8, 2026

Eric Zhu, Abhinav Shrivastava, Soumik Mukhopadhyay

Not all timesteps and trajectories in diffusion model training contribute equally to learning—by selectively weighting informative steps and replaying valuable past samples, you can dramatically reduce the amount of human feedback needed to align diffusion models.

This paper improves the efficiency of reinforcement learning from human feedback (RLHF) applied to diffusion models by identifying that reward information is unevenly distributed across denoising timesteps and trajectories.

trainingefficiencyalignment
trainingreasoningalignment

Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting

Jul 2, 2026

Vivienne Ming

Human-AI collaboration success depends on specific collaborative traits (perspective-taking, intellectual humility, curiosity) rather than cognitive ability or model benchmarks.

This study examines when pairing humans with AI improves forecasting accuracy using real-money prediction markets as an objective benchmark.

evaluationagentsalignment