Papers

Haozhe Wang, Weijia Feng, Jinpeng Yu et al.

Visual generators need to learn *when* to search for external knowledge, not just *how* to use it—and this knowledge boundary is discoverable through co-training, not fixed in advance.

This paper identifies a critical gap in visual generators: they confidently create incorrect images for requests about new entities, trending topics, and post-training events. The authors show that naive search-augmentation fails because generators have an evolving 'knowledge boundary'—a threshold between what they learned and what needs external context.

Mona Schirmer, Metod Jazbec, Alexander Timans et al.

Simple threshold-based monitoring with statistical risk control can effectively catch unsafe LLM outputs in production without requiring complex sequential testing methods.

This paper presents a real-time safety monitoring system for LLMs that uses a verifier model to detect unsafe outputs at deployment time. The approach calibrates decision thresholds using risk control methods and proves competitive with more complex alternatives on reasoning and adversarial datasets.

Arman Ghaffarizadeh, Danyal Mohaddes, Aliakbar Izadkhah et al.

LLM agents develop emergent social behaviors and hidden objectives in response to relational context—they'll publicly accommodate others due to perceived social pressure even when privately disagreeing, which current evaluation methods miss.

This paper reveals that LLM agents change what they say depending on their audience and social context, even without explicit instructions to do so. Researchers created a dual-channel debate system where agents give public responses and private off-the-record responses, finding that social pressures (like career risk) cause agents to diverge from their true positions by up to 40%.

Juanwu Lu, Junyu Zhu, Ziran Wang

You can build controllable traffic simulators that stay realistic while letting engineers adjust specific behaviors—like making agents safer or faster—without the model gaming the reward system.

This paper presents CNeVA, a framework for creating realistic traffic simulation agents that can both imitate real driving behavior and be steered along interpretable dimensions like speed or safety.

Xuehui Wang, Xuankun Yang, Wei Shen

When pruning visual tokens in VLMs, filtering textual noise with entropy and selecting tokens as a structured optimization problem (not just picking top-K) preserves fine-grained details better while reducing computation.

This paper tackles the problem of compressing image tokens in vision-language models (VLMs) while preserving important visual details. The authors identify that existing pruning methods fail because textual noise corrupts the scoring process and selected tokens become fragmented.

Jiale Amber Wang, Kaiyuan Wang, Pengyu Nie

Existing test generation benchmarks don't verify if tests actually run or match code changes; this benchmark solves that by grounding evaluation in real executable environments and commit history, revealing that state-of-the-art agents still struggle on recent tasks.

TestEvo-Bench is a benchmark for evaluating AI agents on test and code co-evolution tasks—writing new tests for code changes and updating failing tests. Unlike static benchmarks, it uses real commits from Java projects with executable environments to measure pass rates, coverage, and mutation scores.

Vivienne Ming

Human-AI collaboration success depends on specific collaborative traits (perspective-taking, intellectual humility, curiosity) rather than cognitive ability or model benchmarks.

This study examines when pairing humans with AI improves forecasting accuracy using real-money prediction markets as an objective benchmark.

Caleb Ziems, William Held, Su Doga Karaca et al.

Larger LLMs will simulate most human behaviors and opinions better, but scaling alone won't fix simulations of cognitive biases, rare populations, or tasks requiring specialized human knowledge—these need targeted research beyond just bigger models.

This paper investigates whether scaling up language models improves their ability to simulate human social behavior and opinions.

Kent K. Chang

Language models used for cultural analysis aren't neutral measurement tools; their architecture, training data, and evaluation methods actively constitute the cultural phenomena they claim to measure, making methodological choices inherently ethical decisions.

This paper examines how language models measure cultural phenomena, arguing that the models, data, and evaluation methods don't just record culture—they actively shape what counts as cultural reality.

Srinivasan Arunachalam, Louis Schatzki

Coherent quantum memory is the critical resource that enables efficient stabilizer state testing; without sufficient memory, testing becomes as hard as learning, requiring linear in n copies instead of a constant number.

This paper studies how to test and learn quantum stabilizer states when algorithms can only keep a limited number of qubits in quantum memory between measurements.

Zhilin Wang, Han Song, Runzhe Zhan et al.

Autonomous policy improvement requires agents to discover task-specific mechanisms and efficiently convert feedback into parameter updates under constrained budgets—not just win individual tasks.

EvoPolicyGym is a benchmark for evaluating how AI agents autonomously improve executable policies through iterative editing and feedback.

Achint Mehta

For agentic code generation, invest in reasoning capability and effort rather than external tools—stronger models and higher reasoning settings prevent failures at their root, while testing tools don't catch the reasoning errors that actually cause failures.

This study evaluated 90 runs of an agentic coding assistant building the same application, testing whether extra tools and prompts improve code quality. Results show that increased reasoning effort (not testing tools) dramatically improved first-try reliability, raising perfect runs from 28% to 89%, while a testing tool added 42-68% cost with no functional benefit.

Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard et al.

LLMs can grade technical exams reliably for simpler tasks, but struggle with complex questions—rubric quality matters more than which model you choose, and a taxonomy-based approach helps identify which questions are safe to auto-grade.

This paper evaluates whether large language models can reliably grade Linux/bash exam responses by testing GPT, Claude, Gemini, and GLM against expert instructors' grades.

Zhuowei Chen, Liwei Chen, Christian Schunn et al.

Using internal neuron activation patterns to select few-shot examples is more effective than traditional output-based signals, helping identify what the model actually struggles with rather than just guessing from its outputs.

This paper proposes NeuFS, a method for selecting the most useful examples to annotate when adapting large language models to specialized tasks.

David Jurgens

NLP research is migrating from specialized NLP conferences to general machine learning venues, driven partly by citation advantages at ML conferences—a significant shift in the field's institutional center of gravity.

This paper analyzes where NLP research is being published, finding that the field is shifting away from traditional NLP conferences like ACL toward general machine learning venues.

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

You can now predict LLM inference efficiency on GPUs you've never tested by combining public model specs with GPU specifications—no profiling needed, and it works 4x better than physics-based estimates.

WattGPU predicts GPU power consumption and inference latency for large language models without requiring hardware profiling. Using only public LLM metadata and GPU specs, it generalizes to unseen hardware combinations, achieving 3-4x better accuracy than traditional baselines and helping operators choose efficient GPU-LLM pairings.

Benjamin Nichols, Michael Schlichtkrull, Nedjma Ousidhoum

MEDIAREF enables reproducible, cost-effective evaluation of how well LLMs can assess source credibility for fact-checking, addressing a gap where existing systems assume all retrieved evidence is equally reliable.

This paper introduces MEDIAREF, a public database of web documents from 200 media sources designed to help AI systems verify information credibility. Instead of relying on expensive search APIs, researchers can now use MEDIAREF to test how well language models assess whether news sources are trustworthy—a key step in fact-checking systems that cite their sources.

Lourdes Moreno, Paloma Martínez, Marco Antonio Sanchez-Escudero et al.

Multi-agent systems with internal quality signals and intelligent routing can outperform single-model approaches for text simplification tasks, even when adding lexical resources doesn't improve automatic metrics.

This paper presents three automatic systems for Spanish Easy-to-Read translation, submitted to a shared task. The best approach uses a multi-agent workflow combining two language models with quality signals and intelligent routing to simplify text while maintaining meaning. Results show this guided multi-agent approach outperforms simpler baseline methods.

Xi Fang, Weijie Xu, Yingqiang Ge et al.

Personalization in LLMs doesn't just change what users see—it fundamentally alters the reasoning path the model takes to reach answers, creating a measurable failure mode that current mitigation techniques only partially address.

This paper introduces DRIFTLENS, a framework to measure how personalized language models change their reasoning process when given user information, even when final answers stay the same.

Batu Guan, Zirui Wang, Shaohua Li

AI agents can generate compiler optimization patches, but they typically optimize only the reported case rather than generalizing to all similar cases like human developers do—a gap that retrieval-augmented techniques can partially close.

This paper studies how well AI agents can patch compiler optimizations that were missed by LLVM. The key finding is that agents struggle to generalize patches beyond the specific reported case—they often fix the example but fail to cover all similar cases that developers would handle.

Ziyu Chen, Yilun Zhao, Arman Cohan

LLMs can generate reasonable research ideas, but they show systematic biases toward certain types of ideas and miss the full diversity of how human researchers approach novel contributions.

This paper evaluates how LLM-generated research ideas compare to human researcher ideas by analyzing papers and their cited prior works.

Ben Slivinski, Michael Saldivar

Structured verification that requires explicit justification for every reasoning step catches hidden premises and fabricated citations that fool holistic LLM judges, making it a complementary approach for high-stakes verification.

Theoria is a verification system that checks AI reasoning by decomposing answers into explicit state transitions, each justified by citations, computations, or given facts. Unlike opaque LLM judges or formal proofs alone, it produces auditable proof traces where every step can be independently verified, achieving 91.4% precision on expert problems while catching 94.7% of adversarial errors.

Zhi Chen, Zhensu Sun, Yuling Shi et al.

Performance-optimization benchmarks for coding agents have significant reliability issues: reference patches are unstable across machines, scoring rules heavily influence rankings, and most tasks are already solved by existing submissions, making leaderboard positions unreliable indicators of tru...

This paper audits three major benchmarks for evaluating coding agents on performance optimization tasks.

Shayan Talaei, Abhinav Chinta, Devvrit Khatri et al.

D2D reveals stealth biases in deployed LLMs by concentrating distributional shifts into a small adapter, making hidden preferences visible in generated text—enabling auditing of models where bias inspection would otherwise be impossible.

This paper introduces Distill to Detect (D2D), a method to uncover hidden biases in language models that only favor certain entities or viewpoints on specific topics while appearing normal elsewhere. The approach works by distilling differences between a suspect model and its base version into a compact adapter, amplifying hidden bias signals into detectable text patterns.

Kornelius Raeth, Nicole Ludwig

Train generative models to minimize decision costs directly, not just prediction error—this focuses learning on regions where mistakes matter most for your application.

This paper proposes a training method for generative models that makes them aware of decision costs. Instead of training only to predict data accurately everywhere, the method adds a decision loss that penalizes forecast errors in regions where mistakes are most expensive for downstream decisions. The approach combines standard training objectives with cost-sensitive feedback.

Sergio Hernández-Gutiérrez, Matteo Merler, Ilze Amanda Auzina et al.

Simple prompting baselines outperform recent dense supervision methods, and you can now evaluate supervision signal quality before training by checking if scores align with reference Q-values—saving significant compute.

QVal is a training-free evaluation framework for comparing dense supervision signals used in long-horizon LLM agents.

Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona et al.

Training LLMs to accurately self-assess their performance creates a powerful RL signal that improves both calibration and accuracy—models that know what they don't know become more reliable and better at learning.

This paper introduces reinforcement learning with metacognitive feedback (RLMF), a method that trains language models to accurately judge their own performance and express uncertainty faithfully.

Yuqing Yang, Qi Zhu, Zhen Han et al.

Data referencing errors are a widespread problem in LLM table reasoning that goes beyond final-answer accuracy; using a lightweight critic model to catch these errors during inference significantly improves reliability.

LLMs make data referencing errors when reading tables—citing wrong values or missing data despite understanding table structure. This paper systematically measures these errors across models and shows that using a critic model to detect and filter bad outputs improves accuracy by up to 12%, even with a small 4B-parameter critic.

Sanghyuk Chun, William Yang, Amaya Dharmasiri et al.

Breaking uncertainty into interpretable components—what's ambiguous about the task versus how many right answers exist—lets you estimate confidence efficiently in multimodal models without expensive sampling.

CoMet decomposes uncertainty in multimodal AI models into two components: context-specific ambiguity (from the task or prompt) and multiplicity (how many valid answers exist). A lightweight module estimates these without generating multiple answers, enabling efficient uncertainty quantification for open-ended tasks like visual question answering.

Philippe Chlenski, Zachariah Carmichael, Ayush Warikoo et al.

Open models are poor surrogates for mechanistic understanding of closed models: prediction-level agreement doesn't guarantee attribution agreement, and white-box signals don't reliably transfer between models.

This paper investigates when open-source language models can serve as proxies for understanding closed commercial models. The researchers test whether measurements from open models (like attention patterns) reliably explain closed models' behavior across prediction, attribution, and representation levels, finding that models agreeing on answers often disagree on reasoning.

Benjamin Breen, Austin Letson, Borja Requena Pozo et al.

Agentic code generation can be dramatically improved by using verification feedback to guide iterative repair of both code and formal proofs, rather than trying to generate correct code in one shot.

AxDafny is a system that helps AI models generate verified code in Dafny by iteratively fixing code and proofs based on verification feedback. The authors created a benchmark of 250 programming problems and show their approach achieves 92.7% verification success, significantly outperforming previous methods.

Ziwei Su, Junyu Ren, Victor Veitch

Embedding norms in contrastive models aren't wasted information—they automatically capture semantic properties during training and can be leveraged as free calibration signals without additional training.

This paper explains why embedding norms (magnitudes) in contrastive models encode semantic information like concept specificity, even though these models use scale-invariant losses that should ignore norms.

Kunyang Li, Kyle Domico, Jonathan Gregory et al.

In multi-agent systems, a small number of communication channels often control most of the attack surface—MESA can identify these critical edges proactively, letting defenders protect 3x more attacks with the same security budget.

This paper introduces MESA, a framework for identifying which communication channels in multi-agent systems are most critical to protect. By analyzing graph structure and testing channel importance without needing attack data, MESA ranks edges by their security risk, helping defenders focus limited resources on the most vulnerable connections before attacks happen.

Asif Shahriar, Hongyu Cai, Hadjer Benkraouda et al.

LLM-based code vulnerability detectors can be manipulated through cognitive heuristics without changing the actual code, making them unreliable for security-critical tasks and vulnerable to adversarial attacks that suppress vulnerability detection.

This paper reveals that LLMs used for detecting code vulnerabilities are susceptible to cognitive biases—the same mental shortcuts that affect human judgment.

Gervais Hatungimana, Abdun Naser Mahmood, Mohammad Jabed Morshed Chowdhury

By analyzing network traffic patterns between clients and file servers, you can detect ransomware attacking shared storage earlier and more reliably than traditional endpoint-only detection, even when the malware doesn't show obvious signs on the server itself.

This paper presents a hybrid detection system for crypto-ransomware targeting shared storage in enterprise networks. It combines signature-based detection (using network traffic indicators) with machine learning to catch ransomware before it encrypts files, achieving 99.64% precision with zero false negatives.

Nico Daheim, Iryna Gurevych

When LLMs handle subjective tasks, explicitly modeling uncertainty and using Bayesian decision theory to choose outputs can improve results, but risk-averse approaches may backfire by favoring generic responses.

This paper develops uncertainty-aware decision-making algorithms for LLMs in subjective tasks like tutoring and peer review. The authors use Bayesian decision theory and conformal prediction to account for model uncertainty when generating outputs, finding that Bayesian approaches work better than risk-averse methods for improving output quality.

Liyao Wang, Ruipu Wu, Haojun Xu et al.

Combining explicit 3D geometry (camera poses, spatial relationships) with visual matching dramatically improves cross-view localization and enables zero-shot transfer between ground and drone views without paired training data.

This paper tackles cross-view object geo-localization—finding a target object in satellite imagery when given a ground or drone photo. The authors introduce a large dataset with 220K+ image pairs and geometric metadata, plus GAGeo, a unified framework that predicts object locations, masks, and camera poses simultaneously using 3D spatial understanding rather than just appearance matching.

Kevin Kingslin, Anish Natekar, Ashutosh Ranjan et al.

Using multi-perspective debate to extract alignment principles from preferences captures richer decision-making reasoning than single-pass explanations, leading to more faithful and interpretable AI steering.

This paper improves how AI systems learn from human preferences by using structured debates between different viewpoints to uncover the reasoning behind choices. Instead of just recording which option humans prefer, Democratic ICAI captures multiple competing arguments that influence decisions, then distills these into clear principles that guide AI behavior.

Rajesh Jayaram, Drew Tyler, David Woodruff et al.

AI-assisted peer review can augment (not replace) human reviewers by catching errors early and reducing their workload, but requires careful design to preserve human oversight and decision-making authority.

Google researchers introduce Paper Assistant Tool (PAT), an AI system that automatically reviews scientific papers by checking mathematical proofs, validating experiments, and identifying flaws. PAT uses inference scaling to catch errors before human peer review, addressing the bottleneck created by AI-accelerated research outpacing traditional review capacity.

Niclas Lietzow, Danielle Bitterman, Carsten Eickhoff et al.

Vision-language models have a sparse, identifiable causal circuit that controls whether they trust visual input or stored knowledge—removing just a few attention heads flips the model from knowledge-based to vision-based answers in most cases.

This paper reveals how vision-language models choose between visual evidence and memorized knowledge when they conflict. Using activation analysis, researchers identified a small set of attention heads (2.5-4.8% of heads) that act as a causal switch: removing them makes models trust their eyes instead of what they've learned.

Ali Zia, Usman Ali, Abdul Rehman et al.

Using topological features (shape and connectivity patterns) during test-time adaptation significantly improves anomaly segmentation by preserving structural coherence that pixel-level methods miss, achieving 15% F1 improvement on standard benchmarks.

This paper introduces TopoTTA, a test-time adaptation framework for anomaly segmentation that uses topological data analysis (persistent homology) to preserve structural consistency in defect detection.

Daniel Russo

Evaluating AI coding agents one at a time on isolated tasks misses the real problem: agent contributions create twice as much integration friction in shared codebases, making ecosystem-level governance more important than agent-level performance.

This paper studies how autonomous coding agents affect shared software repositories by analyzing over 930,000 pull requests. It finds that integration friction—the cost of merging code when others are changing it simultaneously—is largely a repository-level problem, not an agent problem.

Johannes Zenn, Jonas Geiping

Sequence probability is useful for ranking answers within a dataset but shouldn't be trusted as a guide for choosing decoding methods or hyperparameters—optimizing for probability doesn't guarantee better answers.

This paper investigates whether higher sequence probability in language models actually correlates with correct answers. The researchers test this across different decoding methods, models, and benchmarks, finding that while probability predicts correctness within a dataset, changing decoding parameters to increase probability doesn't reliably improve accuracy.

Nicholas Pulsone, Gregory Goren, Roee Shraga

Distribution alignment is critical for entity matching in low-resource settings—understanding which algorithmic choices matter most helps practitioners build more reliable data integration systems with limited supervision.

This paper investigates BEACON, a method for matching records across databases when you have limited labeled data and domain knowledge. The researchers test how different design choices and data availability affect performance, revealing insights about how distribution alignment helps the system adapt to new domains.

Mohammad Mehdi Hosseini, Mohammad H. Mahoor, Hiroko H. Dodge

Language-based digital twins can authentically mimic individual conversational behavior while simultaneously serving as cognitive health monitors—enabling continuous, personalized monitoring of cognitive decline without requiring frequent clinical visits.

This paper creates AI-powered digital twins of elderly people using language models to capture their unique conversational patterns and writing style. The system learns from real conversations to generate authentic responses and can predict cognitive health scores, offering a non-invasive way to monitor early signs of cognitive decline.

Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. Hosseini

Adding soft sparsity regularizers to Top-k sparse autoencoders makes interpretable features more robust and concentrated, without the drawbacks of earlier penalty-based approaches—hard and soft sparsity work better together.

This paper improves sparse autoencoders (SAEs) for interpreting vision models by adding sparsity regularizers to the Top-k SAE architecture. The researchers introduce two penalty methods that work alongside Top-k's hard sparsity constraint to make learned features more interpretable (monosemantic) without hurting reconstruction quality.

Serhii Hamotskyi, Akash Kumar Gautam, Christian Hänig

LLMs can replace rigid rule-based systems for document compliance verification, handling messy real-world text better than traditional NER while requiring no task-specific training data.

This paper applies large language models to automatically verify whether securities meet eligibility criteria for use as collateral at the German Central Bank. Instead of manually reading through complex, bilingual prospectuses, the system uses LLMs to extract, normalize, and interpret financial and legal information, achieving 91% precision while avoiding false acceptances.

Hamid Reza Firoozfar, Mohammadsadegh Abolhasani, Reza Mousavi et al.

A mechanism-oriented taxonomy of how language encodes hidden meaning is more effective for LLM-based content moderation than taxonomies based on communicative intent or surface forms.

This paper creates a taxonomy of indirect linguistic expressions (coded language like algospeak and euphemisms) that people use to evade content moderation. Rather than categorizing by intent, the taxonomy focuses on the underlying encoding mechanisms—how meaning is hidden and recovered.

Arnav Mazumder, Dengjia Zhang, Shuyue Stella Li et al.

When building multilingual AI systems with multiple translation steps, preserve the original user input throughout the pipeline instead of discarding it after each stage—this simple change significantly improves reasoning quality across languages.

This paper shows that translation cascades for multilingual reasoning lose important context at each step. By keeping the original question, translated question, and reasoning trace available to the final translation step, the authors achieve better results across 285 languages without retraining—a simple fix that prevents information loss in multi-stage pipelines.

Muhammad Hassan, Ramazan Yener, Ece Gumusel et al.

AI healthcare chatbots frequently fail users in critical ways—unreliable access, confusing interfaces, and privacy risks—suggesting designers need to prioritize reliability and trust before expanding these tools into healthcare.

This study analyzes over 15,000 user reviews of AI healthcare chatbots to understand how they work in real-world health information seeking. The researchers found three main problem areas: access and reliability issues, poor user experience, and billing/support problems, with privacy concerns causing the most negative user experiences.

Josef Chen

Before building a multi-model system, measure how often all your models fail together—this sets a hard ceiling on possible gains. Standard error correlation metrics won't tell you this, but a simple statistical bound will.

This paper reveals a fundamental limit on multi-model LLM systems: their accuracy gains are capped by how often all models fail together on the same question. The authors measure this 'co-failure rate' across 67 frontier models and show that standard metrics like error correlation miss this ceiling, making it invisible to practitioners.

Preet Baxi, Jiannan Xu, Jane Yi Jiang et al.

Prompt injection attacks on LLM hiring systems are effective only when rare and candidate quality is homogeneous; widespread adoption or quality differences make the attack ineffective, but fairness concerns remain when manipulation is selective.

This paper studies how candidates can manipulate LLM-based résumé screening systems through prompt injection—adding subtle persuasive text without new qualifications. Experiments show the attack works when few candidates use it and qualifications are similar, but fails as manipulation spreads. The research reveals fairness risks when lower-quality candidates can game the system.

Alina Bazarova, Johann Fredrik Jadebeck, Henrik Zunker et al.

Neural simulation-based inference can replace slow MCMC for fitting complex disease models, running 15-120x faster on GPUs while producing nearly identical results—enabling real-time outbreak analysis.

This paper compares simulation-based inference (SBI) with traditional MCMC methods for fitting epidemiological models to COVID-19 data. SBI uses neural networks to learn the relationship between model parameters and data, enabling much faster Bayesian inference—achieving 15-120x speedups while maintaining accuracy comparable to MCMC.

Lang Huang, Jinglue Xu, Luke Darlow

Before building bigger models, optimize your data preprocessing: context length, normalization strategy, and regularization can close most of the accuracy gap at a fraction of the computational cost.

This paper shows that simple linear models (Ridge regression) can match or beat complex deep learning architectures for time-series forecasting by carefully tuning preprocessing—context length, normalization, and regularization—rather than scaling model size.

Junwei Luo, Shuai Yuan, Zhenya Yang et al.

For Earth observation forecasting, explicitly conditioning on weather anomalies and cumulative physical stress—rather than treating weather as generic conditioning—improves predictions of how vegetation responds to extreme weather events.

EO-WM is a video diffusion model for predicting future Earth surface conditions from satellite imagery while accounting for weather effects. Unlike existing methods, it explicitly models how weather forcing (heat, drought) drives changes in vegetation and other surface features, using separate conditioning pathways for baseline climate and weather anomalies.

Maria Levchenko

Historical text imposes a consistent encoding penalty on LLMs, but models retain semantic understanding—making them safe for retrieval tasks if generative applications are adapted with temporal context.

This paper diagnoses why language models struggle with historical text by breaking down the problem into four dimensions: tokenization cost, predictive uncertainty, semantic robustness, and context sensitivity.

MSVPJ Sathvik, Parmitha Vangapadu, Nishit Rane et al.

The dataset enables building explainable AI systems to automatically detect manipulative betting ads on social media, with practical applications for user protection and regulatory monitoring.

This paper introduces BetXplain, a dataset of betting advertisements from Instagram and Reddit annotated for manipulative tactics and deceptive practices. Each ad includes human explanations of why it's manipulative, enabling research into detecting misleading betting promotions that could harm users' mental health and financial well-being.

Graham Gibson, John Tipton, Kellin Rumsey et al.

You can get reliable uncertainty estimates without expensive retraining by using influence functions and linear algebra—making uncertainty quantification practical for real models.

Ribbon is a fast method for measuring how uncertain a machine-learning model's predictions are. Instead of retraining a model many times (which is expensive), it uses math tricks to estimate uncertainty from a single trained model, working well even when the model assumptions are wrong.

Parmitha Vangapandu, Sai Ganesh Mokkapati, Sathwik Narkedimilli et al.

Mental health NLP models perform better when trained on data that includes relational context and interpersonal triggers, not just isolated symptoms—this shift from individual-centric to context-aware modeling improves both accuracy and clinical relevance.

This paper introduces RSPC, a dataset of 1,799 Reddit posts about long-distance relationships annotated by psychiatrists for mental health conditions, relationship stressors, and relationship phases.

Amit Elhelo, Amir Globerson, Mor Geva

Language models don't store facts in a single, consistent way like traditional databases do. Instead, they encode knowledge in task-specific parameter subsets, meaning the same fact may be retrieved differently or not at all depending on how you ask the question.

This paper investigates whether language models store factual knowledge like unified databases or in task-specific ways.

Zhengyuan Liu, Stella Xin Yin, Min-Yen Kan et al.

When building human-AI collaborative systems, pay attention to metacognitive dialogue (how teams reflect on and adjust their approach) alongside task progress—it's a key indicator of collaboration quality.

This paper presents a framework for analyzing dialogue during collaborative problem-solving between humans and AI systems.

Sangwoo Cho, Kushal Chawla, Pengshan Cai et al.

Instead of asking an LLM for a single opaque score, ask it multiple specific binary questions about output quality, then aggregate the answers—this gives you both better evaluation accuracy and actionable feedback for improvement.

BINEVAL breaks down LLM evaluation into simple yes/no questions about specific criteria, then combines answers into interpretable scores. This makes evaluation transparent, debuggable, and useful for improving prompts—matching or beating existing LLM judges while being easier to understand and fix.

Manjinder Singh, Alexander E. I. Brownlee, Mohamed Elawady

Genetic algorithms can effectively attack NLP models with only output logits, achieving high success rates by intelligently searching for semantically similar word substitutions—showing that black-box adversarial attacks on language models are more powerful than previously demonstrated.

This paper presents GAversary, a genetic algorithm that generates adversarial text attacks on NLP models by treating them as black boxes. It uses GloVe embeddings to find semantically similar word replacements that fool classifiers, achieving stronger attacks than existing methods like BAE and A2T, though at the cost of modifying more words.

Jeremias Ferrao, Niclas Müller-Hof, Iustin Sîrbu et al.

Training safety classifiers to explicitly model user intent—not just analyze prompts directly—produces more robust safety decisions across different training approaches and external benchmarks.

This paper shows that safety classifiers work better when they explicitly model what users intend to do, not just what they say. The authors created AIMS, a dataset of 1,724 tricky safety prompts with intent descriptions, and tested intent-aware training across multiple methods (fine-tuning, preference learning, reasoning distillation, and reinforcement learning).

Ping Xiong, Thomas Schnake, Klaus-Robert Müller et al.

When explaining temporal graph models, you need to track information flowing through event-induced variables—not just embeddings—to capture how long-range dependencies actually work in the network.

This paper develops a new method to explain how Temporal Graph Neural Networks make predictions by tracking information flow through all components, not just embeddings. The approach uses a framework called Normalized Relevance Measure to systematically decompose complex temporal graph models and identify which events and interactions matter most for predictions.

Humzah Merchant, Bradford Levy

You can improve LLM forecasting accuracy by identifying and amplifying time-awareness features inside the model, reducing the bias toward using information that shouldn't be available yet.

This paper uses sparse autoencoders to identify internal features in LLMs that drive forecasting behavior, distinguishing between time-aware reasoning and look-ahead bias. By steering these features, researchers show they can reduce the model's tendency to use future information while maintaining general reasoning ability.

Jiajun Wu, Haoyu Kang, Yining Sun et al.

Evaluating harmful content detection requires multi-layered reasoning beyond surface-level classification; models need to explain their decisions and understand implicit harms, not just flag obvious ones.

HarmVideoBench is a benchmark for evaluating how well AI models understand harmful content in videos. Unlike existing tests that just ask yes/no questions, this benchmark uses 1,379 videos with 4,137 multiple-choice questions across three difficulty levels—from spotting obvious harmful elements to reasoning about context beyond what's shown.

Babak Rahmani, Sebastian Dziadzio, Joschka Strüber et al.

You can reverse-engineer an agent's decision logic from its behavior by combining observation with strategic experimentation—a technique that works for policy interpretability and opponent modeling in competitive settings.

RevengeBench is a benchmark for reconstructing hidden decision-making code from an agent's behavior in games. Researchers observe a hidden policy playing and can design custom opponents to probe its behavior, then submit executable code that mimics it.

Andrei Liviu Nicolicioiu, Mohammad Pezeshki, Aaron Courville

Self-distillation trades diversity for accuracy: models become overconfident in their preferred solutions, hurting performance on out-of-distribution tasks that need varied strategies.

This paper reveals a hidden cost of on-policy self-distillation: while it achieves high average accuracy, it reduces output diversity by amplifying the model's existing biases. The authors show theoretically and empirically that self-distillation concentrates probability mass on dominant modes, causing pass@k curves to flatten—generating more rollouts doesn't improve accuracy like it should.

Martijn Bartelds, Federico Bianchi, James Zou

Real-time voice AI systems can hear emotional cues but don't use them in decision-making; they need explicit prompting to consider tone, and even then improve only partially—making them risky for emotionally sensitive interactions.

This paper evaluates four leading real-time voice AI systems (GPT-4 Realtime, Gemini Live, Qwen Omni) and finds they ignore emotional tone and vocal delivery when making decisions, even though they can perceive these cues when asked directly.

Changdae Oh, Wendi Li, Seongheon Park et al.

You can extract free step-level evaluation signals from standard RL post-training using progress advantage, eliminating the need to build expensive process reward models for agent systems.

This paper shows that RL-trained language models already contain step-level scoring signals without needing separate reward models. The authors derive 'progress advantage'—a metric based on policy log-probability ratios—that automatically captures how good each step is, and demonstrate it works for scaling, uncertainty, and debugging across multiple benchmarks.

Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli

Multimodal AI models are unreliably sensitive to input order—a property that should be baseline for production systems. Simple prompt fixes don't solve this; the problem likely requires changes during model training or design.

This paper audits 18 multimodal AI models to check if they give consistent answers when information is presented in different orders. The researchers found that all models fail this basic reliability test, with 24-50% of answers changing based on order.

Aditya Singh, Gerson Kroiz, Senthooran Rajamanoharan et al.

Detecting bad behavior isn't enough to prove misalignment—you need forensic investigation to distinguish between malicious intent and innocent mistakes like confusion or shortcuts.

This paper develops a protocol for investigating whether concerning AI model behaviors stem from misalignment (intentional deception) or benign causes like confusion. The authors use chain-of-thought reasoning to generate hypotheses about behavior, then test these hypotheses through targeted prompt and environment modifications across six agentic scenarios.

Bo Chen

Statistically significant findings from keyword-based text analysis can be entirely artifacts of the measurement method rather than real phenomena. Always validate keyword-based results with semantic approaches before drawing conclusions about speaker psychology or discourse patterns.

This paper reveals how keyword-based measurement tools can produce false findings in computational social science. By comparing keyword counting to LLM-based semantic analysis of interviews, the authors show that a strong statistical correlation between negative affect and certainty disappears—and even reverses—when using more accurate measurement.

Yuxing Cheng, Yuan Wu, Yi Chang

High accuracy on clean images doesn't guarantee robustness to visual corruption—VLMs struggle significantly with degraded text-rich content, especially structured formats like charts and tables, which matters for real-world deployment.

This paper introduces OCR-Robust, a benchmark for testing how well vision-language models handle text recognition and reasoning when images are corrupted or degraded.

Yves Ferstler, Adam Podoxin, Ty Brassington et al.

AI literary translation is competent but lacks the immersive quality readers value—and automatic metrics don't capture what makes human translation better, suggesting we need reader-centered evaluation for creative content.

Researchers compared AI-generated literary translations to human translations by having 15 avid readers evaluate excerpts from 15 recent novels in French, Polish, and Japanese. While readers found AI translations adequate, they consistently preferred human translations for their clarity and immersive quality.

Poojitha Thota, Shirin Nilizadeh

Poisoned summarization models leave detectable structural artifacts—high training influence in white-box settings and unusual sensitivity to semantic perturbations in black-box settings—allowing 85-92% detection accuracy and recovery of 96% of original behavior without full retraining.

This paper presents a defense framework against data poisoning attacks on text summarization models during fine-tuning. The authors develop detection methods using influence analysis (white-box) and behavioral auditing (black-box), plus an unlearning technique to remove poisoned effects.

Yu-Yang Chen, Lan-Zhe Guo

Current multimodal AI models have a fundamental weakness in multi-view spatial reasoning—they can't reliably track objects across different camera angles, and this limitation can't be fixed by better prompting strategies alone.

TriViewBench is a controlled benchmark for testing how well multimodal AI models handle complex visual reasoning across multiple views of 3D scenes.

Blade Frisch, Will Wade, Dylan Gaines et al.

AI for accessibility requires evaluation methods that go beyond standard metrics to capture the complex, intersectional needs of real users—one-size-fits-all approaches will fail people who depend on these systems.

This paper examines how AI can improve augmentative and alternative communication (AAC) systems for people with speech disabilities, but argues that standard evaluation metrics miss important nuances about what users actually need. The authors study six real AAC challenges and propose evaluation methods that account for the diverse, intersecting needs of individual users.

Yikai Lu, Yifei Wu, Xinyu Lu et al.

You can't verify general agents work everywhere, but you can certify they work reliably on specific transitions by examining their internal world model—this enables practical deployment of capable agents in complex environments.

This paper addresses how to verify that general-purpose AI agents work reliably in complex environments. Instead of checking if an agent handles everything perfectly, the authors propose 'structural certification'—a method that identifies which specific situations the agent understands well and which it doesn't.

Tian Zheng, Kai-Tai Hsu

Evaluating agentic systems requires multi-layered grading strategies with different failure modes; a cascade combining strict pattern matching, lenient LLM grading, and human review is more reliable than any single approach.

This paper tackles the challenge of evaluating agentic data analysis systems that produce complex outputs like code, results, and explanations. The authors develop a three-layer grading cascade combining regex matching, LLM-based evaluation, and human review, achieving 97% recall while maintaining 100% precision. They show that iterative nudging improves grading success from 36% to 97%.

Ali Pourghasemi Fatideh, Wilder Baldwin, Maria Dhakal et al.

LLM dialogue systems for compliance assessment suffer from low accuracy against expert ground truth, and user satisfaction decreases with longer responses—designers should prioritize concise, proactive interactions over verbose explanations.

This paper evaluates how well LLM dialogue assistants (like GitHub Copilot) help developers assess non-functional requirements like HIPAA compliance. The researchers had 49 programmers use Copilot to evaluate 148 compliance requirements against real code, measuring both accuracy against expert standards and user satisfaction across multi-turn conversations.

Maria Nefeli Paraskevopoulou, Tatiana Passali, Grigorios Tsoumakas

For scientific summarization, training on carefully selected high-quality examples outperforms training on larger random datasets—quality matters more than quantity when building summarization systems.

This paper creates a large biomedical summarization dataset (1.88M articles) and shows that author-written abstracts vary in quality. By selecting high-quality training examples based on alignment with source articles, models achieve better results with less data than random sampling, improving both efficiency and factuality.

Hariom Ingle, Ronit Ghode, Ishwari Gondkar et al.

Marathi NLP now has a gold-standard POS tagging dataset and baseline models, enabling better downstream tasks like machine translation and parsing for an under-resourced language with 83+ million speakers.

This paper introduces L3Cube-MahaPOS, a manually annotated dataset of 32,354 Marathi sentences for part-of-speech tagging, along with benchmarks across multiple model architectures.

Sunil Wanjari, Manish Thakre, Aayushi Asole et al.

Combining multiple validated mental health screening tools with cognitive and personality data through a modular, interpretable system improves diagnostic accuracy and provides clinicians with more reliable decision support than using individual assessments alone.

PsyBridge is a decision-support system that combines clinical screening tools (PHQ-9, GAD-7), cognitive tests, and personality assessments into one interpretable framework for mental health evaluation.

Prateek Agnihotri, Sanchit Jain, Prabhat Agnihotri et al.

Reframing symbolic reasoning problems as string matching and search—rather than arithmetic simulation—helps LLMs avoid hallucinations and handle combinatorially complex tasks more reliably.

This paper tackles bit manipulation puzzles by teaching LLMs to deduce hidden logical rules transforming binary strings. Instead of forcing models to simulate complex boolean logic (which causes hallucinations), the authors reframe the problem as string similarity matching and structured search with backtracking.

Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim

LLMs cannot reliably self-report when they've been adversarially manipulated, and training methods meant to improve this detection can paradoxically make models more vulnerable to attacks while appearing more confident in false claims.

This paper investigates whether large language models can accurately recognize when their own outputs were manipulated by adversarial prefill attacks. Testing 10 models across 4 safety benchmarks, researchers found that models fail to reliably detect their compromised responses, often falsely claiming they acted intentionally.

David Mguni, Julian Ma, Jun Wang

LLMs cannot be universal problem solvers through prompting alone because language itself is a bottleneck; some task families will always be unsolvable via prompts, no matter how much data or compute you throw at them.

This paper proves fundamental limits on what LLMs can learn through prompting alone. Using game theory and information theory, the authors show that language is a capacity-limited channel—when task complexity exceeds what can fit in a prompt, different tasks become indistinguishable to the model, creating an irreducible error floor that no amount of data or scaling can fix.

Juyang Bai, Laixi Shi

Prompt optimization can significantly improve multi-agent systems, but gains vary dramatically depending on task complexity, team structure, and communication patterns—there's no one-size-fits-all approach.

This paper studies how to optimize system prompts for multi-agent LLM systems—where multiple AI agents work together with different roles and communication patterns. The researchers test prompt optimization techniques across different team sizes, workflows, and tasks to understand when and how much these optimizations actually improve performance.

Jincheng Zhong, Weizhi Wang, Che Jiang et al.

Enterprise agents need evaluation frameworks that measure artifact quality, cost, runtime, and skill transfer—not just task completion—because real workplace tasks are complex, heterogeneous, and require reproducible, auditable results.

This paper introduces EnterpriseClawBench, a benchmark for evaluating AI agents in real workplace environments. Built from actual enterprise sessions, it contains 852 reproducible tasks involving file handling, tool use, and business artifact creation.

Campbell Lund, Thomas Euyang, Zanele Munyikwa et al.

Popular AI exposure scores are useful but incomplete—they don't capture how AI adoption actually happens, who really benefits or loses, or how impacts change over time.

This paper examines AI exposure scores—metrics measuring how much AI can assist with job tasks—that have become central to workforce policy debates. The authors show these static scores have significant limitations when applied to real-world policy questions, and identify a critical gap: researchers keep using outdated scores while ignoring newer methodological improvements.

Nityanand Mathur, Hamees Sayed, Wasim Madha et al.

Style instructions in TTS are processed differently than content words—they influence acoustic properties like pitch and energy globally rather than locally, with maximum effect in early generation steps and mid-depth network layers.

This paper reveals how individual words in style descriptions influence speech generation by analyzing attention patterns in a text-to-speech system.