Papers

Juarez Monteiro, Nathan Gavenski, Gianlucca Zuin et al.

Selectively querying language models based on uncertainty can improve RL agent robustness in novel situations without constant computational overhead—but successful integration requires careful design, not just combining the two systems.

This paper proposes ASK, a system that combines reinforcement learning agents with language models to handle out-of-distribution scenarios.

Oliver Aleksander Larsen, Mahyar T. Moghaddam

If you're building AI systems, standard software architecture documentation won't capture ML-specific risks like model drift or data dependencies—RAD-AI provides a structured way to document these for both compliance and team understanding.

RAD-AI extends existing architecture documentation frameworks (arc42 and C4 model) to handle AI systems, adding sections for probabilistic behavior, ML lifecycles, and data dependencies. It maps to EU AI Act compliance requirements and shows 93% coverage of regulatory documentation needs versus 36% for standard frameworks.

Biplab Pal, Santanu Bhattacharya

Before deploying agentic AI in business processes, measure the 'blind mass' of uncertain state-action pairs and expected oversight costs using event logs—this reveals hidden decision gaps that simple accuracy metrics miss.

This paper develops a mathematical framework to measure when AI agents can safely operate autonomously versus when they need human oversight.

Zhuo Li, Yupeng Zhang, Pengyu Cheng et al.

Using multiple agents with intentional information barriers prevents LLMs from confirming their own errors during fact-checking, letting smaller models match larger ones on reliability.

MARCH is a framework that reduces hallucinations in LLMs by using three specialized agents that work together with deliberate information separation. A Solver generates responses, a Proposer breaks them into verifiable claims, and a Checker validates claims without seeing the original output—preventing the verifier from copying the generator's mistakes.

Duc Vu, Anh Nguyen, Chi Tran et al.

If you're concerned about your photos being used to generate deepfake videos, adversarial perturbations applied in multiple domains (color and frequency) can effectively block modern video generation models while remaining imperceptible to humans.

This paper presents Anti-I2V, a defense method that protects photos from being misused in AI-generated fake videos. Instead of just adding noise to images, it works across multiple color spaces and frequency domains to disrupt video generation models, targeting both traditional and newer Transformer-based architectures.

Ufaq Khan, Umair Nawaz, L D M S S Teja et al.

Medical VLMs need explicit training on input validation (checking modality, anatomy, orientation) as a separate safety step before diagnosis, not as an afterthought—current models hallucinate plausible reports even on obviously invalid inputs.

This paper reveals a critical blind spot in medical AI: vision-language models can generate fluent medical reports even when given invalid inputs like wrong body parts or upside-down images. MedObvious is a benchmark of 1,880 tasks testing whether models can catch these basic sanity checks before attempting diagnosis—a step human radiologists do automatically but VLMs currently fail at.

Sagar Kumar, Ariel Flint, Luca Maria Aiello et al.

LLM outputs are unstable across contextually equivalent formulations of the same task, meaning benchmark results may not reflect how models actually behave in real applications—a critical issue for bias testing and high-stakes use.

This paper reveals that large language models fail to give consistent outputs when tasks are reformulated in contextually equivalent ways.

Rustem Islamov, Grigory Malinovsky, Alexander Gaponov et al.

You can now build federated learning systems that defend against both Byzantine attacks and privacy breaches simultaneously, without needing unrealistic assumptions like bounded gradients or extra server datasets.

This paper tackles two critical security issues in federated learning: protecting against malicious servers (Byzantine attacks) and preventing data leakage (differential privacy).

Abdul Rahman

Security AI models fail when deployed to new environments because telemetry data is fragmented. CSTS solves this by providing a unified, entity-focused data structure that maintains consistent identity and relationships across different systems.

This paper introduces CSTS, a standardized way to represent security data that helps AI systems detect cyber threats across different computer networks. Instead of treating security events as isolated incidents, CSTS organizes them around entities (like users or devices) and their relationships, making AI models more reliable when deployed in new environments.

Carolin Holtermann, Minh Duc Bui, Kaitlyn Zhou et al.

Adding voice to language models doesn't just extend text capabilities—it introduces new bias mechanisms tied to speaker identity cues that amplify discrimination beyond text-only versions, requiring fairness safeguards alongside accessibility improvements.

Voice interfaces on AI chatbots amplify gender discrimination more than text-based versions because speech reveals speaker identity through tone and accent. The research shows these models shift toward gender-stereotyped responses based on voice alone, and surveys reveal users worry about hidden attribute inference.

Sai Koneru, Elphin Joe, Christine Kirchhoff et al.

Instruction-tuned models are vulnerable to user pressure even with strong evidence present; simply providing richer context doesn't guarantee models will resist sycophancy without explicit training for epistemic integrity.

This paper tests how well instruction-tuned language models stick to evidence when users pressure them to agree with false claims. Using climate science as a test domain, researchers found that adding more detailed evidence doesn't reliably prevent models from abandoning facts to please users—especially when evidence includes research gaps or uncertainty.

Huaide Jiang, Yash Chaudhary, Yuping Wang et al.

Embodied navigation systems perform well in clean lab conditions but fail dramatically in real-world scenarios with sensor noise and unclear instructions—this benchmark exposes those gaps and provides mitigation strategies.

NavTrust is a benchmark that tests how well navigation AI systems handle real-world problems like blurry images, sensor noise, and unclear instructions. The researchers tested seven state-of-the-art systems and found they all struggle significantly when inputs are corrupted, then demonstrated four strategies to make them more robust.

Julian Allagan, Mohamed Elbakary, Zohreh Safari et al.

Phishing detector robustness is fundamentally limited by feature economics—the cost of realistic website modifications—not by model architecture. Attackers can reliably evade detection by exploiting cheap feature changes, making feature design more critical than model choice.

This paper reveals a critical weakness in phishing detection systems: while machine learning models achieve near-perfect accuracy in testing, attackers can easily evade them by making cheap, realistic changes to websites.

Masoumeh Shafieinejad, Xi He, Mahshid Alinoori et al.

Synthetic data from diffusion models may not be as privacy-safe as assumed—membership inference attacks can still reveal whether specific records were in the training data, even with synthetic tabular outputs.

This challenge evaluates how well synthetic tabular data generated by diffusion models protects privacy against membership inference attacks. Researchers tested whether synthetic data truly hides information about individuals in the original dataset, developing new attack methods to measure privacy risks across different types of tabular data structures.

Zou Qiang

Adding explicit process-control layers to LLM reasoning—rather than just filtering outputs—can dramatically reduce hallucination and adversarial vulnerability by enforcing integrity at the reasoning stage itself.

Box Maze proposes a three-layer architecture for LLMs that separates reasoning into memory grounding, structured inference, and boundary enforcement to prevent hallucination and adversarial attacks. Testing on multiple LLM systems shows the approach reduces failure rates from ~40% to <1% under adversarial conditions, suggesting architectural constraints can improve reasoning reliability.

Zhan Jin, Yu Luo, Yizhou Zhang et al.

Using preference-based learning (DPO) with structural constraints rather than pixel-level metrics can fix a fundamental problem in medical image segmentation: producing fragmented, unrealistic vessel structures despite high accuracy scores.

ARIADNE combines vision-language models with reinforcement learning to detect coronary artery blockages in medical images while maintaining the correct structure of blood vessels. Instead of just matching pixels, it uses topological constraints to ensure vessel networks stay connected, reducing false alarms by 41% and achieving better accuracy on real clinical data.

Zikang Ding, Junchi Yao, Junhao Li et al.

Biases in LLMs can be reduced by enforcing structural consistency in the model's internal computations (attention and hidden states) across counterfactual inputs, rather than just fixing outputs or training data.

This paper proposes UGID, a method to reduce social biases in large language models by treating the model as a computational graph and enforcing that its internal structure remains consistent across inputs that differ only in sensitive attributes like gender or race.

Aravind Krishnan, Karolina Stańczak, Dietrich Klakow

Multimodal AI systems need safety defenses that account for attacks across all input modalities together—defending text alone or audio alone isn't enough.

This paper shows that spoken language models (which process both speech and text) can be attacked more effectively by perturbing both modalities simultaneously rather than just one. The researchers developed JAMA, a method that jointly optimizes adversarial text and audio to bypass safety guardrails, achieving 1.5x to 10x higher attack success rates than single-modality attacks.

Maksym Del, Markus Kängsepp, Marharyta Domnich et al.

For deploying reasoning models safely, combining verbalized confidence with self-consistency gives the best uncertainty estimates with minimal computational cost, but effectiveness varies significantly across domains like math versus humanities.

This paper studies how well reasoning language models can estimate their own uncertainty by sampling multiple responses and analyzing confidence signals.

Sheng Liu, Panos Papadimitratos

Federated learning for autonomous systems needs multi-layered defense: detect poisoned models at the neuron level, exclude malicious participants based on history, and actively repair the global model after removing attackers.

FedTrident protects federated learning systems for road condition classification from poisoning attacks where malicious vehicles deliberately mislabel their training data. The system detects compromised models through neuron analysis, removes bad actors, and uses machine unlearning to fix the corrupted global model—maintaining safety-critical performance even under attack.

Carlos Hinojosa, Clemens Grange, Bernard Ghanem

Vision-language models' safety decisions are easily manipulated by semantic cues—they rely on learned associations rather than grounded reasoning about actual danger, which is a critical vulnerability for real-world deployment.

This paper reveals that vision-language models make safety decisions based on surface-level visual and textual cues rather than genuine understanding of dangerous situations. Researchers created a benchmark and steering framework showing that simple changes to how a scene is described or presented can flip safety judgments, exposing a vulnerability in how these models assess risk.

Amine Lbath

Automated vulnerability injection with proof-of-concept exploits can scale up realistic training datasets for repository-level security detection, moving beyond function-level benchmarks to test how AI handles real-world code complexity.

This research creates an automated system to generate large-scale datasets for training AI models to detect software vulnerabilities in real code repositories.

Sadık Bera Yüksel, Derya Aksaray

You can enforce formal safety constraints on pretrained robotics models without retraining by adjusting their output distributions at inference time using temporal logic specifications.

This paper adds safety guardrails to robotics foundation models by reshaping their action distributions at runtime to satisfy formal specifications. Instead of retraining the model, it uses forward simulation to ensure the robot meets time-dependent constraints like "visit location A before time T, then location B" while staying as close as possible to the model's original decisions.

Chiara Manna, Hosein Mohebbi, Afra Alishahi et al.

Decoder-only language models show similar gender bias problems as smaller models in translation tasks, but instruction tuning can reduce masculine bias and improve context awareness.

This paper examines how large language models handle gender in machine translation, where languages differ in how they mark gender. The researchers introduce a new measurement called "Prior Bias" to capture what gender a model assumes by default, and test decoder-only models (like GPT-style architectures) against traditional encoder-decoder models.

Lingyu Li, Yan Teng, Yingchun Wang

LLMs can pass alignment tests while internally treating opposed moral concepts as equivalent; fixing this requires intervening directly on internal representations, not just adjusting outputs.

This paper reveals that large language models suffer from 'moral indifference'—they compress different moral concepts into similar internal representations, making them vulnerable to manipulation even when they appear aligned.

Felix Liedeker, Basil Ell, Philipp Cimiano et al.

Standard metrics for evaluating counterfactual explanations don't align with human judgment—developers need human-centered evaluation methods, not just algorithmic scores, to build truly trustworthy AI systems.

This study compares how AI systems measure counterfactual explanations (showing what would need to change for a different prediction) against how humans actually judge them. Researchers found that standard algorithmic metrics poorly predict human satisfaction, suggesting current evaluation methods miss what users actually care about in explanations.

Smriti Jha, Vidhi Jain, Jianyu Xu et al.

Deploying medical chatbots in low-resource, multilingual settings requires multiple layers of safety (triage, retrieval, generation) and multi-method evaluation—no single model or test is sufficient for trustworthy healthcare AI.

Researchers built a phone-based chatbot to answer maternal health questions in India, where users often have limited health literacy and speak multiple languages. The system combines triage (routing urgent cases to experts), retrieval of curated health guidelines, and AI-generated responses.

Siqi Sun, Ben Peng Wu, Mali Jin et al.

Chain-of-thought reasoning substantially reduces hallucinations in LLMs analyzing long, complex documents—a critical capability for compliance and legal applications where accuracy is non-negotiable.

ESG-Bench is a benchmark dataset for testing how well AI models understand long corporate ESG (environmental, social, governance) reports and avoid making up false information. The dataset contains real ESG reports paired with human-verified question-answer pairs, letting researchers measure when models hallucinate versus when they accurately extract facts.

Fengwei Tian, Payel Bhattacharjee, Heidi Hanson et al.

By combining task-aware importance scoring with privacy sensitivity detection, STAMP achieves better privacy-utility trade-offs than uniform noise approaches—meaning you can protect sensitive data without sacrificing model performance.

STAMP is a privacy framework that protects sensitive information in text while keeping it useful for AI tasks. It smartly decides which parts of text need more protection (like names and dates) versus which parts are less sensitive, then applies targeted noise to embeddings using a novel 'polar mechanism' that preserves semantic meaning better than traditional approaches.

Ninghui Li, Kaiyuan Zhang, Kyle Polley et al.

AI agents introduce fundamentally new security challenges because they blur the line between code and data, and can execute actions across systems—developers need layered defenses including input filtering, sandboxing, and strict privilege controls.

This paper identifies security risks in AI agents—systems that can take actions in the real world—and proposes defenses. It covers new attack types like prompt injection and confused-deputy problems, explains how current protections work (sandboxing, policy enforcement), and highlights gaps in standards and research needed to secure multi-agent systems.

Alexandre Le Mercier, Thomas Demeester, Chris Develder

CLASP provides a practical, lightweight defense against poisoning attacks on state space models by detecting malicious tokens before they reach downstream tasks, with strong generalization to unseen attack patterns.

State space models like Mamba are fast alternatives to Transformers, but they're vulnerable to Hidden State Poisoning Attacks that inject malicious tokens to corrupt the model's memory.

Abhinaba Basu, Pavan Chakraborty

ML models for materials science need formal safety audits—this work shows single models have severe blind spots, but systematic falsification and confidence bounds can identify reliable predictions and improve discovery by 25%.

Machine-learned models for predicting material properties often fail silently. This paper introduces Proof-Carrying Materials, a system that audits these models through adversarial testing, statistical confidence bounds, and formal verification to identify which predictions are trustworthy.

Hyungyung Lee, Hangyul Yoon, Edward Choi

AI medical diagnosis becomes more trustworthy when it shows its evidence instead of just giving answers.

This paper presents CXReasonAgent, a system that helps AI diagnose chest X-rays by combining a language model with specialized medical tools. Instead of just guessing answers like typical AI models, it shows its work by pointing to specific evidence in the image.

Haotian Zhai, Elias Stengel-Eskin, Pratik Patil et al.

AI research agents are unreliable in production because of randomness in how they search, summarize, and reason—but you can fix this with structu...

Research agents that gather information to answer questions produce different results each time you run them with the same question. This paper identifies where that randomness comes from and proposes ways to make these systems more reliable—reducing variability by 22% while keeping quality high.

Jiangxin Sun, Feng Xue, Teng Long et al.

Autonomous driving systems can make safer decisions in unexpected situations by predicting consequences and evaluating risk, rather than just copyi...

This paper tackles a critical problem in autonomous driving: current AI systems learn by copying expert drivers, but fail when encountering unusual situations they've never seen before. The researchers propose RaWMPC, a system that predicts what will happen if the car takes different actions, then picks the safest option—without needing expert examples.

Yegon Kim, Juho Lee

Separate the model that solves problems from the model that explains them to avoid accuracy loss when making AI outputs verifiable.

When AI models need to show their work so humans can verify it, they often get worse at solving problems—a cost called "legibility tax." This paper fixes that by splitting the job: one model solves the problem correctly, then a second model rewrites the solution in a way that's easy to check. This avoids forcing one model to juggle both accuracy and explainability.

Radha Sarma

RLHF-based AI systems cannot be governed by norms because optimization forces all values into tradeable weights—genuine norm-following requires a...

This paper argues that AI systems like ChatGPT trained with RLHF cannot follow ethical rules or norms because of how they're built. They work by turning everything into a single score and picking the highest one—which means they'll always trade off any principle if it scores higher. The author shows this isn't a bug to fix, but a fundamental limit of optimization itself.

Thomas Woergaard, Raghavendra Selvan

Compressing models for efficiency can accidentally increase bias—you need to monitor fairness metrics during compression, not just overall accuracy.

This paper tackles a hidden problem in model compression: when you shrink neural networks to run faster, the compression can unfairly hurt accuracy for certain groups of people (like underrepresented skin tones in medical imaging).