Papers

Narges Ahmadi, Yubo Jiao, Jônatas Augusto Manzolli et al.

Large language models can match or exceed traditional machine learning for travel behavior prediction without task-specific training, and adding visual context from survey images improves performance—showing that multimodal AI can enhance behavioral modeling when integrated with human-centered ...

This paper presents a three-agent workflow that combines chatbot surveys, data processing, and prediction to model how weather affects commuter mode choices. The system collected 454 survey responses about travel preferences across different weather scenarios, then compared traditional statistical models with nine different large language models (2-35B parameters) for predicting travel behavior.

Zhenyao Cui, Siyuan Kan, Siyang Li et al.

Brain-based image retrieval can work for new users without retraining by learning to recover the geometric transformation between their brain's coordinate system and a shared visual space, using only unlabeled test-time alignment.

This paper tackles cross-subject EEG-to-image retrieval—retrieving images that match brain signals from new users without labeled training data. The key insight is that different people's brains organize visual concepts similarly but along different coordinate directions.

Zhenyao Cui, Siyuan Kan, Dingkun Liu et al.

Brain-to-image decoding can work with far fewer repetitions by anchoring both neural and visual representations to a shared reference point, rather than just denoising the query alone.

This paper tackles brain-to-image retrieval with limited neural data. Current methods require averaging 80+ brain scans per image, but the authors show that low-repetition queries fail not just due to noise, but because brain and image representations misalign.

Lei Bai, Jiaqi Cao, Chiyu Chen et al.

This work demonstrates how to build AI agents for science by combining multimodal pre-training with agentic reinforcement learning and memory-augmented architectures, achieving strong performance on scientific reasoning and forecasting tasks without requiring task-specific model modifications.

Intern-S2-Preview is a large multimodal AI system designed to tackle scientific discovery tasks by reasoning over diverse data types, using scientific tools, and working across long-horizon problems.

Yi-Chung Chen, Philip Jacobson, Tom Lampo et al.

Using physical trajectory data as training supervision helps video embedding models better understand motion-centric driving events, improving retrieval accuracy by 5-10% while keeping inference simple and efficient.

This paper tackles retrieving relevant driving video clips from large datasets by improving multimodal embedding models. The key innovation is TraVEL, which fine-tunes video embeddings using trajectory (vehicle motion) as training supervision, helping the model understand motion-centric events like turning or accelerating.

Daniel Perkins, John Squires, Janou Milligan et al.

Instead of training separate models for each domain, you can use an MLLM router to dynamically select which vision backbone handles each image, getting both better generalization and easier updates without retraining.

ARMDIL uses a multimodal language model to intelligently route images to the best-suited vision model (CNNs, self-supervised learners, or vision-language models) within an ensemble.

Serli Kopar, Sam Gijsen, Abner Hernandez et al.

Speech-based disease detection models may not be learning genuine disease characteristics but rather dataset-specific patterns, raising serious concerns about their reliability for real-world clinical use.

This paper investigates whether speech models trained to detect Parkinson's disease actually learn disease-specific patterns or just exploit dataset quirks.

Muhammad Hannan Akram, Muhammad Abubakar Rashid, Wassi Haider Kabir et al.

Heterogeneous AI agents can efficiently synchronize their understanding in distributed networks by translating belief updates through edge-deployed models, without requiring shared training or identical architectures.

This paper proposes a framework for AI agents in 6G networks to synchronize their understanding (beliefs) despite using different AI models and operating under different constraints.

Chuyue Li, Jinpeng Yu, Haozhe Wang et al.

Agents can now learn from and generate high-quality videos by working with structured knowledge graph representations instead of raw pixels, improving video generation quality by 20.7% over existing methods.

AVA-Encoder learns video representations as knowledge graphs that agents can reason about and edit. It converts videos into structured text and asset layers, then reconstructs videos from these representations. A natural-language feedback loop optimizes the encoding, enabling agents to work with cinematic-quality videos more effectively.

Yan Deng, Fei Xu

For embodied AI agents navigating from visual instructions, explicitly modeling temporal context (past-only memory), multi-step planning with single-step execution, and decoupled termination detection significantly improves navigation success and efficiency.

DreamFly improves aerial drone navigation by combining three key techniques: a causal memory system that uses only past observations to avoid information leakage, a receding-horizon planning approach that predicts multiple future actions but executes one at a time, and explicit stop detection from action predictions.

Pedro Sousa, Will Tebbutt, Sadiq Jaffer et al.

Foundation models trained on satellite imagery can capture persistent surface properties (terrain, vegetation, water) that explain local weather variations better than hand-crafted features, enabling more accurate probabilistic weather predictions at arbitrary locations.

This paper shows that Earth observation embeddings from satellite data can improve weather downscaling—predicting local weather at specific locations from coarse global weather models.

Weihao Bo, Shan Zhang, Yanpeng Sun et al.

Current multimodal AI models excel at understanding diagrams visually but struggle significantly with converting them to executable code, suggesting this is a critical gap to address for scientific writing tools.

Diagram-MMU is a benchmark with 3.7k scientific diagrams and 18.3k questions that tests how well AI models can understand diagrams and convert them to code. The benchmark evaluates 12 models on three tasks: turning diagrams into LaTeX code, editing diagram code, and answering questions about diagrams—revealing that code generation is much harder for models than visual understanding.

Changhao Xiang, Shangyu Xing, Zhen Wu et al.

By interleaving visual objects directly into text during pretraining, you can teach multimodal models object-level grounding 12x more efficiently than traditional image-text pair training.

This paper introduces MultiModal Code-Switching (MMCS), a new way to train vision-language models by replacing words in text with their corresponding visual objects. Instead of just pairing whole images with descriptions, MMCS explicitly shows the model which objects match which words, making training much more efficient—achieving the same performance with 12x less data.

Oluwanifemi Bamgbose, Simon Rosen, Jash Shah et al.

Automated TTS evaluators need to move beyond generic 'naturalness' scores and evaluate specific linguistic dimensions of speech quality, as current tools miss many errors that human listeners easily detect.

This paper reveals that current automated Text-to-Speech evaluation tools (MOS predictors and Audio-LLM judges) fail to capture the full range of speech quality issues that humans perceive.

Hunar Batra, Lachin Naghashyar, Ashkan Khakzar et al.

Sparse autoencoders can turn multimodal model internals into controllable feature interfaces: you can identify what changed during multimodal training, find features causing specific behaviors, and steer or remove them to improve safety or task performance.

This paper introduces MMDiff, a framework that uses sparse autoencoders to decompose multimodal language models into interpretable features.

Diandian Zhang, Tingyu Song, Lin Fu et al.

Visual realism in video generation doesn't guarantee scientific correctness: models that look good often fail at capturing accurate scientific and causal dynamics, a gap that needs targeted evaluation and improvement.

Sci-VBench is a benchmark with 1,253 expert-annotated examples for evaluating how well AI models generate videos that require scientific knowledge and reasoning across 60 subjects in science, healthcare, humanities, and engineering.

Sarvesh Baskar, Zikui Cai, Shayan Shabihi et al.

Video models fail not because they can't see events, but because they can't reliably track and count them over time—adding more frames helps slightly but doesn't fix the core temporal reasoning problem.

Video language models struggle with counting events in videos, especially when events happen frequently or repeatedly.

Zhiheng Wang, Bo Peng, Lai Wei et al.

Visual tool-use in multimodal models often creates an illusion of improvement: while aggregate accuracy may increase, the returned visual evidence frequently has no causal effect on the final answer, making these expensive operations ineffective for most queries.

This paper investigates whether multimodal AI models actually benefit from visual tool-use operations like cropping and zooming. Using causal analysis, the authors discover that these operations often don't meaningfully improve answers despite higher computational cost—models either ignore the visual evidence they retrieve or use it incoherently.

Yuanjing Xu, Xinyan Liu, Weidong Chen et al.

Adding depth information significantly improves object counting in crowded scenes—the method reduces counting errors by 62% compared to RGB-only approaches, showing that 3D spatial cues are crucial for handling occlusions.

This paper tackles counting objects in crowded, occluded video scenes by combining RGB images with depth information. The method uses a depth-guided detector with cross-attention between color and depth data, plus occlusion prediction, to better identify individual objects even when they overlap. The authors also release a new RGB-D dataset for this task.

Christo Kurisummoottil Thomas, Omar Hashash, Walid Saad

Networks can become active intelligence coordinators for physical AI by deploying reasoning agents (digital twins) that share spatiotemporal context through causal reasoning and transmit only beliefs with cognitive value, rather than optimizing for throughput alone.

This paper proposes holonic digital twins (HDT-Nets)—intelligent network agents that actively reason about their environment rather than passively mirroring physical systems.

Ayoub Kirouane, Christos Petrocheilos

Domain-specific fine-tuning dramatically improves retrieval and generation for underrepresented languages: a modest 1B embedder outperforms multilingual models when trained on just 65K in-domain Greek examples, showing that language adaptation is more important than model size for specialized tasks.

This paper adapts NVIDIA's Nemotron retrieval system for Modern Greek across legal, energy, financial, and medical domains. The authors mine Greek corpora, train specialized retrieval models, and create HERA—the first large-scale Greek RAG benchmark.

Aniri, Jinhe Bi, Peng Liao et al.

Modality imbalance (text drowning out vision) is a real bottleneck in multimodal reasoning. By detecting when visual input is being ignored and training only on those critical tokens, you can make self-distillation much more effective.

This paper identifies and addresses modality imbalance in multimodal language models—where text dominates over visual information during reasoning. OPD-V uses positive and negative teacher models with modified images to detect when the model isn't properly using visual input, then selectively applies self-distillation only on tokens where visual information matters most.

Yuezhang Peng, Yuxin Liu, Changfeng Gao et al.

Framing spoken language understanding as structured function calling—rather than slot-filling—lets audio language models generalize to new tasks without retraining, similar to how code models handle function calls.

This paper introduces Spoken Function Calling (SFC), a new way to understand spoken language that treats semantic extraction like function calls with structured definitions.

Yang Yang, Qinyu Zhao, Mouxiang Chen et al.

By sharing backbone parameters across parallel branches with task-specific computation allocation, you can improve multimodal model performance without increasing model size or inference latency.

ParVL introduces a framework for scaling multimodal AI models by running multiple parallel vision and language processing branches that share the same core parameters. Instead of making models bigger or slower, it reuses existing components more efficiently and lets different tasks use different amounts of vision vs.

Zhen Fang, Yu Zeng, Wenxuan Huang et al.

Multimodal agents need explicit architectural constraints to use visual tools and external knowledge rather than defaulting to text search and internal memory; decoupled perception-exploration pipelines with staged tool unlocking significantly improve performance.

Video-DeepResearch extends multimodal AI agents to handle continuous video streams with web search integration. The system addresses two key problems: agents ignoring visual information in favor of text search, and relying on memorized knowledge instead of actually using tools.

Yao Xiao, Reuben Tan, Zhen Zhu et al.

A single trainable embedding can dramatically improve how vision-language models handle long visual context by acting as a retrieval target—selecting sparse, relevant tokens instead of processing everything, making it practical for real-world deployment.

ReToken is a lightweight technique that adds a single learnable token to vision-language models to intelligently select relevant visual information from long images and videos.

Kangning Zhang, Yixing Li, Shuai Shao et al.

By attributing teacher corrections to visual evidence through counterfactual intervention, VAD extracts purer visual supervision signals for distillation, outperforming naive privileged-view supervision that mixes visual and linguistic signals.

This paper addresses a key problem in multimodal distillation: when a teacher corrects a student's mistakes, it's unclear how much of that correction comes from visual evidence versus linguistic priors. VAD solves this by using counterfactual reasoning—removing visual evidence and measuring how the teacher's predictions change—to isolate visually-grounded corrections.

Jiacheng Tao, Qingyun Sun, Haonan Yuan et al.

Separating global structural reasoning from fine-grained evidence matching into two graph tiers significantly improves retrieval accuracy and reasoning quality in multimodal QA systems.

DualG-MRAG improves multimodal retrieval-augmented generation by using two separate graph layers: a macro-graph for global reasoning and a micro-graph for precise evidence matching. This decoupled approach reduces retrieval noise while preserving fine-grained details, and uses graph neural networks to propagate relevance across text and images for better complex question answering.

Jia Yu, Yan Zhu, Yili He et al.

Routine clinical reports contain rich supervision signals—you can extract finding-to-frame correspondence at scale to train medical vision-language models without expensive frame-level annotation, turning existing documentation into training data.

EndoCLIP is a vision-language model trained on 280,000 colonoscopy reports to link clinical findings with individual video frames.

Roshan Kenia, Stephanie L McNamara, William Lotter

You can efficiently adapt pre-trained medical imaging models to understand anatomy-level details by decomposing scans into regions, learning how they relate to each other, and aligning them with medical text—without retraining from scratch.

This paper presents Anatomy Contextualized Adaptation (ACA), a lightweight method that improves CT scan foundation models by aligning fine-grained anatomical regions with text descriptions from radiology reports while preserving whole-scan context.

Francesco Pinto, Luca Lanzilao, Paco Lopez Dekker et al.

Satellite scatterometer data can be used directly for short-term wind forecasting via deep learning, achieving 23% better accuracy than traditional weather models at 1-hour lead times—opening a new approach to renewable energy forecasting.

WindCastNet is a machine learning model that forecasts offshore wind speed and direction using satellite scatterometer data instead of traditional weather models. It handles irregular satellite observations from multiple sources and predicts wind fields 1-2 hours ahead, outperforming standard numerical weather prediction models for short-term forecasts critical to offshore wind farm operations.

Syed Mhamudul Hasan, Anas AlSobeh, Hussein Zangoti et al.

Combining multimodal inputs (images + symptoms) with structured workflow management and safety checks significantly improves disease screening accuracy compared to image-only predictions, and this pattern applies beyond veterinary use.

VetClaw is an edge-cloud system that helps veterinarians screen animals for diseases by combining camera images with symptom descriptions. It uses a vision-language model on a server and intelligent workflow management to collect evidence, validate inputs, apply safety rules, and alert users when cases need escalation—turning a simple image classifier into a practical diagnostic tool.

Ankang Yang, Jitao Zhao, Di Jin et al.

By organizing multimodal graph information into hierarchical contexts before feeding it to an LLM, CHARM achieves zero-shot transfer across different graph domains—meaning it can work on new graphs without any labeled training data from those domains.

CHARM is a foundation model for graphs that combines text, images, and other data types without needing labeled examples in new domains. It uses hierarchical context around each node to capture relationships across different data types, then converts these into tokens that a large language model can understand.

Hangjie Yuan, Yichen Qian, Zhiwei Tang et al.

Building effective medical AI requires vision-first design: specialized image encoders for diverse medical data types and evaluation metrics that measure clinical accuracy, not just fluency.

ClinFusion is a medical AI system that combines vision and language to understand medical images and generate clinical reports. It uses a specialized architecture to handle both 2D and 3D medical images, and includes new evaluation methods that align with how radiologists actually assess medical findings.

Krithi Shailya, Ananya Lakshmi Ravi, Venkatanathan K. V. et al.

Using mathematically interpretable neural network architectures (KANs) as the foundation for medical AI explanations produces more trustworthy and faithful results than adding language models on top of opaque vision models.

KANEx uses Kolmogorov-Arnold Networks (KANs)—models with interpretable spline-based components—to improve medical AI explainability. Instead of relying on black-box vision models paired with language models, the framework grounds textual explanations in KAN's transparent mathematical structure and introduces KAN-Map for more faithful visual explanations.

Ali Ansari, Yasmin Mohammadi, Farnoush Nili et al.

Vision-language models can extract simple database schema elements from diagrams but fail on advanced ER constructs—this benchmark reveals critical gaps in multimodal understanding of structured technical diagrams that matter for AI-assisted database engineering.

ERUnderstand is a benchmark dataset of 2,960 Entity-Relationship diagrams that tests how well AI vision-language models can understand database schemas from images. The researchers found that while models handle basic diagram elements well, they struggle significantly with complex features like weak entities and N-ary relationships, even when augmented with reasoning capabilities.

Francisco Mena, Dino Ienco, Roberto Interdonato et al.

Co-learning between modalities is more robust than fusion-based approaches when dealing with unpredictable missing modalities at inference time, with different strategies optimal for mild versus extreme missing conditions.

This paper tackles multi-modal classification when some data sources are unavailable at inference time—a common real-world problem from sensor failures or privacy constraints. Instead of designing better fusion methods, the authors use co-learning where modalities teach each other.

Vedant Shah, Onkar Susladkar, Tushar Prakash et al.

Using structured graphs to represent object relationships provides a more scalable and precise way to control video generation than trajectory drawing or text prompts, especially for complex multi-object scenes.

GraphVid enables precise control over multi-object interactions in video generation by using structured interaction graphs instead of text or pixel-level motion inputs. The method outperforms existing motion-control approaches while using less training data, and the authors release GraphVid-Bench, a new dataset with relational annotations for interaction-aware video generation.

Aaron Feller, Kris Deibler, Maxim Secor

Modeling molecules as conformational ensembles rather than single structures substantially improves property prediction—pretraining on ensemble data is critical, as the model fails from scratch but reaches strong performance when pretrained.

This paper introduces EnsembleEGNN, a neural network model that predicts molecular properties by learning from multiple 3D structures (conformations) of cyclic peptides simultaneously, rather than just one.

Yijun Liang, Yunjie Tian, Yijiang Li et al.

VCSD achieves significant improvements in vision-language model performance (up to 5% on benchmarks) by using contrastive image conditioning during self-distillation, requiring no external teachers, privileged data, or extra inference costs.

This paper introduces Visual Contrastive Self-Distillation (VCSD), a training method that improves vision-language models by having them learn from themselves without needing an external teacher.

Wen Ye, Yuxiao Qu, Aviral Kumar et al.

Multimodal models can learn better by having strong views teach weaker views on the same problem—using the model's own best performance as supervision rather than external labels.

Vision-language models struggle with visual reasoning even when problems have equivalent text and diagram versions. This paper shows different views (text, image, combined) expose different reasoning paths and failure modes.

Dongjie Fu, Di Cao, Xize Cheng et al.

Audio-language models can learn complex reasoning by generating their own reasoning trajectories while receiving token-level guidance from text teachers, without requiring massive amounts of audio reasoning data.

This paper presents X³-OPD, a method for teaching audio-language models to reason better by learning from text-based models.

Wael AbdAlmageed

You can now train neural networks that perform logical reasoning directly on images without losing gradient information at the perception-to-logic boundary—the system learns to ground visual facts and apply knowledge graph rules in one differentiable pass.

SoftReason bridges perception and symbolic reasoning by making deductive logic fully differentiable. Instead of converting images to discrete symbols (which breaks gradient flow), it represents reasoning as soft probability tensors over candidate facts and predicates, allowing end-to-end training from pixels through logical deduction.

Aditi Gupta, Yossi Gandelsman

Vision-language models have a circuit-level sensitivity to input order that can be fixed with simple test-time adaptation, improving both consistency and accuracy without retraining.

Vision-language models perform differently depending on whether you show them the image or question first—a semantically irrelevant change that shouldn't matter. This paper identifies this "modality order" failure and proposes a test-time training method that makes models consistent across both orderings while improving overall performance.

Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch et al.

Appearance pointers provide a modality-agnostic way to add regional control to existing Diffusion Transformers, letting you specify exactly where text descriptions or image references should apply in the generated output without expensive retraining.

This paper introduces appearance pointers, a new technique for controlling where and how text and image inputs influence image generation in Diffusion Transformers.

Ingo Ziegler, Martin Krebs, Desmond Elliott

Token, byte, and pixel encodings each have different strengths depending on your task and capacity constraints—there's no universally best choice, only task-specific tradeoffs.

This paper compares three ways to encode text for language models—tokens, bytes, and pixels—by controlling both the linguistic content and model capacity. Using parallel sentences across 13 languages, the researchers measure how well each encoding preserves information under compression.

Mingfei Chen, Zijun Cui, Ruoke Zhang et al.

By pairing semantic understanding with explicit 3D spatial modeling in a unified representation, SceneBind enables cross-modal scene understanding and retrieval that works across vision, audio, and language without requiring task-specific training.

SceneBind is a multimodal AI system that understands scenes by combining what things are (semantics) with where they are (3D spatial location) across vision, audio, and language. Unlike existing systems that focus mainly on identifying objects, SceneBind explicitly models spatial relationships and uncertainty, enabling better scene retrieval and object localization across different modalities.

Sushant Gautam, Vajira Thambawita, Michael A. Riegler et al.

High leaderboard performance on medical AI tasks doesn't mean the system reasons correctly—focus on explainability, evidence grounding, and robustness checks to build trustworthy healthcare AI.

This paper examines nine AI systems for medical visual question answering on endoscopy images, finding that high benchmark scores don't guarantee reliable clinical reasoning.

Yazhi Zhang, Fuqiang Niu, Bowen Zhang

This dataset enables training multimodal models to detect political stance in short-form video discussions while accounting for conversational context—a capability previously limited by lack of labeled data combining video, audio, and nested comment structures.

TikStance is a dataset of 13,876 comments on 161 TikTok videos about three 2024 U.S. political figures, labeled for stance (Favor/Against/None) with both video and comment-level annotations. It preserves audio, video, and hierarchical comment threads to enable research on how people express political opinions across short-video platforms.

Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier et al.

You can now automatically detect recurring caption errors tied to specific visual patterns in MLLM outputs, helping identify and fix systematic biases in vision-language datasets without needing model access.

This paper introduces Symbal, a method to detect systematic errors in image captions generated by multimodal AI models—cases where the same type of mistake repeatedly occurs with specific visual features.

Shaoxiong Zhan, Shi Hu, Boyu Feng et al.

Visual evidence in bug reports is largely ignored or unhelpful for current AI systems—the benchmark reveals that adding images doesn't improve localization performance, suggesting systems need better multimodal reasoning to leverage visual clues effectively.

MM-IssueLoc is a benchmark for evaluating how well AI systems locate bugs in code repositories using both text and visual evidence like screenshots and error dialogs. It contains 652 real issues across 23 programming languages with structured annotations, and tests whether systems actually benefit from images or just rely on text.

Patrick Phuoc Do, Chau M. Ta, Chaoli Wang

Current MLLMs show uneven and often poor understanding of scientific visualizations, particularly failing at quantitative tasks and complex visualization types, suggesting that SciVis literacy should be a core evaluation criterion for multimodal AI systems.

This paper evaluates six multimodal large language models (MLLMs) on their ability to understand scientific visualizations using a standardized 49-item assessment test.

Sarthak Jain, Qiran Hu, Zhen Zhu et al.

By interpolating between old and new model checkpoints with learned per-parameter scalars, you can adapt multimodal models to new data while preserving earlier knowledge—all without changing model size or inference speed.

AlphaWiSE solves a key problem in continual learning for multimodal models like CLIP: when models adapt to new data sequentially, they forget what they learned before. This method blends two frozen model checkpoints using learned scalar weights per parameter, maintaining cross-modal alignment without extra computation cost.

Mustafa Chasmai, Vincent Dumoulin, Jenny Hamer

Adding metadata supervision (location, time, etc.) to bioacoustic models improves species detection and helps models handle domain shifts—showing that non-audio information from citizen science platforms can significantly boost real-world performance.

MetaPerch is a bioacoustic foundation model that uses recording metadata like location and time as additional training signals alongside audio. By leveraging species-habitat correlations in the metadata, the model learns more robust representations that better generalize to real-world acoustic monitoring scenarios with different environments and recording conditions.

Harsha Vardhan Khurdula, Abhinav Kumar Singh, Yoeven D Khemlani et al.

Parallel decoding via discrete diffusion is viable for speech recognition and can match autoregressive performance while being faster and more efficient, using only 0.16% trainable parameters on a frozen backbone.

This paper shows that discrete diffusion language models can transcribe speech by refining entire transcripts in parallel rather than generating one token at a time. The authors adapt DiffusionGemma (a 26B model) for speech by freezing most weights and adding a lightweight audio interface, achieving competitive accuracy while transcribing in just eight parallel steps regardless of speech length.

Ruoran Xu, Wending Gao, Qiufeng Wang

By using formal languages as an intermediate representation between language models and geometric rendering, this framework achieves high-precision diagram generation for math problems without human annotation—a scalable approach for creating training data in underexplored domains.

FormalAnalyticGeo is a framework that automatically generates multimodal analytic geometry problems by combining formal languages with neural models.

Shijie Wang, Honglu Zhou, Ziyang Wang et al.

Video language models can give right answers for wrong reasons; this work forces models to provide verifiable visual evidence (tracked object masks + time segments) alongside answers, revealing and fixing a critical gap between QA accuracy and actual visual understanding.

This paper introduces Evidence-Backed Video Question Answering (E-VQA), which requires video AI models to not just answer questions but also show their work by providing temporal segments and pixel-level object tracking masks as visual proof.

Divya Mereddy, Jeevan Beedareddy

LoRA-based cascaded fusion enables efficient multimodal action recognition in healthcare settings without retraining, making it practical for medical training environments with varying data sources.

This paper proposes a parameter-efficient framework for recognizing actions and activities in medical training videos by combining multiple data types (video, audio, etc.) using LoRA—a technique that adapts pre-trained models with minimal new parameters.

Kaixin Ma, Di Feng, Alexander Metz et al.

Visual tool-calling agents fail primarily due to poor image understanding, not planning: 53% of failures come from misreading images despite correct task logic. This suggests different improvement strategies are needed for smaller vs. larger models.

MM-ToolSandBox is a benchmark for evaluating AI agents that use images to call tools and complete tasks. It includes 500+ tools across 16 domains, 258 test scenarios with images, and a stateful execution environment. Testing 12 models reveals that even the best achieve below 50% success—with visual perception (extracting correct info from images) being the main bottleneck, not task planning.

Yu-Han Huang, Chih-Kai Yang, Ke-Han Lu et al.

Targeting individual neurons inside the audio encoder at inference time is more effective for improving acoustic perception than post-encoder interventions, suggesting that the encoder itself is where acoustic information gets lost in large audio-language models.

This paper introduces IAAN, a training-free method that identifies and amplifies specific neurons in audio encoders of large audio-language models to improve perception of fine-grained speech attributes like emotion.

Seung Hyun Hahm, Minh T. Dinh, SouYoung Jin

For accessibility in long-form video, maintaining narrative context across scenes is as important as describing individual moments—this training-free approach shows how to build story-aware systems without task-specific fine-tuning.

StoryTeller generates audio descriptions for long films by maintaining a narrative memory across scenes, ensuring descriptions preserve character identities, plot connections, and story context.

Xiaokai Bai, Lianqing Zheng, Runwei Guan et al.

Treating occupancy prediction as an evolving scene state rather than a final output enables better radar-camera fusion for autonomous driving, improving both object detection and dense scene understanding simultaneously.

This paper presents 4DR360, a radar-camera fusion system for autonomous driving that jointly detects 3D objects and predicts scene occupancy in 360° views. Unlike prior methods that treat these tasks separately, it models occupancy as a persistent scene state that's refined across processing stages, using radar's Doppler information to maintain temporal consistency.

Nirjhar Das, Md. Al-Mamun Provath

Task-specific reasoning strategies and confidence calibration can outperform complex retrieval-based approaches on multimodal QA benchmarks, especially when operating under efficiency constraints.

This paper presents a task-specific multimodal question-answering system for the QANTA 2026 challenge that answers quiz questions from incrementally revealed text and images.

Thanh-Hoang Nguyen Doan

Practical sign language translation requires co-designing the ML model with the deployment infrastructure—optimizing latency through system-level techniques (streaming, parallelization, state machines) matters as much as model accuracy for real-time applications.

This paper tackles real-time sign language translation at the sentence level by fine-tuning a SHuBERT-ByT5 model on a subset of How2Sign data using QLoRA.

Xinyan Chen, Ziyu Guo, Renrui Zhang et al.

Video generation can be a reasoning mechanism: training models on diverse temporal reasoning tasks and adding explicit reasoning tokens improves their ability to solve logical problems by generating step-by-step visual explanations.

OpenCoF introduces a dataset and fine-tuned video model designed to teach AI systems to reason through generating sequences of video frames. Unlike text-based reasoning, this 'Chain-of-Frame' approach lets models unfold logical steps visually across time. The work shows that video models trained on diverse reasoning tasks with special reasoning tokens perform better at solving complex problems.

Siddharth Damodharan, Radhika Gupta, Ali Alshami et al.

Current vision-language models struggle with safety-critical reasoning in autonomous driving; this benchmark provides a standardized way to measure whether they can understand incident context and predict avoidability.

AUTOPILOT-VQA is a benchmark dataset for evaluating vision-language models on safety-critical dashcam understanding. It uses structured questions about real-world driving incidents to test whether AI systems can reliably reason about weather, traffic, road conditions, and accident scenarios—moving beyond simple object recognition to temporally grounded, safety-aware reasoning.

Kaifeng Zhao, Mathis Petrovich, Haotian Zhang et al.

For interactive applications, ARDY trades off some offline generation quality to achieve real-time streaming motion synthesis with flexible text and kinematic control, making it practical for animation tools and robotics where responsiveness matters as much as precision.

ARDY is a real-time motion generation system that creates realistic 3D human animations from text prompts and pose constraints.

Ayda Eghbalian, Kevin Desai

You can now add biomechanical analysis (forces, activations, loads) to any existing 3D pose estimator without retraining it, making markerless motion capture useful for rehabilitation, sports, and clinical applications.

This paper introduces BioModule, a lightweight transformer that converts 3D skeletal poses from any pose estimator into biomechanical quantities like joint forces and muscle activation. By aligning Human3.6M video data with biomechanical labels, the authors show how pose estimation errors propagate to biomechanical predictions across seven different pose estimators.

Chen Tang, Yizhou Wang, Jianyu Wu et al.

By making molecular and crystal structures inspectable during reasoning, SciReasoner achieves state-of-the-art predictions while generating human-readable explanations—showing how AI can solve chemistry and materials problems while remaining scientifically interpretable.

SciReasoner is a multimodal AI model that reasons about structure-property relationships in proteins, molecules, and crystals by treating structural information as interpretable evidence. It discretizes 3D coordinates and chemical topologies into a unified vocabulary, enabling transparent predictions grounded in scientific principles like bonding and symmetry.

Hyunjae Kim, Dain Kim, Pan Xiao et al.

High-fidelity data curation from scientific literature can create large-scale medical multimodal datasets that rival or exceed models trained on much larger datasets, enabling better medical AI without requiring new data collection.

MedPMC is a framework that automatically extracts and curates 11 million high-quality medical image-text pairs from 6.1 million PubMed Central articles. The resulting dataset trains multimodal models that significantly outperform existing biomedical baselines on medical imaging tasks, with 95.3% of extracted images validated as medically relevant by human reviewers.

Tianjiao Yu, Xinzhuo Li, Yifan Shen et al.

By routing text-to-geometry interactions across matched abstraction scales rather than flattening everything into one representation, you can build faster, more accurate 3D models that understand both coarse structure and fine details.

ELSA3D is a unified 3D foundation model that improves how language and 3D geometry interact by using 'anchor tokens' to match text concepts with the right level of geometric detail. Instead of treating all information equally, it routes language features to specific scales of 3D structure, making the model both more efficient and more accurate for generating and understanding 3D objects.

Zhenyu Liu, Yunxin Li, Xuanyu Zhang et al.

Modality interference—caused by gradient conflicts between audio and semantic processing—is the root cause of poor full-duplex SLM performance; hierarchical parameter separation solves this while maintaining cross-modality coherence.

This paper identifies and solves a critical problem in full-duplex spoken language models: when audio and text processing share the same neural network layers, they create conflicting gradients that degrade performance.

Han-Jun Ko, Jr-Jen Chen, Haobo Yuan et al.

Grounding VLM reasoning directly to visual observations and action consequences—rather than letting models generate free-form explanations—significantly improves physical reasoning generalization and reduces hallucination.

This paper tackles a key problem in vision-language models: they hallucinate reasoning that contradicts physics and misalign their explanations with actual actions. VAORA introduces two reward signals that anchor model reasoning to visual evidence and action outcomes, helping VLMs learn grounded physical reasoning that generalizes to new tasks and environments.

Wenhao Li, Xueying Jiang, Quanhao Qian et al.

Robot policies can achieve view robustness without camera calibration by learning to predict both action in camera space and camera-to-robot geometry, making deployment more practical when camera positions vary.

This paper introduces CamVLA, a robot vision-language-action model that learns to figure out camera positioning automatically instead of requiring explicit calibration. By predicting both camera-relative actions and the geometric relationship between camera and robot, the model works with any camera setup without needing depth data or prior calibration.

Haozhe Wang, Weijia Feng, Jinpeng Yu et al.

Visual generators need to learn *when* to search for external knowledge, not just *how* to use it—and this knowledge boundary is discoverable through co-training, not fixed in advance.

This paper identifies a critical gap in visual generators: they confidently create incorrect images for requests about new entities, trending topics, and post-training events. The authors show that naive search-augmentation fails because generators have an evolving 'knowledge boundary'—a threshold between what they learned and what needs external context.

Xuehui Wang, Xuankun Yang, Wei Shen

When pruning visual tokens in VLMs, filtering textual noise with entropy and selecting tokens as a structured optimization problem (not just picking top-K) preserves fine-grained details better while reducing computation.

This paper tackles the problem of compressing image tokens in vision-language models (VLMs) while preserving important visual details. The authors identify that existing pruning methods fail because textual noise corrupts the scoring process and selected tokens become fragmented.

Junhao Shi, Siyin Wang, Xiaopeng Yu et al.

Separating motor skill learning from language grounding dramatically reduces the labeled data needed for robot learning—TAP matches models trained on 1M+ expert trajectories while using far less labeled data and shows better robustness to real-world perturbations.

This paper proposes Task-Agnostic Pretraining (TAP), a two-stage approach for training Vision-Language-Action robots that separates learning how to move (from unlabeled robot interactions) from learning what to do (from minimal labeled data).

Quoc Bao Phan, Tuy Tan Nguyen

Quantum circuits can replace classical fusion layers in federated learning with 72 parameters instead of 33K, making multi-agent activity recognition more practical for resource-constrained robotic systems.

This paper presents QFedAgent, a federated learning system for activity recognition across multiple robotic agents. It uses quantum circuits to fuse sensor data (accelerometer and gyroscope) more efficiently than classical neural networks, reducing parameters by 10x while maintaining accuracy on distributed, non-uniform data.

Boyang Sun, Jiajie Li, Yung-Hsu Yang et al.

Robots can learn intent-aware camera control from passive human video by mining supervision pairs of language descriptions, observation changes, and target poses—turning everyday egocentric footage into training data for active perception.

This paper tackles language-conditioned camera motion for robots by learning from egocentric video. Given an image and natural language intent, LIME predicts the next camera pose by combining observation-gain prediction with flow-matching, enabling robots to actively position cameras for inspection, occlusion handling, or user-intent-driven viewing.

Xianhui Meng, Zirui Song, Yuchen Zhang et al.

For 3D scene generation in irregular spaces, hierarchical placement strategies and statistical priors about object distributions significantly improve physical plausibility and reduce geometric violations compared to flat optimization approaches.

This paper tackles text-to-3D indoor scene generation in non-Manhattan (non-rectangular) spaces, where existing methods fail. SPG-Layout uses statistical priors about object placement and hierarchical layout strategies (placing large objects first) to generate physically realistic scenes that respect non-orthogonal spatial relationships.

Cristian-Gabriel Florea, Stelian Spînu

By running six specialized deep learning models locally on a smartphone with INT8 quantization, VisionAId achieves real-time visual assistance for blind and low-vision users without cloud dependency, while a few-shot learning pipeline lets users teach the system to find their personal objects.

VisionAId is an Android app that helps visually impaired people navigate and interact with their environment using on-device AI models. It combines depth estimation, object detection, and facial recognition to identify obstacles, locate personal items, and recognize faces—all running locally on a smartphone without requiring cloud connectivity, with optional AI for scene descriptions.

Nina Begus

Literary disciplines offer practical tools for making AI systems more culturally literate and pluralistic, moving beyond the monolingual, automated cultural encounters that current LLMs create.

This essay argues that literary analysis methods—comparative reading, narratology, critical theory, and world literature approaches—are essential for building culturally aware AI systems.

Chenyang Ma, Yue Yang, Radu Corcodel et al.

Progress signals and semantic subtask grounding are critical for long-horizon bimanual manipulation—the model predicts both actions and continuous progress to automatically transition between assembly steps and reduce compounding errors.

FurnitureVLA tackles real-scale bimanual robot furniture assembly using vision-language-action models. The system combines a VR teleoperation interface for data collection, a simulation pipeline for training, and a progress-aware model that predicts both actions and assembly progress to handle long-horizon tasks (up to 1550 steps).

Zekai Chen, Kairui Yang, Xuaner Chen et al.

Federated multimodal graph learning can achieve strong performance while maintaining privacy and interpretability by organizing knowledge into typed semantic codebooks that explicitly track how different modalities and graph structure contribute to predictions.

FedLAB enables federated learning on multimodal graphs (graphs with text, images, and attributes) while preserving privacy by organizing knowledge into traceable semantic codebooks.