ThinkLLM
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
AboutPrivacyTermsRSS

ThinkLLM

Spot an error in our data? Let us know.

Papers

Recent AI research papers with accessible summaries. Updated daily from arXiv, summarized for developers who don't read papers regularly.

2455 papers24 this month12 topics
AllEvaluation 43Agents 42Applications 39Efficiency 29Reasoning 29Training 27Safety 23Multimodal 18Architecture 9Alignment 8Data 4scaling 2

Nov 16 – Nov 22(6)

One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

Nov 19, 2026

Yuxin Jiang, Wenqiang Sun, Jiaqi Li et al.

Pretrained video generators can serve as powerful priors for discovering latent actions from visual transitions, enabling few-shot and zero-shot action transfer across different visual contexts and robot embodiments.

This paper proposes O²LaF, a method for learning actions from unlabeled videos by repurposing pretrained video diffusion transformers. The approach uses a single pretrained model to initialize both inverse dynamics (what action caused this change?) and forward dynamics (what happens after this action?), learning transferable actions without manual labels.

multimodal

Probing the "Conscious" Mind: Selective Access and Global Workspace in Vision-Language Models

Nov 19, 2026

Jiahao Huang, Jun Wang

Vision-language models exhibit selective concept access similar to human consciousness theories—information becomes available to different parts of the model based on what's needed, and this selective access develops as a learned capability during training.

This paper introduces MC-Lens, a tool for understanding how vision-language models process and access concepts across visual and textual information. Using global workspace theory as inspiration, the researchers show that VLMs selectively access concept information based on task needs, and that this information flows through downstream computations.

Nov 9 – Nov 15(7)

StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval

Nov 13, 2026

Dayoon Ko, Yichuan Wang, Soyeong Jeong et al.

By reusing computations across incrementally-changing screenshots, you can index screen history 3.7x faster in real-time without sacrificing visual understanding—making local visual assistants practical.

StreamEmbed enables real-time visual indexing of screenshots by exploiting pixel redundancy across frames. Instead of recomputing embeddings from scratch, it uses spatial token pruning to skip unchanged image blocks and temporal key-value reuse to cache computations across screenshots, achieving 3.7x faster encoding than standard visual encoders while maintaining accuracy.

efficiencymultimodalapplications

Unbundling Multimodal Vision Encoding

Nov 12, 2026

Ellis Brown, Chris Hoang, Shengyi Qian et al.

For building multimodal models, dedicated vision capacity trained jointly with the language model is more important than using pretrained vision encoders, and this capacity can be efficiently placed early in the backbone rather than as a separate pre-fusion module.

This paper investigates how vision encoders should be designed in multimodal language models by systematically comparing three design choices: using a pretrained visual encoder, dedicating capacity to vision processing, and where to place that capacity.

Nov 2 – Nov 8(11)

Saidzoda-CA-EmbeddingGemma-300M: One Small Embedder for Tajik, Kazakh, Uzbek and Kyrgyz

Nov 8, 2026

Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov et al.

A single small embedding model can serve multiple low-resource languages effectively through targeted fine-tuning with synthetic data, but scaling up to larger multilingual models remains necessary for competitive performance on these languages.

Researchers fine-tuned a small 300M embedding model (EmbeddingGemma) to work across four Central Asian languages—Tajik, Kazakh, Uzbek, and Kyrgyz—using synthetic question-answer pairs generated by a language model. The fine-tuned model significantly improved retrieval performance (14-26 percentage points) on a multilingual benchmark, though it still lags behind larger 560M multilingual models.

multimodaltrainingefficiency

Semantically Aligned KV Cache Compression for Efficient Agentic Autoregressive Video Generation

Nov 8, 2026

Weizhong Huang, Huaijia Kou, Tianyu Xie et al.

By aligning KV cache compression with semantic planning constraints, you can reduce memory usage by 3.35x and speed up video generation by 1.22x without retraining, making agentic video generation more practical for resource-constrained settings.

Oct 26 – Nov 1(11)

Latent Reasoning in Continuous Space for Unified Multimodal Models

Oct 30, 2026

Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee et al.

Reasoning in continuous latent space instead of discrete tokens unlocks test-time scaling and better controllability for image generation in multimodal models, showing 7% improvement on GenEval benchmarks.

This paper introduces LARC, a framework that enables unified multimodal models to perform reasoning directly in continuous space rather than discrete tokens, bridging the gap between language-based reasoning and image generation.

reasoningmultimodaltraining

Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers

Oct 29, 2026

Jiongfan Yang, Weichen Fan, Haiwen Diao et al.

By splitting image frequency components across local and global attention paths, you can generate finer image details without the quadratic cost of full global attention—achieving better quality at lower computational cost.

SpecSlot is a pixel-space image generation model that separates low-frequency and high-frequency image details into different processing paths. Low-frequency information uses global attention for context, while high-frequency details are processed locally, reducing computational cost while maintaining image quality.

Oct 19 – Oct 25(12)

SceneBloom: Agentic Indoor Environment Generation with Realistic and Diverse Layouts for Embodied Learning

Oct 23, 2026

Yuhan Liu, Lianhui Qin, Shenji Wan

By combining diffusion-based visual layout generation with geometric constraint-solving, SceneBloom creates training environments so realistic that policies trained on them outperform those trained on real scans—solving a key bottleneck in embodied AI.

SceneBloom is an AI system that generates realistic 3D indoor environments for training embodied agents (robots/AI that interact with physical spaces). It combines visual layout generation with geometric reasoning to create physically valid room layouts with furniture arrangements, achieving better realism and stability than existing methods.

agentsapplicationsmultimodal

Beyond Discrete Tokens: Preserving Numerical Continuity in Medical Vision-Language Models

Oct 23, 2026

Dianye Huang, Jiamei Guo, Yue Zhou et al.

Combining diffusion-based continuous prediction with language models outperforms discrete token prediction for medical measurements, suggesting hybrid architectures better match the continuous nature of real-world quantitative tasks.

This paper presents HAD-VLM, a medical AI system that combines language understanding with precise numerical measurement from ultrasound and X-ray images.

Oct 12 – Oct 18(6)

Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal

Oct 16, 2026

Ailiang Lin, Zhuoyun Li, Yusong Wang et al.

Language-specific noise in LLM embeddings can be removed at inference time using paired multilingual sentences, significantly improving cross-lingual and monolingual embedding quality without any training.

This paper tackles multilingual text embeddings from frozen language models by identifying that semantically equivalent sentences misalign across languages due to language-specific geometry.

multimodalefficiencyevaluation

OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modeling

Oct 15, 2026

Bowei Liu, Xinchen Zhang, Xuhuan Li et al.

Graded, component-level verification with pairwise comparison significantly outperforms binary verification for evaluating complex multimodal outputs, especially in structured reasoning tasks where compositional correctness matters.

This paper introduces OmniVerifier-C1, a visual verifier that moves beyond simple True/False judgments to provide detailed, component-level scores for evaluating generated images.

Oct 5 – Oct 11(19)

Hide and Seek: HIDE Concepts to SEEK explanations for black box models

Oct 9, 2026

Bhanu Tokas, Rahul Nair, Rockwell Jackson et al.

You can now explain any vision model's decisions using concepts without needing model internals or expensive labeled data—just show the model what happens when you hide concept regions.

Hide and Seek is a method for explaining black-box vision models using human-understandable concepts. Unlike existing approaches, it doesn't require access to model weights or labeled concept datasets. The method finds concept-relevant images, masks the concept regions, and measures how the model's output changes to determine what concepts influence predictions.

evaluationmultimodal

Designing Protein Structures from Natural Language

Oct 9, 2026

Joseph Lee, Yanjing Li, Bo Qiang et al.

Natural language is now a practical interface for protein design—you can describe desired protein properties in text and get back 3D structures, with experimental validation showing it works for real-world targets.

Prompteina is an AI model that generates protein structures from natural language descriptions, similar to how text-to-image models work. Trained on 1.8M protein structures with detailed captions, it can design proteins with specific structural and functional properties without needing separate input formats for each goal.

Sep 28 – Oct 4(14)

VISTA: A Visual Harness for Reasoning in an Interactive World

Oct 1, 2026

Qiushi Han, Keya Hu, Linlu Qiu et al.

Adding a simple visual memory system that preserves and lets models retrieve past observations dramatically improves multimodal models' reasoning in interactive visual environments—achieving perfect performance on challenging puzzle games.

VISTA is a visual harness that enhances multimodal models' ability to solve complex interactive visual tasks by giving them long-horizon vision and lossless visual memory. The system lets models directly perceive environments, store past observations, and actively retrieve them while reasoning.

agentsmultimodalreasoning

Generative Cinematographer: Composing Camera and Object Motion in 3D

Oct 1, 2026

Jiahan Zhang, Chaohao Yang, Namitha Guruprasad et al.

By lifting 2D video controls into explicit 3D space, you can resolve ambiguities in object motion and create videos where camera and object movements are geometrically consistent—a major improvement over 2D trajectory-based video control.

GenCine enables artists to control video generation by editing 3D camera paths and object motions in a scene scaffold, rather than using ambiguous 2D trajectories. The system projects these 3D controls into guidance maps that a pretrained video model learns to follow, producing videos with consistent camera-relative motion and improved geometric coherence.

Sep 21 – Sep 27(7)

EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models

Sep 25, 2026

Kunxiong Zhu, Zhihao Shu, Hangyu Zheng et al.

Multimodal LLM serving requires rethinking GPU resource allocation around the Encode stage—treating it as a bottleneck control point rather than a separate service unlocks significant throughput gains.

EAServe optimizes serving multimodal LLMs by treating the Encode stage as a control point for the three-stage Encode-Prefill-Decode pipeline. It uses adaptive micro-batching, partial offloading, and GPU partitioning to balance resource utilization across stages, achieving 4.3x higher throughput than existing systems under latency constraints.

efficiencymultimodalarchitecture

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Sep 24, 2026

Wenhao Li, Zhibin Wu, Chong Xiao et al.

Using LLM-generated semantics as a shared anchor point for aligning incomplete multimodal data is more robust than trying to reconstruct missing modalities or design complex fusion mechanisms.

SemMSA tackles multimodal sentiment analysis when some data is missing by using large language models to create rich semantic representations that ground all modalities together. Instead of reconstructing missing features, it aligns visual, acoustic, and text representations through spectral methods, achieving better results on standard benchmarks.

Sep 14 – Sep 20(7)

Paint-Anything: Unified Any-Color Control for Image Generation and Editing

Sep 17, 2026

Ji Xie, Dewei Zhou, Xinyu Huang et al.

By combining real image supervision with pure-color anchors and a shared hex-prompt interface, you can now control exact object colors in both image generation and editing tasks with professional-grade precision.

Paint-Anything enables precise color control in AI image generation and editing by letting users specify exact colors using hex codes (like #FF5733). The system learns to understand hex values through a training dataset of 500K images with color labels and pure-color reference images, achieving much better color accuracy than existing methods.

multimodaltrainingapplications

JEPA-Anything: Learning Predictive Models across Different Worlds

Sep 17, 2026

Taoyong Cui, Zhongyao Wang, Xinyue Xu et al.

A single learning principle based on factorized predictions can work across radically different domains, suggesting world modeling doesn't need domain-specific architectures—and can even guide real scientific discovery.

JEPA-Anything is a unified framework for building predictive models across completely different domains—from videos to molecules to weather—using a technique called orthogonal predictive factorization. Instead of training separate models for each domain, it learns to decompose predictions into independent factors that work across vision, biology, physics, and clinical data.

multimodalevaluationreasoning

ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images

Nov 19, 2026

Adrià Ciurana Lanau, Harleen Hanspal, Lusine Abrahamyan et al.

By sharing a spatial prototype across layers and using one compact vector per layer, ProtoDiffusion achieves efficient multi-layer image generation that's 19× faster than decomposing flat images, while maintaining high reconstruction quality.

ProtoDiffusion generates multi-layer graphic designs (like Photoshop files) by representing each layer compactly as a single vector, rather than full spatial grids. This reduces memory from N×H×W tokens to H×W+N tokens. The model can generate designs from images or text, automatically determining layer count and layout without requiring them as input.

multimodalarchitectureefficiency

Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs

Nov 19, 2026

Ziqing Yang, Yixin Wu, Michael Backes et al.

Multimodal image editing systems have a dangerous blind spot: they focus on filtering unsafe text prompts but largely ignore unsafe content in the images users upload, creating a systematic safety mismatch that attackers can exploit.

This paper reveals a critical safety gap in multimodal image editing systems: they handle text prompts and visual content inconsistently. The researchers created EditSafetyBench, a dataset of 5,940 image-instruction pairs, and tested four major systems (Gemini, GPT, Qwen).

safetymultimodalevaluation

MM-SystemOneBias: Evaluating Bias in Multimodal Jev-Style System One Decision Models

Nov 19, 2026

Jinzhe Li, Qi Wang, Dandan Guo et al.

Multimodal models show systematic biases in how they weight information across modalities—decision models are more robust to presentation changes than base models, but accuracy metrics alone miss important shifts in confidence that could affect real-world reliability.

This paper introduces MM-SystemOneBias, a benchmark for evaluating how multimodal AI models make decisions when presented with variations like different option orders, irrelevant information, or conflicting descriptions.

evaluationmultimodalsafety

WorldRuler: Interactive World Modeling under Configurable Rules

Nov 18, 2026

Mingliang Zhai, Chuanhao Li, Jiaming Tan et al.

To build customizable interactive worlds with generative models, you need to explicitly track rules and object states separately from visual generation, using techniques like dynamic object binding and counterfactual training to ensure rule effects appear correctly.

WorldRuler is a system for building interactive world models that follow user-defined rules rather than just real-world physics. It uses a 3D rule graph to organize rules and world states, enabling games or simulations to save/load while maintaining rule consistency, and accurately apply rule effects to objects even before they appear on screen.

agentsmultimodaltraining
multimodalarchitecturetraining

Asymmetric Multimodal Learning with Decomposed JEPAs

Nov 12, 2026

Yusuf Shihata

For multimodal AI systems, alignment-only training catastrophically fails on compositional reasoning when modalities carry different types of information—you need to explicitly preserve modality-specific structure alongside alignment.

This paper shows that standard cross-modal alignment alone loses important information when modalities are asymmetric (like text queries and images). It introduces Decomposed JEPA, which splits learned representations into a shared aligned part and private modality-specific parts, recovering information that alignment discards and improving performance on reasoning tasks.

multimodalarchitecturetraining

When Chekhov's Gun Fires Anyway: Over-Realization in Text-to-Image Models

Nov 12, 2026

Jiahao Huang, Kon Woo Kim, Song-Duo Ma et al.

Text-to-image generators fail to respect linguistic context (negations, beliefs, figurative language, hidden viewpoints) and generate unwanted objects anyway; existing metrics don't catch this, making it a blind spot in current evaluation.

Text-to-image models often generate objects mentioned in prompts even when the prompt doesn't ask for them—like showing a gun in a negation or imagination. This paper introduces OverReal, a benchmark measuring this 'over-realization' failure across four linguistic contexts, revealing all major models struggle with it and that standard metrics actually reward the problem.

evaluationmultimodal

Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models

Nov 12, 2026

Bo Han

You can directly control VLM safety by setting explicit budgets on harmful assistance rates and using a two-channel reward design that preserves the safety multiplier signal through training.

This paper addresses how to directly control safety in vision-language models using reinforcement learning. The authors propose a two-channel constrained GRPO method that treats safety as explicit budgets (acceptable rates of harmful outputs and over-refusal) enforced through Lagrange multipliers.

safetyalignmentmultimodal

Emergent Biological Capabilities in a Foundation Model for Molecular Interactions

Nov 10, 2026

Adi Mashiach

Foundation models can develop unexpected biological reasoning abilities through scale alone, and these emergent capabilities don't follow smooth scaling laws—they appear suddenly at larger model sizes, suggesting qualitative jumps in what AI can do with molecular data.

Bonbon is a foundation model trained on 3.2 trillion tokens of protein-ligand sequences that unexpectedly develops three distinct capabilities for predicting molecular interactions—mechanism of action, binding site location, and bond type—without explicit training for these tasks.

scalingmultimodalapplications

C$^2$-Routing: Complementary Cropping and Compression for Efficient Vision-Language Inference

Nov 10, 2026

Guorui Lu, Qinyu Chen

Cropping and compression are complementary strategies for reducing VLM inference cost—use cropping for detail-focused questions and compression for context-heavy ones, with a learned safety mechanism to protect accuracy.

This paper proposes C²-Routing, a method that adaptively chooses between cropping (for local details) and compression (for context) to reduce computational costs in vision-language models for VQA tasks. It includes a safety gate to prevent accuracy loss and achieves 36.53% FLOP reduction with minimal performance drop.

efficiencymultimodalapplications

This paper presents SemanticKV, a method to compress the key-value cache in video generation models by using semantic constraints from a planner to decide which cached information to keep.

efficiencymultimodalagents

Flex-DDCM: Zero-Shot Task-Oriented Image Compression via Diffusion Trajectory Guidance

Nov 8, 2026

Yixin Gao, Runsen Feng, Xin Li et al.

You can compress images for specific AI tasks (like object detection) without retraining by steering a pretrained diffusion model's denoising process with task guidance signals—enabling one codec to serve multiple purposes.

This paper presents Flex-DDCM, a zero-shot image compression method that uses diffusion models to compress images while optimizing for specific downstream tasks. By guiding the diffusion denoising process with task-specific signals, it achieves flexible trade-offs between image quality and task performance without requiring task-specific training.

efficiencymultimodalapplications

GRAC: Graph-Based Retrieval-Augmented Captioning for Surgical Scene Description

Nov 6, 2026

Rochak Dhakal, Chen Zhao, Soup Cobb et al.

Combining structured scene graphs with retrieval-augmented generation substantially improves surgical scene captioning accuracy and factual correctness, suggesting that domain-specific structured representations can enhance LLM-based medical documentation.

GRAC is a system that automatically generates descriptions of surgical scenes by combining graph representations of surgical elements (instruments, actions, anatomy) with retrieval-augmented generation. It retrieves similar past surgical scenes and uses them to guide caption generation, achieving significantly better descriptions than existing methods on surgical video benchmarks.

multimodalapplications

Computer-Use Agents Are Embodied Generalists

Nov 6, 2026

Pranjal Aggarwal, Sean Welleck

General-purpose computer-use agents can solve diverse embodied tasks without specialization, pointing toward a unified approach to embodied AI rather than task-specific models.

This paper shows that computer-use agents—AI systems trained to interact with software via screenshots and mouse/keyboard actions—can perform physical embodied tasks by rendering the environment on a screen.

agentsapplicationsmultimodal

Social Humanoid: Socially Responsive Human-Humanoid Interaction

Nov 5, 2026

Vongani H. Maluleke, James Ni, Kie Horiuchi et al.

Humanoid robots can learn coordinated, anticipatory interaction with humans by training on diverse motion-capture data adapted to the robot's morphology, enabling real-world physical collaboration without explicit coordination rules.

This paper presents Social Humanoid, a robot system that learns to physically interact and coordinate with humans in real-time. Using a two-level approach—a high-level motion planner and low-level controller—the system predicts and responds to human movements, even when the human is partially hidden.

agentsmultimodal

MOMASPLAT: Dynamic Novel-View Demonstration Augmentation for Mobile Manipulation

Nov 5, 2026

Sateesh Kumar, Arnav Balaji, Arpit Bahety et al.

By reconstructing 3D scenes and manipulated objects separately, then re-rendering demonstrations from novel robot base poses, MOMASPLAT makes visuomotor policies robust to viewpoint changes—doubling success rates on unseen positions without collecting more real-world data.

MOMASPLAT augments robot manipulation demonstrations by synthesizing them from unseen camera viewpoints using 3D reconstruction and novel-view synthesis. This solves a key problem in mobile manipulation: robots rarely execute tasks from the exact same position as during training, causing policy failures.

dataapplicationsmultimodal

BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

Nov 4, 2026

Fei Lin, Tengchao Zhang, Ziyang Gong et al.

Multimodal AI models can follow general lab procedures but fail at the precise, action-grounded perception needed for actual chemical lab automation—they can't reliably identify where to act, what equipment is present, or how to sequence complex procedures safely.

BEAKER is a benchmark with 1,000 expert-curated samples for evaluating multimodal AI models as 'embodied brains' in self-driving chemical labs. It tests whether AI can perceive lab scenes, reason about experiments, and execute actions safely—covering tasks like identifying where to act, planning movements, recognizing equipment and hazards, and sequencing steps.

evaluationmultimodal

Perceptive Humanoid Motion from Simulation

Nov 4, 2026

Siyuan Hu, Abulikemu Abuduweili, Yue Meng et al.

Procedurally generated simulation data can replace expensive human teleoperation for training humanoid robots, enabling zero-shot transfer to real-world tasks without paired real-world supervision.

HALO is a humanoid robot system trained entirely in simulation using procedurally generated homes to learn vision-language-motion skills. By synthesizing training data computationally rather than through human teleoperation, it avoids the data bottleneck that typically limits robot learning.

multimodal

CounterFactual Retrieval for Evidence-Aware Individual Animal Identification

Nov 4, 2026

Ankit K. Upadhyay, Aleksandr Algasov, Ekaterina Nepovinnykh et al.

Standard retrieval accuracy hides whether models identify animals by their actual features or by background cues; CFR's three-view evaluation exposes this by measuring performance on full images, foreground-only, and background-only versions.

This paper introduces Counterfactual Foreground Retrieval (CFR), a new evaluation method for animal identification models that reveals whether they rely on actual animal features or background shortcuts.

evaluationmultimodal

DreamX: Video Diffusion Model is a Universal Latent Action Learner

Nov 3, 2026

Jisoo Kim, Chaehyun Kim, Junsu Kim et al.

Video diffusion models can be repurposed as action encoders to create embodiment-agnostic action representations that transfer across robots and humans without retraining, while being 300× more compact than raw video.

DreamX repurposes a pretrained video diffusion model to learn universal action representations that work across different robots, humans, and environments.

multimodalagentsefficiency
architectureefficiencymultimodal

Probing Spatial State Representations in Visual Foundation Models

Oct 29, 2026

Baiqiao Yin, Zhongang Cai, Ruisi Wang et al.

Visual foundation models excel at recognizing visible scenes but struggle with spatial persistence, predictive updates, and action guidance—capabilities essential for embodied AI. Strong readout accuracy doesn't transfer to maintaining spatial state across time or motion.

This paper investigates whether visual foundation models learn spatial representations that support persistent spatial awareness—tracking objects beyond the current view, predicting future scenes, and guiding actions.

evaluationmultimodalreasoning

BURGER: Benchmarking and Understanding Robotic Generalization via Embodied Drawing

Oct 29, 2026

Xiangyu Xu, Bangshuai Peng, Ziyi Liu et al.

VLA models show promise in learning scalable, language-conditioned robotic behaviors with multimodal action generation, but their success depends on task diversity, data scale, and how well they ground spatial language into continuous control.

BURGER is a robotic drawing benchmark that tests how well vision-language-action models learn diverse behaviors from scaled training data.

evaluationmultimodalreasoning

Discovering Interpretable Acoustic Features in Autoregressive Voice-Cloning TTS Models

Oct 29, 2026

Anuj Diwan, David Harwath, Eunsol Choi

Voice-cloning TTS models learn interpretable acoustic concepts, but the relationship between understanding what a model knows and controlling it is complex—strong probing doesn't guarantee effective steering, and different feature extraction methods work better for different tasks.

This paper investigates what acoustic features voice-cloning TTS models learn and use by applying interpretability techniques like sparse autoencoders, PCA, and probing to Qwen3-TTS and five other models.

multimodalevaluation

ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning

Oct 27, 2026

Chaehyun Kim, Chi-Pin Huang, Jisoo Kim et al.

By grounding action reasoning in world dynamics rather than explicit action labels, robots can learn from unlabeled human videos alongside labeled robot data, dramatically expanding the training data available for robotic manipulation.

ThinkWorld is a framework that teaches robots to learn manipulation skills from both labeled robot videos and unlabeled human videos. It represents actions as sequences of motion tokens (Chain-of-Dynamics) that predict how the world will change, then uses reinforcement learning to strengthen predictions that match real outcomes.

agentsmultimodalreasoning

VideoLVR: Latent Visual Reasoning over Grounded Video Evidence

Oct 27, 2026

Shilong Li, Xingyuan Bu, Hao Shi et al.

Instead of describing or reinserting video regions during reasoning, VideoLVR reconstructs visual tokens from relevant moments—letting models "see" evidence again while staying efficient.

VideoLVR enables video understanding models to reason over video content by reconstructing visual evidence as latent tokens during reasoning steps, rather than relying on text descriptions or expensive visual crops. This approach keeps visual information accessible without bloating the context, improving performance on fine-grained video reasoning tasks.

multimodalreasoning

Beyond Answer Changes: Matched Same-Class Controls for Local Audio Evidence

Oct 27, 2026

Yunuo Wang, Maoyuan Chen, Yuanjin Huang et al.

To properly evaluate audio-language model robustness, use matched same-class sound replacements as a control—this isolates whether answer changes reflect genuine understanding of sound events versus sensitivity to waveform artifacts.

This paper introduces a method to test how audio-language models respond to local sound changes by comparing what happens when you replace a sound event with another from the same class versus a different class.

evaluationmultimodalsafety

Zero-Shot Multimodal Event Spotting in Soccer Broadcasts

Oct 27, 2026

Sadia Kamal, Mahmudul Hasan, Tony Braskich

General-purpose multimodal models can effectively spot sports events without fine-tuning by leveraging compact visual representations and aligned text, suggesting that domain-specific training may be unnecessary for event detection tasks.

This paper presents a zero-shot approach for detecting and localizing soccer events in video broadcasts using multimodal models, without requiring task-specific training. By combining visual storyboards and commentary transcripts processed by a general-purpose model, it achieves strong performance while reducing computational cost by 72% compared to frame-by-frame analysis.

multimodalapplications

Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR

Oct 26, 2026

Yuchen Zhang, Haralambos Mouratidis, Ravi Shekhar

When adapting multilingual speech systems to new languages, simple fine-tuning causes catastrophic forgetting of old languages—but selective merging strategies like SSU-Wanda+TIES can preserve both old and new language performance simultaneously.

This paper addresses expanding multilingual speech recognition systems to new languages while preserving performance on existing languages, without replaying original training data. The researchers test different adaptation and merging strategies for a connector module in speech-based LLMs, finding that careful update merging balances learning new languages with retaining old ones.

trainingmultimodalefficiency

Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

Oct 26, 2026

Yifei Yao, Yude Zou, Guanbo Jiao et al.

Frontier language models can control robots competitively with specialized policies on many tasks, but hybrid approaches that let the agent delegate to specialized policies achieve better overall performance—suggesting future robot agents should be designed to strategically invoke specialized c...

This paper evaluates GPT-6-Astra, a frontier multimodal language model, as a robot control agent using EBench. The study compares direct control (agent generates commands directly) versus a harness approach (agent can invoke specialized policies).

agentsmultimodalapplications
multimodalarchitecture

Yalda-Embedding: Open Development of a Competitive Persian Retrieval Model

Oct 23, 2026

Yalda Taheri, Mohammad Hassan Heydari, Armon Rasooli

Language-specific retrieval models can match much larger models through targeted fine-tuning with distillation, and open-sourcing training data and procedures enables reproducible progress in underserved languages like Persian.

Yalda Embedding is a 596M-parameter Persian retrieval model fine-tuned from Qwen3-Embedding using over 1 million query-document pairs. It outperforms comparable 0.6B models on Persian retrieval tasks and approaches 4B model performance, with gains from knowledge distillation and contrastive learning. The authors release all code, weights, and training data to support Persian NLP research.

trainingmultimodalefficiency

Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

Oct 22, 2026

Yunfei Xie, Howard H. Qian, Shiyi Lan et al.

You can predict whether a robot model will succeed at tasks by checking if its visual representations align with motion geometry, without running expensive robot experiments—this cheap offline check helps prioritize which models to test in the real world.

This paper proposes Sensorimotor Alignment (S²), a training-free metric that measures how well a robot model's visual representations align with the geometry of physical motion.

evaluationmultimodal

Rethinking Sequence Compression with Test-time Regression

Oct 22, 2026

Sihan Yang, Jie Huang, Hanxuan Li et al.

Test-time regression offers a more effective way to compress token sequences than standard pooling or projection methods, enabling better efficiency in both language and multimodal models without sacrificing performance.

This paper proposes TTR-Compressor, a new method for compressing sequences of tokens in language and multimodal models. Instead of using pooling or learned projections, it frames compression as a regression problem solved at test time with coefficients computed efficiently in closed form.

efficiencyarchitecturemultimodal

Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

Oct 20, 2026

Noam Rotstein, Yoad Tewel, Chen Tessler et al.

You can now generate video and 3D animation together for any character rig—not just humans—and automatically place that animation in a 3D scene with correct camera positioning, enabling practical animation workflows from text or video.

This paper presents a model that generates both realistic video and replayable 3D motion simultaneously for any character rig. By keeping 3D motion in a separate model connected to a video generator through cross-attention layers, it avoids adding billions of parameters while supporting arbitrary skeletons.

multimodalapplications

UniWALM: Unified World Action Language Model for Embodied Robot Control

Oct 20, 2026

Rahul Sajnani, Diego Romeres, Mingyuan Zhang et al.

Integrating language generation directly into robot control models—not as an afterthought but as a core part of the same network—enables robots to use language as working memory, significantly improving performance on tasks that require remembering past events.

UniWALM is a unified AI model for robot control that learns video, actions, and language together in one shared representation.

agentsmultimodalarchitecture

Industrial ReMEmbR: Scaling Spatiotemporal Memory to Robot Fleets

Oct 20, 2026

Jad Yammine, Malek Wahidi, Abrar Anwar et al.

Splitting large vision-language models at their architecture boundary—vision encoder on robots, language decoder on servers—enables efficient fleet-scale memory systems that generate descriptions on-demand rather than at write time, cutting server costs while preserving retrieval quality.

IndustrialRemembr extends a spatiotemporal memory system to robot fleets by splitting vision-language models between edge devices and servers. Robots encode video into embeddings and tokens without uploading raw frames, while an AI agent retrieves memories on demand and generates descriptions only when needed, reducing computation and storage costs while maintaining GDPR compliance.

efficiencyagentsmultimodal

MindView: Benchmarking Generative Multi-View Spatial Intelligence

Oct 20, 2026

Yanbo Wang, Siqi Luo, Siyu Zhang et al.

Generative multi-view spatial understanding is a measurable capability that improves when models are fine-tuned on spatial generation tasks, and this improvement transfers to other spatial reasoning benchmarks without additional training.

MindView is a benchmark with 1,500 samples testing how well AI models can generate spatially consistent views from partial observations using language guidance. The benchmark includes seven tasks split between geometry-focused (camera position/orientation) and semantics-focused (object grounding, spatial relations) challenges.

evaluationmultimodalreasoning

Text-to-Image Generative Modeling Without Language Pretraining

Oct 19, 2026

Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas et al.

You can build competitive text-to-image systems without relying on pretrained language models, which could make these systems cheaper and more accessible to build from scratch.

Halcyon shows that text-to-image models don't need large language pretraining to work well. By training from scratch on 100M image-text pairs, Halcyon achieves better results than existing systems while using far less compute—18× to 275× less than competitors. This challenges the assumption that internet-scale language models are essential for image generation.

trainingmultimodalefficiency

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Oct 19, 2026

Yu Zhou, Cheng-Fu Yang, Rui Sun et al.

Adding tool access and persistent multimodal memory to vision-language models dramatically improves performance on real-world tasks that require grounded perception and numerical reasoning, while actually reducing computational cost compared to scaling inference alone.

OpenVLHarness is a system that helps multimodal AI models solve complex visual reasoning tasks by combining external tools, persistent memory, and code execution.

agentsmultimodalreasoning

LatComm: Reusable Context Compression for Latent Communication between Large Language Models

Oct 19, 2026

Huawei Lin, Jinting Liu, Tan Yu et al.

Separating context encoding from reading allows LLM systems to compress context once and reuse it across multiple questions with different models, dramatically reducing computation and latency in multi-turn scenarios.

LatComm enables efficient communication between multiple LLMs by compressing context once into a compact, reusable representation that different models can selectively read.

efficiencymultimodalarchitecture
evaluationreasoningmultimodal

GeoState: Learning View-Decoupled Geometry States for 3D World Modeling

Oct 15, 2026

Hongchang Liu, Jian Liu, Yujuan Ding et al.

By decoupling scene state from camera viewpoint, GeoState makes 3D world predictions more accurate and generalizable—the same predicted scene can be rendered from different camera angles without retraining.

GeoState is a 3D world modeling approach that separates scene geometry from camera viewpoint by learning view-decoupled state representations. It uses a scene tokenizer to aggregate multi-view geometry features into a unified state, then predicts future states with an autoregressive model while handling camera motion separately.

architecturereasoningmultimodal

Twice Upon a Time: Complementary Representations for Time-Series Reasoning

Oct 15, 2026

Hedi Zisling, Liran Nochumsohn, Ilan Naiman et al.

Combining multiple complementary representations of the same time-series data—each with different invariance properties—helps language models better capture both absolute values and temporal structure, improving performance on prediction and reasoning tasks.

This paper proposes using two complementary visual representations for time-series reasoning in language models: line plots that preserve raw measurements, and normalized delay embeddings that expose temporal patterns. A dual-view model combining both through visual encoders significantly outperforms single-representation baselines on time-series understanding benchmarks.

multimodalreasoningarchitecture

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

Oct 15, 2026

Yuruo Wu, Kaihang Pan, Kai Shen et al.

For audio-video generation, evaluating consistency at the event level—rather than treating entire clips as single units—better captures real-world complexity and aligns more closely with human judgment.

This paper introduces AVC-Reward, an event-centric evaluation framework for assessing audio-video consistency in generated content. Unlike existing metrics that treat clips holistically, it breaks down complex scenes into individual audio and visual events, checks their temporal alignment and attribute consistency across modalities, and aggregates results into a three-level metric.

evaluationmultimodalapplications

PhySGA: A Physics-Space Grounded Agentic Framework for Generalizable and Interactive 3D Composition

Oct 14, 2026

Weichen Lyu, Chong Xia, Mingyu Dong et al.

Grounding 3D composition in physics simulation with collaborative agents (planning + feedback) enables AI to handle complex assembly tasks that pure language or vision models cannot solve.

PhySGA is a framework that helps AI agents assemble 3D objects by combining language models with physics simulation. Instead of just predicting where objects go, it uses an Assembly Agent for planning and a Critic Agent for physical feedback, allowing it to handle complex tasks like hanging objects, gravity settling, and containment that require interactive trial-and-error.

agentsreasoningmultimodal
applicationsmultimodaldata

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

Oct 9, 2026

Shubh Garg, Chen Shani

VLMs encode spatial relationships at the token level but lose this information when aggregating into global representations used for downstream tasks—a failure invisible to standard output-based evaluation.

This paper reveals that Vision-Language Models fail to preserve spatial relationships in their internal representations, even though they perform well on standard benchmarks. The authors introduce EICS, a metric to test whether models consistently encode concepts, and CASP-VL, a benchmark with controlled spatial perturbations.

evaluationmultimodalreasoning

Studying Comics at Scale with Artificial Intelligence: The PAGE Framework and Three Studies of American Comics and Japanese Manga

Oct 9, 2026

Emilio Ferrara

AI can transform humanities scholarship by enabling large-scale analysis of visual archives—PAGE shows how vision-language models, grounded in human annotation, can extract meaningful patterns from thousands of comics that would be impossible to study manually.

This paper presents PAGE, an AI framework for analyzing comics and manga at scale. It automatically parses pages into panels and text, annotates visual content using vision-language models following human codebooks, and validates findings against human annotation.

multimodalevaluationapplications

SpatialASM: Internalizing Active Mentalization for Spatial Reasoning

Oct 9, 2026

Yuhong Liu, Shenxi Wu, Junyi Chi et al.

LVLMs can be trained to proactively imagine scenes from task-relevant viewpoints by providing explicit supervision through generated spatial reasoning trajectories and perspective-taking tasks, significantly improving performance on spatial reasoning benchmarks.

This paper introduces SpatialASM, a method to teach Large Vision-Language Models to actively imagine scenes from different viewpoints when solving spatial reasoning tasks.

reasoningmultimodaltraining

FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?

Oct 8, 2026

Yuxuan Hu, Weikang Shi, Yang Bo et al.

Streaming VLMs fail at perceiving fast events in videos—the best model scores only 50.7% on FastBench—because they can't adaptively determine when to sample frames densely, revealing a fundamental gap between sparse uniform sampling and what's actually needed for dynamic understanding.

FastBench is a benchmark for evaluating how well streaming video language models can understand fast-moving events in real-world videos. Current models struggle with this task because they must balance limited context budgets across temporal history, spatial resolution, and frame sampling rates.

evaluationmultimodal

Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching

Oct 8, 2026

Luping Liu, Bingyi Kang, Yifan Wang et al.

By combining foundation models with diverse training data, you can build correspondence matching systems that work across both classical vision tasks and modern generative applications like image editing.

FreeMatching is a framework for matching visual correspondence across images that goes beyond traditional motion and geometry assumptions. It combines generative and semantic models trained on diverse data sources to handle challenging transformations in image editing and generation tasks, while maintaining competitive performance on standard benchmarks.

multimodalevaluationarchitecture

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

Oct 8, 2026

Zheyu Fan, Yue Zhang, Mingkai Deng et al.

Visual transition reasoning—understanding how scenes change—is a foundational skill that multimodal LLMs can learn from diverse video examples and reuse across many downstream tasks, offering a systematic way to improve spatial and temporal reasoning.

WOVEN is a training dataset and benchmark for teaching multimodal LLMs to reason about how visual scenes change over time. The authors create 36,000+ examples showing different actions in various scenes, train multiple models on this data, and find that this 'visual transition reasoning' skill transfers to 22 other benchmarks—improving performance by up to 27 percentage points.

trainingmultimodalreasoning

ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

Oct 8, 2026

Hongxing Li, Dingming Li, Yixin Li et al.

Visual skills outperform text-based skill representations for embodied agents because they preserve spatial structure; coupling skill learning with policy optimization creates a virtuous cycle of mutual improvement.

ViSkill teaches vision-language model agents to learn and reuse visual skills from successful interactions. Instead of converting spatial information into text (which loses important details), the system stores skills as visual cards that guide both action selection and reward signals, creating a feedback loop where better policies improve the skill library and vice versa.

agentsreasoningmultimodal

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Oct 8, 2026

Hongxing Li, Jinyue Su, Dingming Li et al.

Vision-language models are far better at describing visible spatial relationships than predicting unseen outcomes after changes—a gap that matters for real-world applications like robotics and planning.

SpaceCast-Bench is a new benchmark that tests whether vision-language models can predict how scenes change after interventions, not just describe what they see. With 3,862 questions from real scenes, it reveals that even top models reach only 58% accuracy versus 87% for humans, and that models struggle without explicit 3D information or bridge views connecting different observations.

evaluationmultimodalreasoning

Learning Kilometer-Scale Weather Prediction with Global-Regional Alignment

Oct 8, 2026

Guowen Li, Yang Liu, Yujie Wang et al.

You can build accurate kilometer-scale weather forecasts by aligning global and regional model representations, allowing a single regional model to work with multiple pretrained global forecasters without retraining.

ScaleCast is a weather forecasting framework that combines global weather models with high-resolution regional predictions by aligning their representations across different grid scales. It enables kilometer-scale weather forecasts for local warnings while reusing pretrained global models, eliminating the need to retrain when switching between different global forecast sources.

architecturemultimodalscaling

SpaceFlow: Locally Controllable 3D Generation

Oct 8, 2026

Neil De La Fuente, Joan Lafuente, Mukhammadali Sayfiddinov et al.

Users can now control different regions of generated 3D objects independently—some parts can follow input geometry exactly while others generate freely—solving the problem of global-only control in current 3D generation methods.

SpaceFlow enables fine-grained control over 3D object generation by letting users specify which parts should strictly match input shapes versus allow creative variation. It combines geometric primitives as spatial constraints with text/image conditioning to control both structure and appearance locally, avoiding unwanted blending between object parts.

multimodalarchitectureapplications

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Oct 8, 2026

Qun Dai, Liangjian Wen, Jiang Duan et al.

Synergistic multimodal information requires explicit higher-order tensor modeling rather than standard pairwise interactions; HRIL's approach consistently outperforms existing methods on tasks where modalities must work together.

This paper addresses how to capture synergistic information in multimodal learning—signals that only emerge from combining multiple modalities together, not from any single modality alone. The authors propose HRIL, which uses higher-order tensor mathematics (specifically Tucker decomposition) to model complex cross-modal interactions and preserve the capacity to learn these synergistic signals.

multimodaltrainingarchitecture

GeoReform: Reflective Formalization Evolution for Multimodal Geometry Problem Solving

Oct 8, 2026

Jialu Wang, Ruichen Zhang, Xiaoou Liu et al.

Effective geometry reasoning requires not just extracting geometric facts, but organizing them intelligently—and this organization can be learned and improved automatically rather than hand-crafted.

This paper tackles a key problem in geometry problem-solving: converting visual diagrams into text that AI models can reason about. Rather than using a fixed parser, the authors propose GeoReform, which treats formalization as a learnable process that improves through trial and error.

reasoningmultimodalevaluation

Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models

Oct 7, 2026

Mikey Watts, Yuchen Cui

Language phrasing sensitivity in VLAs is a learnable, systematic problem that can be mitigated at inference time using LLM-distilled rephrasing rules, improving robustness to out-of-distribution instructions without model retraining.

Vision-language-action models (VLAs) that control robots are surprisingly fragile to how instructions are phrased—changing one word can drop success rates by 50+ points.

multimodalagents

WorldSonus: Bringing Sound to Worlds

Oct 6, 2026

Pengjun Fang, Jingyi Fa, Kam Man Wu et al.

Real-time spatial audio generation for interactive video is now feasible with streaming diffusion models, enabling world models to produce complete audiovisual experiences rather than silent videos.

WorldSonus adds realistic spatial sound to AI-generated video environments in real-time. It uses a streaming diffusion model to generate audio that matches video content, responds to text instructions mid-generation, and creates stereo sound that aligns with scene geometry and camera movement.

multimodalefficiencyapplications

Learning to Read the Contextual Tokens in Diffusion Transformers

Oct 5, 2026

Omer Dahary, Etai Sella, Hadar Averbuch-Elor et al.

Text tokens in image-generating transformers develop interpretable semantic representations of the emerging image that can be read with an LLM probe—and explicitly training to strengthen these representations improves generation quality.

This paper reveals what text tokens learn during image generation in multimodal diffusion transformers. Researchers built a tool to 'read' these hidden representations by connecting them to a language model, discovering they encode rich scene information early in generation. They then used these insights to improve image quality through a new training technique.

multimodaltraining

Recursive Video In-Context Learning for Agentic Robot

Oct 5, 2026

Wenrui Bao, Xinxin Liu, Bingxin Xu et al.

By organizing demonstration videos into navigable hierarchies rather than static prompts, agents can access task details only when needed, reducing context overhead while improving learning from single examples.

This paper presents Recursive Video In-Context Learning (RV-ICL), a method that helps robot agents learn from demonstration videos more effectively. Instead of feeding entire videos as prompts, RV-ICL organizes a single demo video into a hierarchical structure of sub-events (like grasps and releases) that the agent can navigate on-demand.

agentsreasoningmultimodal

PlotGround: Grounding Plot Digitization in Real Scientific Figures and Their Source Data

Oct 5, 2026

Yaohui Zhang, Binxu Li, Haoyi Duan et al.

Current multimodal models can approximate values from scientific figures but struggle with precision; providing source data instead of figures dramatically improves accuracy (90% to 97.4%) while reducing computational cost, suggesting a practical path for scientific data extraction.

PlotGround is a benchmark for evaluating how well AI models can extract numerical values from scientific figures.

evaluationmultimodaldata
multimodalarchitectureapplications

DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communication

Oct 1, 2026

Hanchu Zhou, Dechen Gao, Hang Wang et al.

Using semantic communication between robots—where they exchange meaningful descriptions rather than sensor data—enables better coordination on long-horizon tasks while keeping each robot's execution independent and reliable.

DuoMind is a framework that enables multiple robots to coordinate and work together on complex tasks by combining vision-language models for high-level reasoning with vision-language-action models for precise execution. Robots communicate through semantic messages rather than raw data, allowing them to share understanding of the task and environment while maintaining independent control.

agentsmultimodalreasoning

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes

Oct 1, 2026

Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc et al.

Spatially grounded self-distillation with synthetic data can teach multimodal models better visual reasoning that transfers to real-world tasks, without requiring human annotations or external teachers.

This paper improves multimodal AI models by having them learn from a smarter version of themselves that receives spatial hints about where to look in images. Using synthetic scenes with automatic object labels, the approach trains models to understand spatial relationships without human annotation, and surprisingly, these improvements transfer to real-world vision tasks.

trainingmultimodaldata

GeoLatent: Geometry-Guided Latent Structuring with Routed Optimization for 3D Reasoning

Oct 1, 2026

Yakun Zhu, Yi Bin, Yujuan Ding et al.

Separating geometry into common and residual components while routing visual learning through structured latents prevents representation collapse and improves 3D spatial reasoning from images.

This paper addresses 3D spatial reasoning from 2D images by introducing GeoLatent, which uses decomposed spatial representations (position, direction, geometry) with geometric supervision and routed optimization. The method prevents geometry collapse and ensures latents are actively used during learning, achieving state-of-the-art results on spatial reasoning benchmarks.

reasoningmultimodalarchitecture

Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

Sep 30, 2026

Tian Xia, Minghao Liu, Yiqing Liang et al.

For imbalanced clinical tasks, optimizing prompts for ranking metrics (AUROC) instead of accuracy can dramatically improve model performance—up to 16 percentage points—because accuracy-based optimization fails when one class dominates the data.

This paper addresses class imbalance in clinical diagnosis by optimizing multimodal language models for AUROC instead of accuracy. The authors introduce Ranking-PE, a prompt optimization method that evaluates candidate prompts based on how well they rank positive cases above negative cases, rather than raw correctness.

evaluationmultimodalapplications

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

Sep 30, 2026

Xinghao Chen, Xiangbo Gao, Jiongze Yu et al.

Video text editing requires balancing three competing goals: correct text, smooth motion, and unchanged background. This benchmark and dataset help measure those trade-offs and establish baselines for the community.

ViTeX-Bench is a benchmark for video scene text editing—replacing text on surfaces like signs and labels while keeping the rest of the video unchanged. It includes 387 real videos, evaluation metrics for text accuracy and visual quality, and a baseline editor that achieves strong results. This addresses a gap where video text editing lags behind image editing.

evaluationmultimodalapplications

Image Classifiers are Efficient Self-Supervised Video Representation Learners

Sep 30, 2026

Owais Iqbal, Sudipta Sarkar, Shyam Marjit et al.

You can efficiently learn video representations by repurposing pretrained image models with clever masking strategies, avoiding expensive 3D architectures and reconstruction overhead.

VideoMSN uses standard image Vision Transformers to learn video representations without 3D models or reconstruction. It treats videos as grids of frames and masks either spatial patches or temporal frames, then aligns the two views using a Siamese loss. This achieves top results on video benchmarks while needing 32-160x fewer training epochs than prior methods.

efficiencymultimodal

WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents

Sep 30, 2026

Ziyan Jiang, Jingbo Yang, Jiabao Ji et al.

Multimodal agents struggle to couple exploration and visual reasoning in 3D worlds—they can see anomalies or navigate, but struggle to do both together effectively, suggesting a fundamental gap in how these systems integrate action and perception.

WorldAuditBench is a benchmark for testing how AI agents find problems in 3D virtual worlds—like floating objects or walls you can walk through. It evaluates multimodal AI systems (vision-language models and vision-language-action models) on 213 anomaly detection tasks across 13 environments, measuring how well agents can explore systematically and visually identify issues.

evaluationmultimodalagents

MatLoom: Layered Text-to-Material Generation in a Compact Program Space

Sep 30, 2026

Anson Y. Lam, Shuqing Li, Michael R. Lyu

Text-to-material generation can be more effective and controllable by outputting human-readable programs rather than raw images—users get both the final material and the explicit rules that construct it.

MatLoom generates realistic materials from text descriptions by creating compact, readable programs that define how layers combine to form appearance and physical properties.

multimodalapplications

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Sep 29, 2026

Jaewoo Jung, Hyeonseo Yu, Honggyu An et al.

Training MLLMs to internally reconstruct 3D scene geometry (even in compact form) improves spatial reasoning and 3D understanding, suggesting that learning to imagine scenes is more effective than explicit geometric supervision alone.

This paper teaches multimodal AI models to reason about 3D scenes by first imagining a compact 3D representation before answering questions. Instead of relying on detailed geometric details, the model learns to assemble a coarse 3D layout from multiple viewpoints—similar to how humans understand 3D space—then uses this mental model to answer spatial reasoning questions more accurately.

multimodalreasoningarchitecture

EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

Sep 29, 2026

Kuan-Po Huang, Haohe Liu, Puyuan Peng et al.

Emotion vectors in TTS models can be decomposed into a neutral-shift component and an emotion-specific component—controlling them separately via steering achieves much better emotion control than treating them as a single direction.

This paper improves emotional speech generation by decomposing emotion vectors into shared and residual components, then controlling them separately without retraining the model. The method, EmoRES, significantly outperforms prior vector steering approaches on multiple emotion metrics and human evaluation.

efficiencymultimodaltraining

Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies

Sep 29, 2026

Hui Ren, Lei Fan, Henry Pao et al.

For long-video question answering, visually grounding object identity across time—not just retrieving relevant clips—is critical. GEB shows that organizing observations into entity biographies improves accuracy by 4+ points on day-long and week-long videos.

This paper solves a key challenge in long-video understanding: tracking the same physical object across hours or days of footage. The authors introduce Grounded Entity Biographies (GEB), which groups visual observations of the same object into retrievable "biographies" that preserve context.

multimodalreasoning

Learning Native Reflection in Unified Models with Interleaved Reinforcement Learning

Sep 28, 2026

Yijia Fan, Ziqi Huang, Zhongang Cai et al.

Unified models can learn to self-correct their outputs by applying RL to complete reflection loops, where both the reasoning about what's wrong and the actual image fixes improve together without needing external verifiers.

This paper presents UMM-Reflection, a method that teaches unified multimodal models to critique and fix their own image generations through reinforcement learning. Instead of just generating images once, the model can now look at what it created, identify problems, revise the image, and repeat—all within a single model.

trainingreasoningmultimodal
multimodalalignmenttraining

To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

Sep 24, 2026

Debajyoti Mazumder, Mamta, Abhirama Subramanyam Penamakuri

Audio language models have a significant blind spot: they verify written facts reliably but fail on identical claims when spoken. Retrieval-augmented approaches help only when combined with explicit reasoning, not retrieval alone.

VeriSpeak is a benchmark for fact-checking spoken claims using audio language models. It contains nearly 4,000 spoken statements about real-world facts and tests whether models can verify claims directly from speech, especially when given retrieved text evidence.

evaluationmultimodalreasoning

The Alignment Illusion in Multimodal Large Language Models

Sep 24, 2026

Hong-Han Wang, Yuntao Wang, Hu Ding

Don't trust standard alignment scores as proof that multimodal models understand images—they can be fooled by the model's internal structure. Use task-based validation and the PA gap metric instead to verify genuine cross-modal integration.

This paper reveals that high visual-text similarity scores in multimodal AI models don't actually mean the model is properly integrating images and text. Using experiments across 13 models, the researchers show that replacing images with random noise barely changes similarity scores, even though task performance drops sharply.

evaluationmultimodal

Do Audio Language Models Hear and Read Distinctive Features Alike?

Sep 24, 2026

Yuanhao Chen, Peter Chin

Audio language models don't reliably represent the same phonetic features in the same direction across speech and text—suggesting these models may process the two modalities quite differently despite using a shared decoder.

This paper investigates whether audio language models represent phonetic features consistently across speech and text inputs. Using minimal pairs of phonemes that differ in single features, researchers measured whether the same distinctive features (like voicing) are encoded in the same direction in both modalities across 6 models, 7 features, and 15 languages.

multimodalevaluationarchitecture

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

Sep 21, 2026

Wangbo Yu, Kunhao Liu, Wenbo Hu et al.

By storing observations in a viewpoint-aware implicit 3D memory rather than explicit depth maps, video world models can generate longer, more consistent videos across different camera angles without running out of token budget.

WorldCrafter is a video world model that maintains consistent 3D-aware memory across different camera viewpoints and long time horizons. It uses an implicit memory system that compresses multi-view observations into tokens optimized for the requested viewpoint, enabling realistic minute-long video generation from a single image or text prompt while respecting what was seen before.

architecturemultimodalreasoning

Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks

Sep 21, 2026

Muzhe Wu, Zuchen Li, Xu Wang et al.

Contextualizing visual instructions to match a user's real workspace—through AI-generated images and videos—improves task performance and trust compared to generic pre-authored tutorials.

This paper presents a system that generates live visual instructions tailored to a user's specific workspace and task progress, rather than showing pre-recorded tutorials. Using AR and generative AI, it creates goal images and demo videos that match the user's actual environment, helping them complete physical tasks more accurately and confidently.

applicationsmultimodalevaluation
architecturereasoningmultimodal

Semantic Action Graph: A Shared Representation for Agent Grounding and Human Interpretation of Sports Highlights

Sep 17, 2026

Tica Lin, Deepak Chandran, Gauri Jagatap et al.

A shared, human-readable schema can simultaneously ground agent generation and enable human interpretation, making AI-generated content more transparent and controllable.

This paper introduces a semantic action graph—a structured representation of sports matches using performers, actions, recipients, moments, and states—that enables both AI agents to generate video highlights and humans to understand and control them.

agentsmultimodalapplications

Agile-WAM: An Agile Tactile World Action Model for Contact-Rich Robot Control

Sep 17, 2026

Hanchu Zhou, Brendan Lynch, Raman Goyal et al.

By treating vision and tactile signals at different timescales in a lightweight architecture, you can build fast tactile-aware robot controllers (11.9ms latency) that outperform larger models on real contact-rich tasks.

This paper presents Agile-WAM, a tactile world action model that predicts future robot states and actions for contact-rich manipulation tasks.

architecturemultimodalefficiency

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation

Sep 17, 2026

Haocheng Xi, Yiming Xie, Hexu Zhao et al.

Hybrid attention architectures that combine local Softmax with linear memory can dramatically accelerate video generation without sacrificing quality—enabling practical real-time video synthesis on modern hardware.

Video DeltaNet combines local attention with efficient linear memory to speed up video diffusion models. By mixing Softmax attention for fine details with a new Video Delta Attention mechanism for long-range context, it generates high-quality videos 14.5x faster than baseline models while maintaining visual quality.

efficiencyarchitecturemultimodal

HerHealthEval: Evaluating Multilingual and Register-Sensitive Understanding of Women's Health Communication

Sep 17, 2026

Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi et al.

Multilingual healthcare AI needs explicit testing for how language, communication style, and missing information affect risk assessment—aggregate accuracy scores alone don't catch safety failures that emerge in specific languages or contexts.

HerHealthEval is an evaluation framework that tests how well AI language models understand women's health concerns across multiple languages (English, French, Arabic) and communication styles (clinical, casual, emotional, vague).

evaluationmultimodalsafety

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

Sep 16, 2026

Sara Pieri, Evangelos Kazakos, Shizhe Chen et al.

Combining dense captioning with pixel-level grounding requires jointly optimizing text generation and mask selection—PANORAMA shows this can be done effectively by conditioning a segmenter on phrase representations and learning which masks correspond to each phrase.

This paper introduces PANORAMA, a vision-language model that generates detailed image captions while simultaneously grounding each phrase with pixel-level segmentation masks.

multimodalevaluationarchitecture