Papers

Gil Harari, Yoel Zimmermann, Ola Tangen Kulseng et al.

For scientists training ML models of molecular systems, switching from Adam to SOAP or SOAP-Muon optimizers can improve both training speed and final model accuracy, with bigger gains when you have less labeled data.

This paper compares advanced optimizers (SOAP, Muon, SOAP-Muon) against Adam for training machine learning interatomic potentials—AI models that simulate molecular behavior. The researchers find these newer optimizers converge faster and achieve better accuracy, especially when training data is limited, suggesting optimizer choice significantly impacts MLIP performance.

Xuehui Wang, Xuankun Yang, Wei Shen

When pruning visual tokens in VLMs, filtering textual noise with entropy and selecting tokens as a structured optimization problem (not just picking top-K) preserves fine-grained details better while reducing computation.

This paper tackles the problem of compressing image tokens in vision-language models (VLMs) while preserving important visual details. The authors identify that existing pruning methods fail because textual noise corrupts the scoring process and selected tokens become fragmented.

Junhao Shi, Siyin Wang, Xiaopeng Yu et al.

Separating motor skill learning from language grounding dramatically reduces the labeled data needed for robot learning—TAP matches models trained on 1M+ expert trajectories while using far less labeled data and shows better robustness to real-world perturbations.

This paper proposes Task-Agnostic Pretraining (TAP), a two-stage approach for training Vision-Language-Action robots that separates learning how to move (from unlabeled robot interactions) from learning what to do (from minimal labeled data).

Donghyun Lee, Jitesh Chavan, Duy Nguyen et al.

By rotating activations into a normalized basis before quantization, OrbitQuant eliminates the need to recalibrate for different inputs, timesteps, or models—enabling practical low-bit quantization of diffusion transformers without per-checkpoint tuning.

OrbitQuant is a quantization method for diffusion transformers that works without needing to recalibrate for different inputs or models. It uses a mathematical rotation technique to normalize activations so they stay consistent across different timesteps and prompts, allowing a single quantization scheme to work everywhere.

Zhuowei Chen, Xiang Lorraine Li

By analyzing which neurons activate during model predictions, you can automatically select better training data and improve self-supervised learning without any human annotations—useful when expert labels are expensive or unavailable.

This paper proposes Neuron-OPSD, a method for improving large language models without human labels by using the model's internal neuron activations to select which training examples to learn from and how to construct better teacher models. The approach trains the model on its own predictions, achieving better performance on specialized tasks while maintaining general knowledge.

Xuanyu Chen, Nan Yang, Shuai Wang et al.

When training on decentralized, non-uniform data, use Masked Image Modeling instead of Contrastive Learning—it's theoretically more robust. Better network connectivity always improves robustness, so federated learning is a viable alternative to fully decentralized systems.

This paper analyzes how distributed self-supervised learning systems handle non-uniform data across devices. The researchers prove that Masked Image Modeling is more robust to data heterogeneity than Contrastive Learning, and that federated learning performs as well as fully decentralized approaches. They introduce MAR loss, a practical improvement that aligns local and global representations.

Srinivasan Arunachalam, Louis Schatzki

Coherent quantum memory is the critical resource that enables efficient stabilizer state testing; without sufficient memory, testing becomes as hard as learning, requiring linear in n copies instead of a constant number.

This paper studies how to test and learn quantum stabilizer states when algorithms can only keep a limited number of qubits in quantum memory between measurements.

Yuquan Xue, Le Xu, Zeyi Liu et al.

Using a world model trained on real robot data to generate synthetic transitions—combined with careful sample selection—lets robots learn manipulation tasks with 59% fewer real interactions while improving success rates by 28%.

WorldSample combines real robot interactions with a world model to generate synthetic training data for reinforcement learning. By closing a loop between physical rollouts, synthetic data generation, and policy improvement, it reduces the number of costly real-world interactions needed while maintaining high-quality learning.

Quoc Bao Phan, Tuy Tan Nguyen

Quantum circuits can replace classical fusion layers in federated learning with 72 parameters instead of 33K, making multi-agent activity recognition more practical for resource-constrained robotic systems.

This paper presents QFedAgent, a federated learning system for activity recognition across multiple robotic agents. It uses quantum circuits to fuse sensor data (accelerometer and gyroscope) more efficiently than classical neural networks, reducing parameters by 10x while maintaining accuracy on distributed, non-uniform data.

Zhuowei Chen, Liwei Chen, Christian Schunn et al.

Using internal neuron activation patterns to select few-shot examples is more effective than traditional output-based signals, helping identify what the model actually struggles with rather than just guessing from its outputs.

This paper proposes NeuFS, a method for selecting the most useful examples to annotate when adapting large language models to specialized tasks.

Thomas Winninger

RFM-AGOP enables rapid identification of multi-dimensional safety subspaces in LLMs, offering a computationally efficient alternative to existing methods that could scale safety monitoring across larger models.

This paper presents a fast method for identifying multi-dimensional refusal subspaces in large language models using an adapted Recursive Feature Machine (RFM) algorithm.

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

You can now predict LLM inference efficiency on GPUs you've never tested by combining public model specs with GPU specifications—no profiling needed, and it works 4x better than physics-based estimates.

WattGPU predicts GPU power consumption and inference latency for large language models without requiring hardware profiling. Using only public LLM metadata and GPU specs, it generalizes to unseen hardware combinations, achieving 3-4x better accuracy than traditional baselines and helping operators choose efficient GPU-LLM pairings.

Cristian-Gabriel Florea, Stelian Spînu

By running six specialized deep learning models locally on a smartphone with INT8 quantization, VisionAId achieves real-time visual assistance for blind and low-vision users without cloud dependency, while a few-shot learning pipeline lets users teach the system to find their personal objects.

VisionAId is an Android app that helps visually impaired people navigate and interact with their environment using on-device AI models. It combines depth estimation, object detection, and facial recognition to identify obstacles, locate personal items, and recognize faces—all running locally on a smartphone without requiring cloud connectivity, with optional AI for scene descriptions.

Zijian Zhang, Rizhen Hu, Athanasios Glentis et al.

You don't need to update all transformer layers during RL training—focusing on middle layers can match full-model performance while dramatically reducing compute and memory costs.

This paper reveals that training just a single transformer layer during RL fine-tuning can recover most or all of the performance gains from updating the entire model. The authors find that RL improvements concentrate in middle layers, with input and output layers contributing far less, and this pattern holds consistently across different models, algorithms, and tasks.

Giovanni Monea, Nathan Godey, Kianté Brantley et al.

Splitting Transformer computation into separate streams for token prediction and state maintenance improves both training efficiency and model performance—a simple architectural change with consistent gains across scales.

This paper proposes separating two functions in Transformers: predicting the next token and maintaining state for future predictions. The authors design a dual-stream architecture and show it improves language modeling efficiency and downstream task performance by 2-3% compared to standard Transformers.

Patrick Podest, Marco Pichler, Elias Bürger et al.

TiRex-2 enables efficient streaming multivariate forecasting with constant per-patch inference cost and zero-shot generalization, solving the quadratic complexity problem of Transformer-based time series models.

TiRex-2 is a time series foundation model built on xLSTM that handles multiple variables and streaming data efficiently. Unlike Transformer-based models that slow down with longer sequences, TiRex-2 uses a memory-based recurrent design that processes new data at constant cost, even when variables evolve together and some future values are known in advance.

Michael Y. Li, Anthony Zhan, Kanishk Gandhi et al.

You can generate better-coverage samples in parallel by using quasi-Monte Carlo instead of random sampling—achieving the same performance with significantly fewer inference calls, making scaling compute more efficient.

QuasiMoTTo improves inference efficiency by generating correlated rather than independent samples during test-time scaling. Instead of wasting compute on redundant solutions, it uses quasi-Monte Carlo sampling to spread samples across the output space more evenly, achieving the same accuracy with 25-47% fewer samples while maintaining correct marginal distributions for training.

Zekai Chen, Kairui Yang, Xuaner Chen et al.

Federated multimodal graph learning can achieve strong performance while maintaining privacy and interpretability by organizing knowledge into typed semantic codebooks that explicitly track how different modalities and graph structure contribute to predictions.

FedLAB enables federated learning on multimodal graphs (graphs with text, images, and attributes) while preserving privacy by organizing knowledge into traceable semantic codebooks.

Srijan Tiwari, Aditya Chauhan, Manjot Singh

Penalizing radial expansion of neural network activations forces learning of compact, structured representations and dramatically speeds up generalization on algorithmic tasks—a simple geometric insight with practical training benefits.

Neural networks memorize before generalizing on algorithmic tasks because hidden representations inflate radially during training. This paper proposes a geometric penalty that constrains activations to a hypersphere, forcing the network to learn structured circuits faster—accelerating grokking 6x on arithmetic tasks and halving training time for addition.

Philip Zmushko, Egor Petrov, Nursultan Abdullaev et al.

Asynchronous pipeline parallelism with one-step gradient delay is practical for large LLM training if you use the right optimizer; the performance gap with synchronous training can be closed with modern optimizers and error feedback corrections.

This paper shows that asynchronous pipeline parallelism for LLM training isn't fundamentally limited by stale gradients—the problem depends on which optimizer you use. Modern optimizers like Muon handle one-step gradient delays well, while older ones like AdamW struggle.

Xinlei Yu, Gen Li, Qingyi Si et al.

When distilling from privileged teachers or students, routing supervision based on advantage gaps prevents students from learning to exploit information asymmetry instead of real capabilities—improving both LLM and vision-language model performance.

This paper addresses a key problem in on-policy distillation where adding privileged information (extra inputs) to teachers or students creates a 'privilege illusion'—students learn to mimic information asymmetry rather than transferable skills.

Haoran Jin, Xiting Wang, Shijie Ren et al.

When scaling sparse autoencoders for interpretability, enforcing cross-sample consistency prevents features from fragmenting or developing exceptions, making the learned representations more reliable for understanding language model behavior.

This paper identifies and fixes two major problems in Sparse Autoencoders (SAEs) used to interpret language models: feature splitting (where single concepts fragment into multiple latents) and feature absorption (where general features develop arbitrary exceptions).

Nathanaël Jacquier, Maria Vakalopoulou, Mahdi S. Hosseini

Adding soft sparsity regularizers to Top-k sparse autoencoders makes interpretable features more robust and concentrated, without the drawbacks of earlier penalty-based approaches—hard and soft sparsity work better together.

This paper improves sparse autoencoders (SAEs) for interpreting vision models by adding sparsity regularizers to the Top-k SAE architecture. The researchers introduce two penalty methods that work alongside Top-k's hard sparsity constraint to make learned features more interpretable (monosemantic) without hurting reconstruction quality.

Yu-Neng Wang, Sara Achour

Analog hardware can generate images 100x more efficiently than digital systems, but requires rethinking model design to match fixed physics-based dynamics rather than flexible neural networks.

This paper proposes Analog Interaction Systems (AIS), a framework for building generative models on analog hardware like coupled oscillators. The key innovation is bridging the gap between what neural networks can do and what analog physics naturally computes—using time-varying parameters and hidden states to improve expressivity while keeping energy costs 100x lower than digital approaches.

Alina Bazarova, Johann Fredrik Jadebeck, Henrik Zunker et al.

Neural simulation-based inference can replace slow MCMC for fitting complex disease models, running 15-120x faster on GPUs while producing nearly identical results—enabling real-time outbreak analysis.

This paper compares simulation-based inference (SBI) with traditional MCMC methods for fitting epidemiological models to COVID-19 data. SBI uses neural networks to learn the relationship between model parameters and data, enabling much faster Bayesian inference—achieving 15-120x speedups while maintaining accuracy comparable to MCMC.

Lang Huang, Jinglue Xu, Luke Darlow

Before building bigger models, optimize your data preprocessing: context length, normalization strategy, and regularization can close most of the accuracy gap at a fraction of the computational cost.

This paper shows that simple linear models (Ridge regression) can match or beat complex deep learning architectures for time-series forecasting by carefully tuning preprocessing—context length, normalization, and regularization—rather than scaling model size.

Graham Gibson, John Tipton, Kellin Rumsey et al.

You can get reliable uncertainty estimates without expensive retraining by using influence functions and linear algebra—making uncertainty quantification practical for real models.

Ribbon is a fast method for measuring how uncertain a machine-learning model's predictions are. Instead of retraining a model many times (which is expensive), it uses math tricks to estimate uncertainty from a single trained model, working well even when the model assumptions are wrong.

Wen Ye, Peiyan Li, Tingyu Yuan et al.

Test-time scaling for robots works better when you combine reasoning with action planning, track historical context, and use closed-loop feedback—enabling significant performance gains without retraining.

E-TTS is a framework that improves robot manipulation by combining reasoning and action planning at test time, using historical context and feedback loops. It works with existing vision-language-action models without retraining, achieving up to 33% performance gains in simulation and 27% in real-world tasks.

John Sweeney

FisherSketch enables practical source selection for LLM families by measuring task similarity through Fisher alignment signatures (16 KB per task) instead of expensive full Fisher matrices, revealing whether tasks differ in activations, errors, or their interaction.

This paper solves the problem of selecting training data sources for language models that share vocabularies but differ in tasks (like SMILES vs protein sequences).

Sayak Dutta

Recurrent models can match Transformer efficiency by making forget gates content-aware (looking at stored memory) rather than memory-blind, enabling a mathematical solver that speeds up training while improving language understanding.

CARVE improves recurrent neural networks by fixing how they decide what to forget. Instead of gates that only see new incoming data, CARVE's gates look at what's already stored in memory before deciding what to erase. This single change fixes three architectural problems, enables faster training, and achieves better performance on language tasks while using less memory than competing approaches.

Ziyuan Tang, Tianshi Xu, Yousef Saad et al.

HiMuon makes Muon optimization 10-100x faster by processing weight-matrix tiles independently rather than as full matrices, enabling practical use of this advanced optimizer on large models without sacrificing training quality.

This paper introduces Hierarchical Muon (HiMuon), a faster version of the Muon optimizer for training neural networks. Instead of updating all weights at once, HiMuon splits weight matrices into tiles and updates each tile independently, reducing computation from O(r²sK) to O(HWТK) while maintaining similar training performance.

Dong Jing, Tianqi Zhang, Jiaqi Liu et al.

Pretraining action modules on motion structure before vision-language alignment significantly improves robot learning efficiency and cross-embodiment generalization, particularly in data-scarce real-world settings.

This paper proposes a two-stage training approach for robot manipulation models that first learns motion patterns from action trajectories alone, then transfers this knowledge to vision-language-action models.

Hovhannes Tamoyan, Sean Narenthiran, Erik Arakelyan et al.

By combining structured diagnostic reasoning with efficient repository exploration, SHERLOC helps coding agents spend less time searching for bugs and more time fixing them—improving fix success rates by ~6% while cutting token usage by a quarter.

SHERLOC is a framework that helps AI coding agents quickly find and diagnose bugs in large codebases. Instead of just pointing to buggy files, it provides the reasoning and context needed to actually fix them. The system uses reasoning-focused language models with repository tools and achieves state-of-the-art results while using 36% fewer tokens than competing approaches.

Reza Bayat, Ali Behrouz, Aaron Courville

You can improve language model efficiency by tapering MLP width across depth—allocating more capacity to early layers and less to later ones—a free performance gain that works across different architectures.

This paper shows that language models waste parameters by allocating them uniformly across layers. The authors propose Tapered Language Models, which gradually reduce the width of MLPs (the largest parameter-consuming components) from early to later layers. Across multiple architectures and scales, this simple change improves performance without extra cost.

Florian Hübler, Kai Lion, Antonio Orvieto et al.

Muown's effectiveness comes from implicit angular step-size decay; making this explicit in AngularMuown gives you a faster, more controllable optimizer for Transformer pre-training with decoupled learning rate scheduling for directions vs. magnitudes.

This paper analyzes how Muown, a matrix-aware optimizer for training Transformers, implicitly controls step sizes through angular (directional) updates. The authors reformulate this insight into AngularMuown, which explicitly separates angular step-size scheduling from magnitude updates, improving training speed and stability across model sizes from nanoGPT to 1.1B parameter models.

Changxiao Cai, Yuchen Jiao, Gen Li

Diffusion models robustly adapt to low-dimensional structure across a wide range of coefficient choices, meaning practitioners don't need to fine-tune these hyperparameters precisely to get dimension-independent speedups on structured data.

This paper proves that diffusion models can efficiently sample from low-dimensional data structures regardless of how you set their update coefficients, as long as they fall within a broad class. The key finding is that sampling takes only O(k/ε) iterations (where k is the intrinsic dimension), independent of the ambient dimension—showing this efficiency isn't fragile to implementation details.

Liang Su

For on-device AI agents that need to pause, branch, and resume execution frequently, capsules provide sub-millisecond state snapshots and 27x speedup on long contexts—a different optimization target than high-throughput LLM serving.

This paper introduces execution-state capsules, a checkpoint-restore mechanism for LLM serving on resource-constrained devices.

Harshit Singh, Ayush Pratap Singh, Nityanand Mathur

You can add lifelong learning to frozen TTS models by storing pronunciation fixes in a memory network instead of updating weights—enabling fast adaptation to new proper nouns without retraining.

FlowEdit enables text-to-speech systems to learn and remember pronunciation corrections for proper nouns without retraining. It stores corrections as edits in a memory network, then retrieves and applies them at inference time, reducing pronunciation errors by 93% while keeping the original model frozen.

Mingyu Yang, Keye Zheng, Congchao Cheng et al.

MAA enables agents to learn which memory operations consistently help by accumulating cross-batch evidence, making agent self-improvement more efficient and reliable without requiring online training.

This paper addresses a problem in training AI agents: when the same memory operation gets conflicting feedback across different training batches, it's hard to know which operations actually work. MAA solves this by accumulating evidence for each operation across batches and filtering out unreliable ones, improving agent learning while using 75% fewer tokens during training.

Inesh Chakrabarti, David Limpus, Aditi Ghai Rana et al.

4-bit KV cache compression can dramatically speed up multi-turn agent interactions by reducing memory pressure, but requires careful design choices like asymmetric K/V treatment and hardware-specific optimizations to work reliably in production.

This paper optimizes key-value cache memory for AI agents that maintain long conversation histories by compressing KV data to 4-bit precision. The authors develop practical techniques including asymmetric compression, specialized rotations, and GPU-optimized kernels that achieve 3.47x faster response times in later conversation turns while maintaining output quality.

Rostislav Makarov, Timo Gerkmann

You can reuse existing discriminative models (classifiers) for generative tasks by freezing them and training lightweight adapters, cutting the model footprint in half while keeping performance—useful when you already have trained classifiers lying around.

This paper shows how to repurpose a pre-trained speech classifier for generating speech by attaching a lightweight denoising network on top of it. Instead of training separate classifier and diffusion models, the authors freeze the classifier and train only a small adapter to guide generation, reducing memory and computation while maintaining high speech quality.

Fedor Buzaev, Dmitry Efremenko, Egor Bugaev et al.

Evolutionary algorithms can efficiently find good hyperparameter configurations for PINNs by combining fast screening of many candidates with full training of the best ones, avoiding the manual tuning and convergence issues that plague standard approaches.

This paper tackles the challenge of tuning Physics-Informed Neural Networks (PINNs) by proposing a two-stage evolutionary algorithm approach. Instead of manually searching for good hyperparameters, the method first quickly screens many configurations using short training runs, then fully trains the most promising ones.

Siqi Miao, Shitij Govil, Jack P. Rodgers et al.

End-to-end transformers can match or beat graph neural networks on complex physics tasks while being dramatically faster and more memory-efficient—showing that careful architecture design beats multi-stage pipelines.

HEPTv2 is an end-to-end transformer model that reconstructs particle tracks from detector measurements at the Large Hadron Collider. It uses locality-sensitive hashing and sectorized decoding to achieve 98.6% accuracy while running 7-50x faster than competing approaches, making it practical for real-time physics experiments.

José A. Chávez

Timestep embeddings in diffusion models may be redundant—models can achieve competitive image quality without them by inferring noise scales directly from input corruption patterns.

This paper questions whether diffusion models actually need explicit timestep embeddings for denoising. The authors show theoretically and empirically that removing timestep information entirely doesn't significantly hurt performance on image generation tasks, and models can implicitly learn noise levels from corrupted inputs alone.

Mohamed Nabail, Leo Cheng, Jingmin Wang et al.

By jointly reasoning over uncertainty in rewards, dynamics, and values during planning, preference-based RL can achieve sample efficiency comparable to model-based methods while avoiding explicit reward design.

This paper presents UBP2, a method that learns reward models from human preference comparisons while actively exploring the environment. Unlike passive approaches, UBP2 uses ensemble models to balance learning about rewards, environment dynamics, and value functions, enabling efficient sample use during early training stages.

Christopher B. Womack, Shahine Bouabid, Andrei Sokolov et al.

For climate emulators, optimizing training data diversity through iterative refinement produces better generalization than simply using more standard scenarios, even with smaller datasets.

This paper shows that training data quality matters more than quantity for climate AI models. Instead of using many standard climate scenarios, researchers created a method to design fewer but more diverse training scenarios that teach AI models to better predict climate behavior across different conditions—like distinguishing how greenhouse gases versus aerosols affect the climate.

Zhaofeng Wu, Oliver Sieberling, Shawn Tan et al.

Not all transformer layers need the same width—narrowing middle layers while keeping early and late layers wide improves efficiency and performance, suggesting different layers have different computational roles.

This paper proposes Variable-Width Transformers, which use wider layers at the beginning and end of the network while narrowing middle layers. This non-uniform design outperforms standard transformers of the same size on language modeling, while reducing computation by 22% and memory usage by 15%.

Rishit Dagli, Donglai Xiang, Vismay Modi et al.

Sparse adaptive voxel structures with transformers can predict volumetric material properties 16× more efficiently than fixed-grid approaches, enabling high-fidelity physics simulation of complex 3D objects.

AdaVoMP predicts physical material properties (stiffness, elasticity, density) for 3D objects at high resolution. Instead of using fixed voxel grids, it uses a sparse adaptive structure with a transformer model, achieving 16× higher resolution than previous methods while using less computation. This makes 3D assets ready for realistic physics simulations.

Byung-Kwan Lee, Ximing Lu, Shizhe Diao et al.

Teaching small models through prompt-based learning (showing them correct vs incorrect answers to discriminate) works better than traditional distillation or standard RL, especially for models under 1B parameters.

This paper introduces ZPPO, a training method that improves small AI models by learning from larger teacher models without copying their exact outputs. Instead of forcing students to imitate teacher predictions, ZPPO keeps the teacher in the prompt—creating special question formats that help students learn to discriminate correct from incorrect answers and identify their own failure patterns.

Trisha Mittal, Akshay Mehra, Joshua Kimball

Dataset distillation doesn't consistently outperform simpler coreset selection methods on real-world tasks, and coresets are often more computationally efficient for creating compact training datasets.

This paper challenges the effectiveness of dataset distillation by comparing it against simpler coreset selection methods across large-scale benchmarks.

Hongyuan Adam Lu, Z. L. Victor Wei, Qun Zhang et al.

Iterative refinement of latent states through looped computation offers a new scaling axis for world models—you can trade off depth for parameter efficiency without sacrificing simulation quality.

This paper introduces Looped World Models, which use parameter-shared transformer blocks that iteratively refine latent states instead of stacking deep layers. This approach achieves 100x parameter efficiency while maintaining faithful long-horizon simulation, and introduces adaptive computation that automatically adjusts depth based on prediction complexity.

Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin et al.

Fixed-point convergence provides a natural halting mechanism for iterative reasoning models, letting them use fewer steps on easy problems and more on hard ones—without explicit stopping signals.

This paper introduces FPRM, a looped Transformer that solves reasoning tasks by repeatedly applying the same layer until reaching a fixed point, automatically stopping when the model's internal state stabilizes. The approach addresses signal propagation problems in deep networks and adapts computation based on task difficulty.

Yifu Luo, Zeyu Chen, Haoyu Wang et al.

Self-distillation can be adapted for non-autoregressive language models by learning from the model's own future outputs rather than privileged prefixes, achieving better results with 10x fewer training steps than reinforcement learning baselines.

This paper introduces d-OPSD, a self-distillation method designed specifically for diffusion language models (dLLMs) that generate text in arbitrary order rather than left-to-right.

Mufei Li, Shikun Liu, Dongqi Fu et al.

You can efficiently erase stale or harmful information from an LLM's KV cache by learning to replace cached states rather than recomputing—enabling practical context correction in long-context applications without massive latency penalties.

KVEraser is a learned method for efficiently removing unwanted information from an LLM's cached key-value states after processing. Instead of recomputing all tokens after a deleted span (which is slow), it replaces only the cached states of the erased text with learned steering states, achieving near-full-recomputation quality with 3-4x speedup on long-context tasks.

Alper Yıldırım

Forecasting can be performed by passive optical diffraction rather than learned digital networks, achieving competitive or better results while suggesting the forecasting task itself may be fundamentally simpler than transformer-based approaches assume.

HAMON replaces learned digital layers in time-series forecasting with a passive optical system using diffraction and phase masks. Historical data is encoded onto an optical aperture, and future predictions emerge directly from light propagating through trainable phase masks—no digital sequence mixing needed.

Buqiang Xu, Zirui Xue, Dianmou Chen et al.

Effective context pruning for agents requires preserving prompt cache structure—TokenPilot achieves 56-87% cost reduction by removing content conservatively rather than aggressively rewriting prompts.

TokenPilot manages context in long-running AI agents by smartly removing unnecessary information while keeping the prompt cache valid. It uses two strategies: cleaning up noise when information enters the system, and removing old context only when it's no longer useful. This cuts inference costs by 56-87% while maintaining performance.

Mohammed Arif Mainuddin, Najifa Tabassum, Omar Ibne Shahid et al.

You can compress state-of-the-art vision transformers into tiny, CPU-deployable models for real-time fire detection without sacrificing accuracy by using uncertainty-aware, multi-stage knowledge distillation with spatial attention guidance.

HumP-KD is a knowledge distillation framework that compresses large transformer models (Swin-Tiny, ViT-Base) into a lightweight MobileViT-S student for real-time fire detection.

Shikun Liu, Mufei Li, Dongqi Fu et al.

Direct cache-based synthesis enables LLM agents to efficiently combine parallel branches without redundant computation, making multi-agent workflows faster and more aligned with how modern systems actually work.

This paper introduces Parallel-Synthesis, a framework that lets LLM agents directly process cached outputs from multiple parallel worker branches instead of concatenating text. By working with KV caches directly, it reduces computation time by 2.5-11x while maintaining or improving performance across math, code, and reasoning tasks.

Yining Huang

The key insight is that knowledge editing isn't just about writing new information—deciding when NOT to apply edits is equally important. A routing mechanism that separates edit injection from suppression outperforms simply adding more model capacity.

This paper tackles knowledge editing—updating specific facts in language models while keeping other knowledge intact. It proposes a dual-adapter system with a router that decides when to apply edits: some prompts get an edit adapter (to learn new facts), while others get a locality adapter (to preserve original knowledge).

Dimitri Kachler, Damien Sileo, Pascal Denis

You can dramatically speed up identifying problematic training data by training a lightweight encoder to predict influence rankings, rather than computing expensive influence functions directly.

This paper proposes Influcoder, a method that speeds up data attribution for large language models by distilling influence function computations into a compact encoder. Instead of expensive influence calculations, it learns to quickly identify which training samples contributed to specific model outputs, enabling efficient dataset curation and quality filtering.

Yaxin Du, Yifan Zhou, Yujie Ge et al.

Bundling multiple tool calls into executable code blocks dramatically improves agent performance on multi-step tasks—doubling accuracy on some benchmarks by reducing the overhead of step-by-step tool management.

HyperTool lets AI agents call multiple tools in a single code block instead of one at a time, reducing context waste and improving performance. Rather than exposing each tool call separately, agents write code that chains tools together locally, making complex workflows more efficient.

Guo Yu, Wenlin Liu, Yulan Hu et al.

On-policy distillation produces sparse, structured parameter updates that preserve geometric properties of on-policy training despite dense supervision—meaning you can train efficient subnetworks instead of full models without losing performance.

This paper analyzes how on-policy distillation (combining student trajectories with teacher supervision) changes model parameters. The researchers found that parameter updates are sparse and concentrated in specific layers (especially feed-forward networks), yet remain geometrically structured—updates avoid principal weight directions and target near-zero weight coordinates.

Marek Šuppa, Andrej Ridzik, Daniel Hládek et al.

For low-resource languages, adapting existing multilingual embedding models through vocabulary trimming and task-specific fine-tuning can produce efficient, locally-deployable alternatives to large proprietary models without sacrificing performance.

This paper introduces SkMTEB, the first comprehensive benchmark for evaluating text embedding models on Slovak, a low-resource language.

Jialin Gan, Xin Qiu, Guangzhe Chen et al.

Time series and text tokens don't need equal treatment in LLMs—compressing redundant time series patterns and reducing prompt tokens at deeper layers can speed up inference 7.68× without hurting performance.

This paper shows that time series tokens and prompt tokens in language models have different information patterns, so treating them equally wastes computation. The authors develop a compression method that removes redundant frequency patterns from time series data and gradually drops prompt tokens deeper in the model, achieving up to 7.68× faster inference while maintaining or improving accuracy.

Daniel Scalena, Sara Candussio, Luca Bortolussi et al.

Chain-of-thought reasoning in large models contains a sharp 'commitment boundary' where the answer solidifies—you can safely stop reasoning early and save 55% compute without losing performance.

This paper reveals that large language models often decide their final answer early in chain-of-thought reasoning, with many subsequent steps having no causal effect on the result. By measuring when models commit to an answer, researchers show this happens in a single step on average, followed by 'epiphenomenal' reasoning that doesn't change the outcome.

King Yeung Tsang, Zihao Zhao, Vishal Venkataramani et al.

You can train effective multi-agent orchestrators without human labels by learning from the artifacts agents produce—making it practical to build and improve systems that coordinate specialized LLM agents.

This paper introduces OrchRM, a self-supervised method for training orchestrators that coordinate multiple AI agents without human feedback. Instead of expensive rollouts, it learns from intermediate results of agent executions to build a reward model that guides which agent to use when.

Dachun Kai, Jiayao Lu, Yueyi Zhang et al.

Event cameras can significantly boost video super-resolution quality by providing high-temporal-resolution texture cues and motion information, improving detail recovery by up to 1.55 dB PSNR on texture-rich videos.

This paper introduces EvTexture++, a framework that uses event camera data to improve video super-resolution by enhancing texture details and temporal consistency. Event cameras capture high-frequency visual information that helps recover fine details in upscaled videos and reduce flickering artifacts caused by motion.

Sophia Tang, Yuchen Zhu, Molei Tao et al.

You can now fine-tune discrete diffusion models for any-length generation with theoretical guarantees—the method optimizes both token insertion and unmasking policies together, improving reward alignment while maintaining generation flexibility.

A2D2 enables reward-guided fine-tuning of discrete diffusion models that generate sequences of any length. The method jointly optimizes how tokens are inserted and unmasked during generation, plus the inference schedule, using a theoretically grounded approach that converges to reward-optimized outputs without needing target examples.

Shengqiang Zhang, Ruotong Liao, Volker Tresp et al.

By working with a VAR model's native bitwise predictions and residual code composition instead of token streams, you can edit images more precisely while preserving backgrounds—no retraining needed.

BitResEdit is a training-free editor for visual autoregressive image generators that uses two techniques: BitEdit guides bit-level predictions toward text descriptions, while ResEdit applies edits through the model's native residual code structure. This approach preserves unedited regions exactly while making precise, localized changes to match text prompts.

Hoin Jung, Xiaoqian Wang

You can improve RAG retrieval without retraining by combining multiple chunk sizes and weighing them based on uncertainty—fine-grained chunks find relevant content, larger chunks provide context.

This paper proposes UMG-RAG, a training-free retrieval system that combines dense and sparse retrievers across multiple chunk sizes to improve RAG quality. Instead of choosing one chunk size, it intelligently fuses results from different granularities based on confidence estimates, and optionally returns larger parent chunks for better context while using fine-grained chunks for precise retrieval.

Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu

Visual token importance changes as the model processes information deeper—tokens deemed unimportant early on may matter later, so recoverable routing outperforms permanent removal for vision-language tasks.

Vision-language models use thousands of visual tokens, making inference slow. Instead of permanently removing low-scoring tokens, this paper proposes Reroute: tokens can be temporarily skipped and re-evaluated later when they become important. The method works with existing token-reduction techniques and improves performance on grounding tasks without extra computational cost.

Yeongseo Jung, Jaehyeok Kim, Eunseo Jung et al.

Instead of truncating or summarizing conversations (which loses information), C-DIC stores revisable compression states organized by conversation threads, allowing the model to update and share information across turns efficiently without degrading dialogue quality.

This paper tackles the problem of long conversations becoming inefficient and error-prone in dialogue systems. The authors propose C-DIC, a method that compresses conversation history by organizing it into separate memory threads that can be updated and revised at each turn, rather than keeping the full history.

Jadelynn Dao, Milan Ganai, Yasmina Abukhadra et al.

Test-time compute scaling helps embodied AI, but different scaling strategies (reasoning depth, model size, memory) have different costs and benefits—smart routing based on scene context can match stronger models at 65% lower latency.

This paper introduces DIRECT, a routing framework that intelligently allocates test-time compute for vision-language models used as robot planners. Instead of uniformly scaling compute (which increases latency and cost), DIRECT analyzes scene context to decide when to use chain-of-thought reasoning, larger models, or extended memory—achieving better performance per dollar spent on real robots.

Xingjian Diao, Wenbo Li, Yashas Malur Saidutta et al.

Breaking documents into composable micro-adapters instead of monolithic ones improves both efficiency and accuracy for long-document reasoning in LLMs.

This paper introduces Doc-to-Atom, a system that breaks down documents into small, typed knowledge units called 'atoms' and converts each into a tiny adapter for language models. Instead of creating one large adapter per document, the system selectively combines only relevant atoms based on the query, reducing memory usage and improving performance on question-answering tasks.

Songhao Wu, Ang Lv, Ruobing Xie et al.

Aligning router weights with the principal singular directions of experts improves MoE routing efficiency—a simple mathematical principle that scales from 1B to 11B parameter models.

This paper improves Mixture-of-Experts (MoE) models by redesigning how routers select which experts to use. The authors propose aligning each router with the most important direction of its expert using a mathematical technique called Manifold Power Iteration, which helps routers better match tokens to appropriate experts.

Zhi Wei Xu, Torbjörn E. M. Nordling

Robots can now reliably measure human heart rate from RGB cameras in varying lighting by combining spatial-temporal transformers with illumination-aware training, enabling non-contact physiological monitoring for human-robot interaction.

This paper develops a camera-based system for robots to measure human heart rate from video, even when lighting conditions change dramatically.

Yucheng Li, Huiqiang Jiang, Yang Xu et al.

Multi-token prediction with rejection sampling can accelerate RL training by 1.8x, but requires a specialized loss function (TV loss) and pre-training strategy to maintain high acceptance rates as model entropy increases during RL.

This paper tackles a major bottleneck in RL training for large language models: slow rollout generation. The authors show that Multi-Token Prediction (MTP) with rejection sampling can dramatically speed up inference by predicting multiple tokens at once, but acceptance rates drop during RL training due to increasing model entropy.

Weixian Xu, Shilong Liu, Mengdi Wang

Test-time prompt learning now works in real-world multi-dataset scenarios through intelligent task routing and co-evolution—enabling agents to adapt their behavior to diverse, heterogeneous data streams without retraining.

EEVEE is a framework that lets AI agents improve themselves during real-world use by learning better prompts on-the-fly. Unlike existing methods that work on single datasets, EEVEE handles messy real-world data from multiple sources by routing different types of tasks to specialized prompts, then co-evolving both the router and prompts together.

Megan Frisella, Shubham Tiwari, Andy Ruan et al.

By decoupling strategy declaration from runtime execution through an intermediate representation, Piper lets researchers quickly prototype and compose different parallelism strategies without reimplementing the entire distributed training system.

Piper is a distributed training system that separates how you describe a training strategy from how it actually runs. Instead of manually coding each parallelism approach (like data parallelism or pipeline parallelism), users annotate their model and declare scheduling rules.

Wenhao Liu, Hao Shi, Yunhe Li et al.

For reasoning models with long chain-of-thought outputs, you can reduce memory requirements by 50-75% while maintaining accuracy by using hierarchical KV cache allocation instead of uniform budget distribution.

ReasonAlloc optimizes memory usage during AI reasoning by intelligently allocating the KV cache budget across different layers and attention heads during decoding. Instead of treating all layers equally, it identifies which layers need more memory at different reasoning steps and dynamically reallocates resources to the most important attention heads in real-time.

Zhengkai Pan, Peter Potaptchik, Wenxi Yao et al.

Itô maps enable fast, differentiable sampling from stochastic differential equations at inference time, making it practical to do posterior sampling and stochastic control without expensive iterative procedures.

This paper introduces Itô maps, a new way to learn stochastic flow maps that can predict future states from any intermediate point in a stochastic process. Unlike recent one-step generative models that work with deterministic flows, Itô maps handle randomness (Brownian motion) directly, enabling efficient sampling and control in a single forward pass.

Guido Di Federico, Wenchao Teng, Louis J. Durlofsky

When using neural network-based model compression for inverse problems, traditional ensemble Kalman methods can fail due to nonlinearity—rigorous Monte Carlo sampling with surrogate models offers a more reliable alternative.

This paper tackles a key challenge in subsurface modeling: updating geological parameters to match well observations while keeping models realistic. The authors use latent diffusion models to compress high-dimensional geological data into a smaller space, then compare different update methods (ensemble Kalman, MCMC, SMC).

Anton Bolychev, Georgiy Malaniya, Sinan Ibrahim et al.

You can accelerate RL training by blending a working baseline policy with a learnable policy, progressively shifting control to the learner—this keeps success rates high early on while producing a better final policy.

This paper presents a method for training RL policies more efficiently by starting with an existing suboptimal policy and gradually transferring control to a new learning policy. The approach maintains high success rates throughout training and produces a final standalone policy that outperforms the baseline, without requiring the baseline at test time.

Jiarui Yao, Xiangxin Zhou, Penghui Qi et al.

When training LLMs with RL, use smooth regularization on policy shifts instead of hard cutoffs—it gives better training stability without throwing away useful learning signals.

This paper improves how language models learn from reinforcement learning by fixing how we measure when a model's behavior has changed too much during training. Instead of abruptly cutting off gradient updates (like existing methods do), the authors propose DRPO, which smoothly reduces their impact. This keeps training more stable and efficient across different model sizes.

Jisong Cai, Long Ling, Shiwei Chu et al.

Decoupling world prediction and action execution into asynchronous temporal streams—where the world model runs slowly and the action model runs fast—improves both robot control performance and computational efficiency without requiring robot pretraining data.

This paper presents AHA-WAM, a robot control system that separates world prediction from action execution at different speeds. A slow video model learns long-term scene patterns while a fast action model executes short movements by reusing the video model's learned context, enabling responsive closed-loop control without redundant computation.

Shizhe Lin, Ladan Tahvildari

You can reliably detect buggy code from LLM agents using cheap embedding comparisons instead of expensive LLM verification—FASE achieves this with minimal computational overhead.

FASE is a fast method to measure code quality uncertainty in multi-agent AI systems without needing expensive LLM checks. It uses structural and semantic similarity graphs to estimate whether generated code is likely correct, running 300x faster than existing approaches while achieving better accuracy.

Alexander Chulzhanov, Soeren Eberhardt, Arjun Mukherjee

Synthetic data can efficiently teach a model grammatical structure for low-resource languages, but semantic understanding requires authentic data—synthetic bootstrapping works best as a primer before curriculum learning with real examples.

This paper tackles machine translation for Q'eqchi' Mayan, an Indigenous language with almost no digital text. Instead of scraping the web (which violates data sovereignty), researchers created synthetic training data from dictionaries and fine-tuned a multilingual model using LoRA adapters.

Ming Sun, Kun Yuan

For distributed machine learning without a central server, this algorithm achieves state-of-the-art communication efficiency by coupling gossip rounds with batch sizes, meaning you can train faster across networks with fewer total messages sent between nodes.

This paper presents MG-ADSGD, a decentralized learning algorithm where multiple agents optimize a shared problem by communicating only with neighbors. The algorithm combines acceleration techniques with efficient message-passing to achieve better communication efficiency than prior methods, requiring fewer total messages exchanged across the network to reach a solution.

Jamie J. Alnasir

AI workflows on HPC systems need different optimization strategies than traditional scientific computing: focus on containerization for portability, smart job scheduling, explicit feedback mechanisms, and I/O efficiency rather than just raw compute throughput.

This guide offers twelve practical strategies for running AI workloads efficiently on HPC clusters. It addresses the unique challenges of AI workflows—which are iterative and data-driven—compared to traditional scientific computing, covering containerization, job scheduling, feedback loops, and file I/O optimization to help researchers build scalable, reproducible AI pipelines.