Recent AI research papers with accessible summaries. Updated daily from arXiv, summarized for developers who don't read papers regularly.
Shivam Kumar, Nabarun Deb
Discrete diffusions can sample from structured categorical data with provable sample complexity guarantees, and weight-sharing neural networks learn scores more efficiently than fully-connected ones for this task.
This paper develops theoretical guarantees for sampling from high-dimensional categorical distributions using discrete diffusion models.
Daniel McBride, Pratik Khandagale, Cristina Garcia-Cardona et al.
Popular diffusion-based sampling methods have a fundamental limitation for multimodal distributions—they require exponentially long times to transition between well-separated modes, making theoretical convergence guarantees misleading about practical efficiency.
This paper challenges the claim that Wasserstein gradient flows and forward-only diffusion can efficiently sample complex multimodal distributions. Using tools from statistical physics, the authors prove these methods suffer from exponentially slow mixing times when modes are well-separated, regardless of theoretical convergence guarantees.
Carolina Fortuna, Blaz Bertalanic
Multi-agent LLM scaling isn't automatic—task structure and how you combine outputs matter more than team size. On some tasks, adding agents barely helps because the underlying model's systematic biases affect all team members the same way.
This paper analyzes how multi-agent LLM teams scale based on task structure, using Steiner's taxonomy to distinguish disjunctive tasks (where one correct answer helps) from compensatory tasks (where averaging helps).
Zhihao Zhan, Ting Song, Li Dong et al.
Decentralized multi-agent systems can outperform centralized ones by letting workers self-organize through shared workspaces and asynchronous communication, enabling practical scaling to thousands of agents for complex tasks.
Agensh is a multi-agent system that scales to 1,024 agents without needing a central coordinator. Instead of one orchestrator managing all tasks, workers self-organize by sharing a workspace, communicating asynchronously, and claiming their own work.
Alexandre Andre, Shivashriganesh P. Mahato, Vinam Arora et al.
Large-scale neural recording pretraining improves transfer learning, but no single approach generalizes well across behavior prediction, neural dynamics, and anatomical organization—suggesting general-purpose brain models remain an open challenge.
BrainWideBench is a benchmark for evaluating whether neural network models trained on large-scale brain recordings from many mice can learn generalizable representations that transfer to new animals and tasks. The benchmark tests three key abilities: predicting behavior from brain activity, forecasting neural patterns, and recovering brain anatomy.
Zixi Chen, Akshay Vegesna, Samip Dahal et al.
Architectural design choices like recursive depth and boundary operators can fundamentally change scaling laws, enabling exponential compute efficiency improvements—challenging the idea that scaling laws are fixed.
This paper shows that architectural changes—specifically model growth through recursive depth (looping) and boundary operators—can improve how efficiently transformers use compute during training. A 7.4B looping model matches GPT-3 13B's performance with 20× less computation, with efficiency gains that grow larger at bigger scales.
Atindra Jha, Margaret Li, Jure Leskovec et al.
MoE models are significantly more vulnerable to data repetition than dense models—a critical concern as training data becomes scarce. Regularization helps, but the fundamental mismatch between sparsity and repeated data suggests new architectural approaches are needed.
This paper investigates how Mixture-of-Experts (MoE) models—which use sparse, specialized sub-networks—overfit more severely than dense models when training data is repeated.
Hasan Amin, Wei-Kai Chang, Rajiv Khanna
Normalized networks follow an exact mathematical law where a single scalar quantity controls the interaction between learning rate and weight decay—and this interaction is fundamentally unstable, causing training to oscillate predictably rather than settle at a fixed point.
This paper reveals that normalization in neural networks creates a hidden feedback loop between learning rate schedules and weight decay. The authors derive an exact mathematical law governing this interaction and show it produces unstable dynamics that cause training to oscillate rather than converge smoothly.
MohammadHossein Bateni, Zahra Hadizadeh, MohammadTaghi Hajiaghayi et al.
When agents in a network each see partial data and pass only predictions forward, information loss follows a precise mathematical pattern: errors stay constant until the network depth reaches M², then shrink proportionally to M²/D.
This paper solves a theoretical problem about how information degrades when passed through networks of AI agents. Each agent sees only partial data and passes predictions forward, like a game of telephone.
Robert Hu, Carlo Luschi, Paul Balanca
Block-scaled FP4 pretraining can match or exceed current quantization methods while being simpler and faster—no randomized transforms needed, and you can keep FP4 in all layers, not just some.
This paper presents a simpler and more efficient recipe for training large language models using 4-bit floating-point (FP4) precision.
Maksim Utushkin, Andrei Ovsiannikov, Alexander D'yakonov
Multi-hash embeddings and timestamp-indexed neighbor sampling are practical techniques that make GNNs deployable on billion-scale social graphs without sacrificing ranking quality or inference speed.
This paper presents a production-scale GNN system for friend recommendations on massive social graphs (194M users, 28B edges). The key innovations are multi-hash embeddings that reduce ID storage by 98% while maintaining quality, and temporal neighbor sampling optimized to O(log(deg(v)) + k) complexity. The system achieved 16% more friend additions in live A/B testing.
Kairong Luo, Jiarui Cui, Yaorui Yin et al.
You can now train competitive small language models on a budget using consumer hardware and low-precision training—the authors prove it's possible for under $7K and share everything needed to reproduce it.
This paper demonstrates how to train a capable 1.5B language model from scratch for under $7K using consumer GPUs and FP8 precision, making LLM pretraining accessible to resource-constrained teams. The authors release their full training recipe, data, and models openly, and derive a cost scaling law showing similar performance is achievable for ~$4.4K.
Sahil Sharma
After AGI, human employment becomes irrelevant to economic growth, but human welfare depends entirely on ownership policy: without legal protections, the machine economy will grow exponentially while human wealth share decays to zero.
This paper models a post-AGI economy where AI and robots are both producers and consumers, creating a self-sustaining corporate system independent of human demand. It shows that without human ownership stakes, GDP growth becomes decoupled from human welfare—machines reinvest all output for exponential growth while humans face wealth erosion unless laws protect their ownership share.
Tate Berenbaum, Muthaiah Venkatachalam
Pipeline parallelism with pre-compiled shards lets you run 70B models on clusters of consumer AI PCs at interactive speeds by splitting computation across machines and optimizing GPU utilization through careful kernel fusion and speculative decoding.
This paper shows how to run large language models (like Llama 70B) across multiple Intel AI PCs by splitting the model into layers and running each layer on a different machine. Using OpenVINO optimization, speculative decoding, and request interleaving, they achieve near-optimal speed while serving multiple users simultaneously on modest hardware.
Daniel Paulin, Victor Elvira
VARMA models are now computationally practical for high-dimensional time series because optimization cost no longer scales with series length—enabling better forecasts than VAR models on real data.
This paper solves a long-standing computational problem in time series modeling: estimating VARMA models (which capture dependencies efficiently) at scale. The key innovation is making each optimization step independent of series length by using Fourier-based sufficient statistics and a reparametrization that ensures valid models by construction.
Iosif Lytras, Nikolaos Makras, Sotirios Sabanis
SG-TULA provides a theoretically-grounded alternative to standard optimizers for non-smooth, non-convex problems with explicit convergence guarantees—useful for understanding and improving LLM pretraining when standard assumptions break down.
This paper introduces SG-TULA, a sampling algorithm for training machine learning models when the loss landscape is non-smooth, non-convex, and has steep gradients. Unlike existing methods, it works directly with subgradients without expensive smoothing, uses taming techniques for stability, and comes with theoretical guarantees on convergence speed.
Yiming Zhang, Zhonghan Zhao, Wenwei Zhang et al.
Language models can learn more effectively from visual documents than from text-only versions of the same content, suggesting that current pretraining pipelines waste information by converting documents to plain text.
This paper shows that training language models on visual documents (with figures, equations, and layouts intact) outperforms traditional text-only pretraining. The researchers systematically study how to extract knowledge directly from visual representations of documents and web pages, demonstrating that this visual pretraining approach scales efficiently across different model architectures.
Eitan Levin, Venkat Chandrasekaran
Models trained on small inputs can generalize to larger ones if they're continuous with respect to appropriate sampling operations—the paper provides explicit rates and identifies which sampling strategies (replacement, binning, species sampling) work for different problem types.
This paper develops a unified framework for understanding how machine learning models generalize from small to large inputs of variable sizes (like point clouds or graphs). Using random sampling maps, the authors characterize when models can reliably extrapolate to unseen input sizes and how to compress large inputs while preserving model predictions.
Caleb Ziems, William Held, Su Doga Karaca et al.
Larger LLMs will simulate most human behaviors and opinions better, but scaling alone won't fix simulations of cognitive biases, rare populations, or tasks requiring specialized human knowledge—these need targeted research beyond just bigger models.
This paper investigates whether scaling up language models improves their ability to simulate human social behavior and opinions.
David Jurgens
NLP research is migrating from specialized NLP conferences to general machine learning venues, driven partly by citation advantages at ML conferences—a significant shift in the field's institutional center of gravity.
This paper analyzes where NLP research is being published, finding that the field is shifting away from traditional NLP conferences like ACL toward general machine learning venues.
Julius Girardin, Emanuele Troiani, Yizhou Xu et al.
Generalization doesn't scale uniformly with width and data—the relationship changes dramatically across different regimes, with the data's spectral structure determining how performance improves.
This paper analyzes how neural networks generalize as both model size and training data scale together. Using a simplified quadratic network model with structured data, the researchers derive exact formulas showing that generalization error follows different power-law patterns depending on the ratio of parameters to samples, revealing distinct phases like interpolation onset.
Josef Chen
Before building a multi-model system, measure how often all your models fail together—this sets a hard ceiling on possible gains. Standard error correlation metrics won't tell you this, but a simple statistical bound will.
This paper reveals a fundamental limit on multi-model LLM systems: their accuracy gains are capped by how often all models fail together on the same question. The authors measure this 'co-failure rate' across 67 frontier models and show that standard metrics like error correlation miss this ceiling, making it invisible to practitioners.
Jake J. Xia
Collective intelligence isn't about maximizing order—it's about finding the right balance between coordination and flexibility based on what the system needs to achieve.
This paper develops a framework for understanding multi-agent systems by analyzing how individual agent properties (power and response functions) create emergent collective behaviors. It shows that synchronization increases output but can reduce resilience, and derives an optimal balance between order and adaptability based on system objectives.
Zhaofeng Wu, Oliver Sieberling, Shawn Tan et al.
Not all transformer layers need the same width—narrowing middle layers while keeping early and late layers wide improves efficiency and performance, suggesting different layers have different computational roles.
This paper proposes Variable-Width Transformers, which use wider layers at the beginning and end of the network while narrowing middle layers. This non-uniform design outperforms standard transformers of the same size on language modeling, while reducing computation by 22% and memory usage by 15%.
Songhao Wu, Ang Lv, Ruobing Xie et al.
Aligning router weights with the principal singular directions of experts improves MoE routing efficiency—a simple mathematical principle that scales from 1B to 11B parameter models.
This paper improves Mixture-of-Experts (MoE) models by redesigning how routers select which experts to use. The authors propose aligning each router with the most important direction of its expert using a mathematical technique called Manifold Power Iteration, which helps routers better match tokens to appropriate experts.
Duc-Cuong Dang, Andre Opris, Dirk Sudholt
SPEA2's density estimation method is theoretically insufficient for maintaining solution diversity on complex problems; switching to all-pairwise distance calculations fixes this while keeping the algorithm practical.
This paper analyzes the theoretical performance of SPEA2, a popular multi-objective optimization algorithm, and identifies a weakness in how it maintains diversity among non-optimal solutions. The authors propose SPEA2+, an improved version that uses all pairwise distances instead of just nearest-neighbor distances, proving it can efficiently cover Pareto fronts like competing algorithms.
Ming Sun, Kun Yuan
For distributed machine learning without a central server, this algorithm achieves state-of-the-art communication efficiency by coupling gossip rounds with batch sizes, meaning you can train faster across networks with fewer total messages sent between nodes.
This paper presents MG-ADSGD, a decentralized learning algorithm where multiple agents optimize a shared problem by communicating only with neighbors. The algorithm combines acceleration techniques with efficient message-passing to achieve better communication efficiency than prior methods, requiring fewer total messages exchanged across the network to reach a solution.
Amil Dravid, Yasaman Bahri, Alexei A. Efros et al.
Neural networks don't just get better at tasks with scale—their internal neuron populations reorganize predictably, with interpretable neurons becoming more specialized while others remain general, offering a new lens for understanding how model size shapes network structure.
This paper reveals that as neural networks grow larger, certain neurons called Rosetta Neurons become more selective and specialized, following predictable scaling patterns. While these interpretable neurons increase in absolute number, they shrink as a percentage of total neurons, and the remaining neurons become less selective—a phenomenon the authors call Neuron Polarization.
Xu Ouyang, Deyi Liu, Yuhang Cai et al.
LLMs have a fundamental capacity limit based on signal-to-noise ratio: scaling parameters or data without maintaining sufficient signal clarity causes performance degradation, explaining phenomena like catastrophic overtraining and quantization failures that standard scaling laws can't capture.
This paper explains why large language models sometimes get worse with more training or smaller precision—not just better. Using information theory, the authors model LLM training like sending signals through a noisy channel. When you scale up the model or data without keeping the signal clear relative to noise, performance actually drops in a U-shape.
Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong et al.
You can now tune hyperparameters on a single dense model and transfer them directly to MoE models of any size or configuration, eliminating the need for expensive hyperparameter search when scaling with MoE.
Complete-muE is a framework that solves the problem of transferring hyperparameters (like learning rate and weight decay) from dense neural networks to Mixture-of-Experts (MoE) models without expensive retuning.
Ellwil Sharma, Arastu Sharma
Sparse mixture-of-experts routing can solve the problem of conflicting physics domains in foundation models by automatically routing different physics problems to specialized experts while maintaining shared knowledge for universal principles.
This paper tackles negative transfer in multi-physics AI models—where training on different physics problems simultaneously hurts performance. The authors propose Shodh-MoE, which uses sparse expert routing to let different parts of the model specialize in different physics regimes (like fluid dynamics vs. porous media flows) while sharing knowledge where it helps.
Alan Z. Song, Yinjie Chen, Mu Nan et al.
You can build efficient vision transformers by routing all patch interactions through a small set of learned core tokens instead of using all-to-all attention, achieving linear complexity without sacrificing performance.
This paper proposes VECA, a vision transformer that replaces quadratic all-to-all attention with linear-time attention using learned "core" tokens as communication hubs. Instead of every patch attending to every other patch, all patches only interact through a small set of learned cores, reducing computation from O(N²) to O(N) while maintaining competitive accuracy on vision tasks.
Minbin Huang, Han Shi, Chuanyang Zheng et al.
You don't need separate expert sets per layer in MoE models—a shared expert pool with independent routers works better and uses fewer parameters, suggesting the standard per-layer expert allocation is unnecessarily wasteful.
UniPool replaces the standard Mixture-of-Experts design where each layer has its own expert set with a single shared pool of experts accessed by all layers. This reduces redundancy and allows expert parameters to grow sublinearly with model depth while improving performance and reducing parameter count by 30-60% compared to standard MoE.
Nicholas Barnfield, Juno Kim, Eshaan Nichani et al.
Linear memory systems face a fundamental logarithmic penalty for top-1 retrieval but can achieve quadratic capacity if you only need the correct answer ranked highly rather than first—a distinction that matters for building efficient retrieval systems.
This paper analyzes how many key-value pairs a linear memory matrix can store, showing the answer depends on the retrieval task. For winner-take-all retrieval (finding the single best match), capacity scales as d² ≈ n log n due to extreme-value statistics. For listwise retrieval (keeping the correct answer in a top-k list), capacity improves to d² ≈ n.
Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García et al.
Noise in transformers can synchronize token behavior and stabilize learning—a counterintuitive finding that suggests randomness plays a constructive role in how these models process sequences.
This paper proves that transformer models with finite depth and width converge to a stochastic particle system as they scale. The researchers show that token evolution follows a continuous-time process with noise-driven synchronization, meaning random perturbations actually help tokens align rather than diverge.
Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh et al.
You can efficiently extend pretrained LLMs to handle much longer contexts by converting them to hybrid architectures without retraining from scratch—this is more practical than building new models entirely.
This paper presents HyLo, a method to convert pretrained Transformer language models into hybrid architectures that combine Transformers with efficient linear sequence models (like Mamba2). By reusing existing model checkpoints and adding long-context training, HyLo extends context length by 32x while reducing memory use by 90%, enabling 2M-token processing on standard hardware.
Sijie Li, Shanda Li, Haowei Lin et al.
Use active learning to strategically pick which small experiments to run when fitting scaling laws—you can predict large-scale model performance with 90% less compute by choosing experiments that reduce uncertainty about the target region you care about.
Training large AI models costs millions, and figuring out how they'll scale costs millions more. This paper proposes a smarter way to choose which smaller pilot experiments to run so you can accurately predict how a massive training run will perform, using only about 10% of the budget that naive approaches would need.
Max Defez, Filippo Quarenghi, Mathieu Vrac et al.
A single neural network architecture can handle multiple super-resolution scales by adapting just three hyperparameters (noise schedule, context length, and mass conservation), eliminating the need to train separate models for each upscaling factor.
This paper presents a flexible deep-learning framework for video super-resolution that works across different spatial and temporal upscaling factors without retraining from scratch.
Paul Quinlan, Qingguo Li, Xiaodan Zhu
ADAPT solves a critical problem in time-series AI: you can now pre-train on many diverse datasets together instead of just one, making it possible to build generalist foundation models that work across different time-series domains.
This paper introduces ADAPT, a new pre-training method that lets AI models learn from many different time-series datasets simultaneously, even when those datasets have different sizes and structures. By aligning the physical properties of diverse time-series data, the approach enables training a single foundation model on 162 datasets at once—something previous methods couldn't do well.
David Picard, Nicolas Dufour, Lucas Degeorge et al.
You can replace attention with a linear-time polynomial mixer and get similar results with much faster inference—especially valuable for long sequences where attention becomes prohibitively expensive.
PoM replaces the expensive attention mechanism in transformers with a polynomial-based token mixer that runs in linear time instead of quadratic. It compresses all tokens into a learned polynomial representation, letting each token extract relevant context from this compact form.
Takuya Shiba
For robot learning systems, discrete action tokenization creates a hard ceiling on performance gains from better vision models—you need to increase action representation capacity, not just encoder quality, to see improvements.
This paper explains why upgrading vision encoders in robot learning models doesn't always improve performance. The key issue is the 'Compression Gap': when robot actions are represented as discrete tokens (like a limited vocabulary), the token codebook becomes an information bottleneck that prevents improvements from better vision encoders from helping.
Torque Dandachi, Sophia Diggs-Galligan
go-mHC enables efficient learned mixing of residual streams in transformers with a single tunable hyperparameter that trades off between speed and expressivity, potentially unlocking a new dimension for scaling model capacity.
This paper solves a mathematical problem in neural network design: how to efficiently mix information across different processing paths (residual streams) in transformers.
Shashank Subramanian, Alexander Kiefer, Arnur Nigmetov et al.
Neural scaling laws can predict weather model performance and guide efficient resource allocation—models trained with periodic cooldowns outperform standard approaches and enable longer, more accurate forecasts.
This paper studies how neural networks for weather forecasting improve as you scale up the model size, training data, and compute.
Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu et al.
Foundation models can effectively predict clinical outcomes from EHR data, but scaling model size alone doesn't improve performance—you need proportionally more training data, and careful handling of repeated events is critical to avoid inflated evaluation metrics.
RAVEN is a foundation model trained on electronic health records (EHRs) from over one million patients to predict what clinical events will happen at a patient's next visit.
Skyler Seto, Pierre Ablin, Anastasiia Filippova et al.
You can train better domain-specific models by mathematically optimizing how many tokens to spend on general pretraining versus specialized training, rather than using a fixed two-stage recipe.
This paper shows how to efficiently train multiple specialized language models by splitting compute between general pretraining and domain-specific training. Using scaling laws, the authors predict optimal token allocation for each stage, improving performance on reasoning and knowledge tasks across different model sizes.
Xuyang Cao, Qianying Liu, Chuan Xiao et al.
By measuring how much each language helps other languages learn during training, you can predict model performance more accurately and find better language mixture ratios than methods that ignore cross-lingual transfer effects.
This paper treats multilingual language model training as a cooperative game where each language contributes to overall performance. It uses game theory to measure how much each language helps others learn (cross-lingual transfer), then uses these insights to predict the best mix of languages for training data.