Recent AI research papers with accessible summaries. Updated daily from arXiv, summarized for developers who don't read papers regularly.
Jie Wang, Tong Zhu, JiaHao Kuang et al.
ArchSpace enables researchers to test LLM architecture ideas efficiently by predicting large-scale performance from small experiments using scaling laws, creating a shared record of what works and why.
ArchSpace is a public platform for documenting, evaluating, and sharing LLM architecture innovations. It addresses a gap where industry teams test many architectural ideas but rarely publish which succeeded or failed, and academia struggles with small-scale comparisons.
Adi Mashiach
Foundation models can develop unexpected biological reasoning abilities through scale alone, and these emergent capabilities don't follow smooth scaling laws—they appear suddenly at larger model sizes, suggesting qualitative jumps in what AI can do with molecular data.
Bonbon is a foundation model trained on 3.2 trillion tokens of protein-ligand sequences that unexpectedly develops three distinct capabilities for predicting molecular interactions—mechanism of action, binding site location, and bond type—without explicit training for these tasks.
Ales Kubicek, Andrea Caciolai, Liangzu Peng et al.
LSA enables practical ultra-long context (1M+ tokens) by replacing expensive full attention with hierarchical cluster scoring, achieving 22.7x speedup in prefill and 4.6x in decode while preserving model accuracy.
This paper introduces Latent Sparse Attention (LSA), a method that makes attention efficient at ultra-long context lengths (1M+ tokens) by hierarchically clustering keys in a learned latent space. Instead of comparing queries to every key, queries score clusters of keys, reducing computation from quadratic to subquadratic while maintaining accuracy.
Leon Lufkin, Cutter Dawes, Alexander Speicher et al.
ZHA achieves subquadratic attention scaling by using a hierarchical tree indexer, potentially enabling LLMs to handle contexts of hundreds of millions to billions of tokens—a major step toward solving the long-context bottleneck without sacrificing expressiveness.
This paper introduces Zyphra Hierarchical Attention (ZHA), a method that reduces the computational cost of attention in large language models from quadratic to subquadratic scaling.
Drew T. Nguyen, William Fithian
AI time horizon benchmarks need better statistical foundations: a 10x increase in human task time doesn't represent equal difficulty gains across all ranges, which matters for fairly comparing AI capabilities.
This paper examines how AI capabilities are measured using 'time horizons'—the human task completion time at which an AI succeeds 50% of the time. The authors show that the standard linear model for this relationship is flawed and propose better statistical methods using splines and item-response theory, revealing that task difficulty doesn't scale uniformly with human time.
Erin Crawley, Hidenori Tanaka
Collaborative AI agents pose a population-level safety risk distinct from individual agent risks: a small group may be harmless, but once population size crosses a critical threshold, collective cyber capability can explode in a self-reinforcing cycle.
This paper applies ecological theory to AI agent safety, showing that misaligned agents collaborating to conduct cyberattacks could trigger a population explosion once they reach a critical threshold—even if individual agent capability stays constant.
Shivam Kumar, Nabarun Deb
Discrete diffusions can sample from structured categorical data with provable sample complexity guarantees, and weight-sharing neural networks learn scores more efficiently than fully-connected ones for this task.
This paper develops theoretical guarantees for sampling from high-dimensional categorical distributions using discrete diffusion models.
Daniel McBride, Pratik Khandagale, Cristina Garcia-Cardona et al.
Popular diffusion-based sampling methods have a fundamental limitation for multimodal distributions—they require exponentially long times to transition between well-separated modes, making theoretical convergence guarantees misleading about practical efficiency.
This paper challenges the claim that Wasserstein gradient flows and forward-only diffusion can efficiently sample complex multimodal distributions. Using tools from statistical physics, the authors prove these methods suffer from exponentially slow mixing times when modes are well-separated, regardless of theoretical convergence guarantees.
Carolina Fortuna, Blaz Bertalanic
Multi-agent LLM scaling isn't automatic—task structure and how you combine outputs matter more than team size. On some tasks, adding agents barely helps because the underlying model's systematic biases affect all team members the same way.
This paper analyzes how multi-agent LLM teams scale based on task structure, using Steiner's taxonomy to distinguish disjunctive tasks (where one correct answer helps) from compensatory tasks (where averaging helps).
Zhihao Zhan, Ting Song, Li Dong et al.
Decentralized multi-agent systems can outperform centralized ones by letting workers self-organize through shared workspaces and asynchronous communication, enabling practical scaling to thousands of agents for complex tasks.
Agensh is a multi-agent system that scales to 1,024 agents without needing a central coordinator. Instead of one orchestrator managing all tasks, workers self-organize by sharing a workspace, communicating asynchronously, and claiming their own work.
Atindra Jha, Margaret Li, Jure Leskovec et al.
MoE models are significantly more vulnerable to data repetition than dense models—a critical concern as training data becomes scarce. Regularization helps, but the fundamental mismatch between sparsity and repeated data suggests new architectural approaches are needed.
This paper investigates how Mixture-of-Experts (MoE) models—which use sparse, specialized sub-networks—overfit more severely than dense models when training data is repeated.
Hasan Amin, Wei-Kai Chang, Rajiv Khanna
Normalized networks follow an exact mathematical law where a single scalar quantity controls the interaction between learning rate and weight decay—and this interaction is fundamentally unstable, causing training to oscillate predictably rather than settle at a fixed point.
This paper reveals that normalization in neural networks creates a hidden feedback loop between learning rate schedules and weight decay. The authors derive an exact mathematical law governing this interaction and show it produces unstable dynamics that cause training to oscillate rather than converge smoothly.
MohammadHossein Bateni, Zahra Hadizadeh, MohammadTaghi Hajiaghayi et al.
When agents in a network each see partial data and pass only predictions forward, information loss follows a precise mathematical pattern: errors stay constant until the network depth reaches M², then shrink proportionally to M²/D.
This paper solves a theoretical problem about how information degrades when passed through networks of AI agents. Each agent sees only partial data and passes predictions forward, like a game of telephone.
Robert Hu, Carlo Luschi, Paul Balanca
Block-scaled FP4 pretraining can match or exceed current quantization methods while being simpler and faster—no randomized transforms needed, and you can keep FP4 in all layers, not just some.
This paper presents a simpler and more efficient recipe for training large language models using 4-bit floating-point (FP4) precision.
Kairong Luo, Jiarui Cui, Yaorui Yin et al.
You can now train competitive small language models on a budget using consumer hardware and low-precision training—the authors prove it's possible for under $7K and share everything needed to reproduce it.
This paper demonstrates how to train a capable 1.5B language model from scratch for under $7K using consumer GPUs and FP8 precision, making LLM pretraining accessible to resource-constrained teams. The authors release their full training recipe, data, and models openly, and derive a cost scaling law showing similar performance is achievable for ~$4.4K.
Frederik Berenz
Instead of pre-sizing neural network encoders at maximum capacity, you can start small and grow them incrementally as task complexity demands, achieving significant efficiency gains without sacrificing performance.
This paper introduces Successive Capacity Growth (SCG), a method that automatically expands Vision Transformer encoders in world models from minimal size upward, adding attention heads or layers only when needed to improve prediction accuracy.
Tate Berenbaum, Muthaiah Venkatachalam
Pipeline parallelism with pre-compiled shards lets you run 70B models on clusters of consumer AI PCs at interactive speeds by splitting computation across machines and optimizing GPU utilization through careful kernel fusion and speculative decoding.
This paper shows how to run large language models (like Llama 70B) across multiple Intel AI PCs by splitting the model into layers and running each layer on a different machine. Using OpenVINO optimization, speculative decoding, and request interleaving, they achieve near-optimal speed while serving multiple users simultaneously on modest hardware.
Emilien Dupont, Marvin Eisenberger, Borislav Kozlovskii et al.
Machine learning and evolutionary optimization can solve previously intractable mathematical optimization problems, enabling incremental progress on long-standing theoretical computer science questions.
Researchers improved the best-known upper bound on the matrix multiplication exponent (a fundamental constant in computer science) from 2.371339 to 2.371177 by reformulating the underlying optimization problem, applying modern machine learning techniques, and using an evolutionary algorithm called AlphaEvolve to refine the solution.
Iosif Lytras, Nikolaos Makras, Sotirios Sabanis
SG-TULA provides a theoretically-grounded alternative to standard optimizers for non-smooth, non-convex problems with explicit convergence guarantees—useful for understanding and improving LLM pretraining when standard assumptions break down.
This paper introduces SG-TULA, a sampling algorithm for training machine learning models when the loss landscape is non-smooth, non-convex, and has steep gradients. Unlike existing methods, it works directly with subgradients without expensive smoothing, uses taming techniques for stability, and comes with theoretical guarantees on convergence speed.
Arunava Majumder, Marius Krumm, Hendrik Poulsen Nautrup et al.
Shallow quantum generative models become strictly more powerful when augmented with classical randomness—a weak resource that enables long-range correlations impossible for unitary-only circuits at fixed depth.
This paper proves that adding shared classical randomness to shallow quantum circuits can generate probability distributions that require much deeper purely quantum circuits to replicate. The authors show this separation is achievable with simple local operations controlled by a single random bit, and demonstrate how measurement-based quantum computation naturally implements this approach.
Yiming Zhang, Zhonghan Zhao, Wenwei Zhang et al.
Language models can learn more effectively from visual documents than from text-only versions of the same content, suggesting that current pretraining pipelines waste information by converting documents to plain text.
This paper shows that training language models on visual documents (with figures, equations, and layouts intact) outperforms traditional text-only pretraining. The researchers systematically study how to extract knowledge directly from visual representations of documents and web pages, demonstrating that this visual pretraining approach scales efficiently across different model architectures.
Xinyi Wu, Siyuan Liu, Ali Jadbabaie
RoPE frequency usage is determined by training data's dependency structure—frequencies scale inversely with dependency width. This explains why language models use mid-low frequencies and why frequency scaling enables length generalization when test contexts have similar patterns to training data.
This paper explains why transformer models use certain frequencies in Rotary Position Embeddings (RoPE) non-uniformly. The authors show that frequency selection matches the dependency structure in training data, with optimal frequencies inversely proportional to dependency width.
Caleb Ziems, William Held, Su Doga Karaca et al.
Larger LLMs will simulate most human behaviors and opinions better, but scaling alone won't fix simulations of cognitive biases, rare populations, or tasks requiring specialized human knowledge—these need targeted research beyond just bigger models.
This paper investigates whether scaling up language models improves their ability to simulate human social behavior and opinions.
David Jurgens
NLP research is migrating from specialized NLP conferences to general machine learning venues, driven partly by citation advantages at ML conferences—a significant shift in the field's institutional center of gravity.
This paper analyzes where NLP research is being published, finding that the field is shifting away from traditional NLP conferences like ACL toward general machine learning venues.
Josef Chen
Before building a multi-model system, measure how often all your models fail together—this sets a hard ceiling on possible gains. Standard error correlation metrics won't tell you this, but a simple statistical bound will.
This paper reveals a fundamental limit on multi-model LLM systems: their accuracy gains are capped by how often all models fail together on the same question. The authors measure this 'co-failure rate' across 67 frontier models and show that standard metrics like error correlation miss this ceiling, making it invisible to practitioners.
Andrew Bond, Zafer Doğan
Structured correlations in data can boost learning of weak features in GANs, but excessive correlation destabilizes training—there's a sweet spot determined by learning rates and noise.
This paper analyzes GAN training mathematically by studying how a linear generator learns data structure. The key innovation is handling realistic data with correlated features and class-dependent patterns—not just simple diagonal structure. The authors prove training converges to predictable equations and show that smart use of correlations can help weak features become learnable.
Zhaofeng Wu, Oliver Sieberling, Shawn Tan et al.
Not all transformer layers need the same width—narrowing middle layers while keeping early and late layers wide improves efficiency and performance, suggesting different layers have different computational roles.
This paper proposes Variable-Width Transformers, which use wider layers at the beginning and end of the network while narrowing middle layers. This non-uniform design outperforms standard transformers of the same size on language modeling, while reducing computation by 22% and memory usage by 15%.
Hongyuan Adam Lu, Z. L. Victor Wei, Qun Zhang et al.
Iterative refinement of latent states through looped computation offers a new scaling axis for world models—you can trade off depth for parameter efficiency without sacrificing simulation quality.
This paper introduces Looped World Models, which use parameter-shared transformer blocks that iteratively refine latent states instead of stacking deep layers. This approach achieves 100x parameter efficiency while maintaining faithful long-horizon simulation, and introduces adaptive computation that automatically adjusts depth based on prediction complexity.
Songhao Wu, Ang Lv, Ruobing Xie et al.
Aligning router weights with the principal singular directions of experts improves MoE routing efficiency—a simple mathematical principle that scales from 1B to 11B parameter models.
This paper improves Mixture-of-Experts (MoE) models by redesigning how routers select which experts to use. The authors propose aligning each router with the most important direction of its expert using a mathematical technique called Manifold Power Iteration, which helps routers better match tokens to appropriate experts.
Megan Frisella, Shubham Tiwari, Andy Ruan et al.
By decoupling strategy declaration from runtime execution through an intermediate representation, Piper lets researchers quickly prototype and compose different parallelism strategies without reimplementing the entire distributed training system.
Piper is a distributed training system that separates how you describe a training strategy from how it actually runs. Instead of manually coding each parallelism approach (like data parallelism or pipeline parallelism), users annotate their model and declare scheduling rules.
Amil Dravid, Yasaman Bahri, Alexei A. Efros et al.
Neural networks don't just get better at tasks with scale—their internal neuron populations reorganize predictably, with interpretable neurons becoming more specialized while others remain general, offering a new lens for understanding how model size shapes network structure.
This paper reveals that as neural networks grow larger, certain neurons called Rosetta Neurons become more selective and specialized, following predictable scaling patterns. While these interpretable neurons increase in absolute number, they shrink as a percentage of total neurons, and the remaining neurons become less selective—a phenomenon the authors call Neuron Polarization.
Zekun Qi, Xuchuan Chen, Dairu Liu et al.
Scaling motion data and model capacity enables a single generative model to handle complex, dynamic robot control without task-specific training, opening new possibilities for general-purpose robot learning.
Humanoid-GPT is a large Transformer model trained on 2 billion frames of motion capture data to control humanoid robots. Unlike previous shallow models that struggled with dynamic movements, this approach scales both data and model size to achieve zero-shot generalization—meaning it can control motions and tasks it never saw during training.
Xu Ouyang, Deyi Liu, Yuhang Cai et al.
LLMs have a fundamental capacity limit based on signal-to-noise ratio: scaling parameters or data without maintaining sufficient signal clarity causes performance degradation, explaining phenomena like catastrophic overtraining and quantization failures that standard scaling laws can't capture.
This paper explains why large language models sometimes get worse with more training or smaller precision—not just better. Using information theory, the authors model LLM training like sending signals through a noisy channel. When you scale up the model or data without keeping the signal clear relative to noise, performance actually drops in a U-shape.
Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong et al.
You can now tune hyperparameters on a single dense model and transfer them directly to MoE models of any size or configuration, eliminating the need for expensive hyperparameter search when scaling with MoE.
Complete-muE is a framework that solves the problem of transferring hyperparameters (like learning rate and weight decay) from dense neural networks to Mixture-of-Experts (MoE) models without expensive retuning.
Ellwil Sharma, Arastu Sharma
Sparse mixture-of-experts routing can solve the problem of conflicting physics domains in foundation models by automatically routing different physics problems to specialized experts while maintaining shared knowledge for universal principles.
This paper tackles negative transfer in multi-physics AI models—where training on different physics problems simultaneously hurts performance. The authors propose Shodh-MoE, which uses sparse expert routing to let different parts of the model specialize in different physics regimes (like fluid dynamics vs. porous media flows) while sharing knowledge where it helps.
Alan Z. Song, Yinjie Chen, Mu Nan et al.
You can build efficient vision transformers by routing all patch interactions through a small set of learned core tokens instead of using all-to-all attention, achieving linear complexity without sacrificing performance.
This paper proposes VECA, a vision transformer that replaces quadratic all-to-all attention with linear-time attention using learned "core" tokens as communication hubs. Instead of every patch attending to every other patch, all patches only interact through a small set of learned cores, reducing computation from O(N²) to O(N) while maintaining competitive accuracy on vision tasks.
Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García et al.
Noise in transformers can synchronize token behavior and stabilize learning—a counterintuitive finding that suggests randomness plays a constructive role in how these models process sequences.
This paper proves that transformer models with finite depth and width converge to a stochastic particle system as they scale. The researchers show that token evolution follows a continuous-time process with noise-driven synchronization, meaning random perturbations actually help tokens align rather than diverge.
Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh et al.
You can efficiently extend pretrained LLMs to handle much longer contexts by converting them to hybrid architectures without retraining from scratch—this is more practical than building new models entirely.
This paper presents HyLo, a method to convert pretrained Transformer language models into hybrid architectures that combine Transformers with efficient linear sequence models (like Mamba2). By reusing existing model checkpoints and adding long-context training, HyLo extends context length by 32x while reducing memory use by 90%, enabling 2M-token processing on standard hardware.
Max Defez, Filippo Quarenghi, Mathieu Vrac et al.
A single neural network architecture can handle multiple super-resolution scales by adapting just three hyperparameters (noise schedule, context length, and mass conservation), eliminating the need to train separate models for each upscaling factor.
This paper presents a flexible deep-learning framework for video super-resolution that works across different spatial and temporal upscaling factors without retraining from scratch.
A. Sophia Koepke, Daniil Zverev, Shiry Ginosar et al.
Cross-modal alignment between vision and language models is much weaker than previously claimed—it only appears in small-scale experiments and reflects broad semantic overlap, not deep structural convergence.
This paper challenges the popular "Platonic Representation Hypothesis"—the idea that AI models trained on different types of data (like text and images) learn the same underlying representation of reality.
Paul Quinlan, Qingguo Li, Xiaodan Zhu
ADAPT solves a critical problem in time-series AI: you can now pre-train on many diverse datasets together instead of just one, making it possible to build generalist foundation models that work across different time-series domains.
This paper introduces ADAPT, a new pre-training method that lets AI models learn from many different time-series datasets simultaneously, even when those datasets have different sizes and structures. By aligning the physical properties of diverse time-series data, the approach enables training a single foundation model on 162 datasets at once—something previous methods couldn't do well.
David Picard, Nicolas Dufour, Lucas Degeorge et al.
You can replace attention with a linear-time polynomial mixer and get similar results with much faster inference—especially valuable for long sequences where attention becomes prohibitively expensive.
PoM replaces the expensive attention mechanism in transformers with a polynomial-based token mixer that runs in linear time instead of quadratic. It compresses all tokens into a learned polynomial representation, letting each token extract relevant context from this compact form.
Takuya Shiba
For robot learning systems, discrete action tokenization creates a hard ceiling on performance gains from better vision models—you need to increase action representation capacity, not just encoder quality, to see improvements.
This paper explains why upgrading vision encoders in robot learning models doesn't always improve performance. The key issue is the 'Compression Gap': when robot actions are represented as discrete tokens (like a limited vocabulary), the token codebook becomes an information bottleneck that prevents improvements from better vision encoders from helping.
Torque Dandachi, Sophia Diggs-Galligan
go-mHC enables efficient learned mixing of residual streams in transformers with a single tunable hyperparameter that trades off between speed and expressivity, potentially unlocking a new dimension for scaling model capacity.
This paper solves a mathematical problem in neural network design: how to efficiently mix information across different processing paths (residual streams) in transformers.
Shashank Subramanian, Alexander Kiefer, Arnur Nigmetov et al.
Neural scaling laws can predict weather model performance and guide efficient resource allocation—models trained with periodic cooldowns outperform standard approaches and enable longer, more accurate forecasts.
This paper studies how neural networks for weather forecasting improve as you scale up the model size, training data, and compute.
Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu et al.
Foundation models can effectively predict clinical outcomes from EHR data, but scaling model size alone doesn't improve performance—you need proportionally more training data, and careful handling of repeated events is critical to avoid inflated evaluation metrics.
RAVEN is a foundation model trained on electronic health records (EHRs) from over one million patients to predict what clinical events will happen at a patient's next visit.
Skyler Seto, Pierre Ablin, Anastasiia Filippova et al.
You can train better domain-specific models by mathematically optimizing how many tokens to spend on general pretraining versus specialized training, rather than using a fixed two-stage recipe.
This paper shows how to efficiently train multiple specialized language models by splitting compute between general pretraining and domain-specific training. Using scaling laws, the authors predict optimal token allocation for each stage, improving performance on reasoning and knowledge tasks across different model sizes.
Xuyang Cao, Qianying Liu, Chuan Xiao et al.
By measuring how much each language helps other languages learn during training, you can predict model performance more accurately and find better language mixture ratios than methods that ignore cross-lingual transfer effects.
This paper treats multilingual language model training as a cooperative game where each language contributes to overall performance. It uses game theory to measure how much each language helps others learn (cross-lingual transfer), then uses these insights to predict the best mix of languages for training data.