ThinkLLM
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
AboutPrivacyTermsRSS

ThinkLLM

Spot an error in our data? Let us know.

Papers

Recent AI research papers with accessible summaries. Updated daily from arXiv, summarized for developers who don't read papers regularly.

2455 papers2 this month12 topics
AllEvaluation 43Agents 42Applications 39Efficiency 29Reasoning 29Training 27Safety 23Multimodal 18Architecture 9Alignment 8Data 4scaling 2

Nov 9 – Nov 15(2)

ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research

Nov 12, 2026

Jie Wang, Tong Zhu, JiaHao Kuang et al.

ArchSpace enables researchers to test LLM architecture ideas efficiently by predicting large-scale performance from small experiments using scaling laws, creating a shared record of what works and why.

ArchSpace is a public platform for documenting, evaluating, and sharing LLM architecture innovations. It addresses a gap where industry teams test many architectural ideas but rarely publish which succeeded or failed, and academia struggles with small-scale comparisons.

architecturescalingevaluation

Emergent Biological Capabilities in a Foundation Model for Molecular Interactions

Nov 10, 2026

Adi Mashiach

Foundation models can develop unexpected biological reasoning abilities through scale alone, and these emergent capabilities don't follow smooth scaling laws—they appear suddenly at larger model sizes, suggesting qualitative jumps in what AI can do with molecular data.

Bonbon is a foundation model trained on 3.2 trillion tokens of protein-ligand sequences that unexpectedly develops three distinct capabilities for predicting molecular interactions—mechanism of action, binding site location, and bond type—without explicit training for these tasks.

Oct 26 – Nov 1(4)

Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

Oct 30, 2026

Ales Kubicek, Andrea Caciolai, Liangzu Peng et al.

LSA enables practical ultra-long context (1M+ tokens) by replacing expensive full attention with hierarchical cluster scoring, achieving 22.7x speedup in prefill and 4.6x in decode while preserving model accuracy.

This paper introduces Latent Sparse Attention (LSA), a method that makes attention efficient at ultra-long context lengths (1M+ tokens) by hierarchically clustering keys in a learned latent space. Instead of comparing queries to every key, queries score clusters of keys, reducing computation from quadratic to subquadratic while maintaining accuracy.

efficiencyarchitecturescaling

Subquadratic Attention with a Hierarchical Tree-Structured Indexer

Oct 30, 2026

Leon Lufkin, Cutter Dawes, Alexander Speicher et al.

ZHA achieves subquadratic attention scaling by using a hierarchical tree indexer, potentially enabling LLMs to handle contexts of hundreds of millions to billions of tokens—a major step toward solving the long-context bottleneck without sacrificing expressiveness.

This paper introduces Zyphra Hierarchical Attention (ZHA), a method that reduces the computational cost of attention in large language models from quadratic to subquadratic scaling.

Oct 19 – Oct 25(1)

Efficient Edge-aware Attention Network for Graph Generation

Oct 20, 2026

Vincent Jung, Alba Carballo-Castro, Yiming Qin et al.

Graph generative models can achieve significant memory and speed improvements by redesigning attention to work with hardware-optimized implementations, making graph foundation models more practical to scale.

This paper proposes an efficient attention mechanism for graph generation models that reduces GPU memory usage by 65% while maintaining generation quality. The key innovation is a head-wise score modulation approach that incorporates graph structure into attention, implemented using PyTorch's FlexAttention to leverage modern hardware optimizations.

efficiencyarchitecturescaling

Oct 12 – Oct 18(1)

Rethinking Routing Exploration in Modern Mixture-of-Experts

Oct 15, 2026

Bingcong Li, Hao Ma, Michael Muehlebach et al.

When designing MoE models with many experts and shared components, you need to actively manage routing exploration—suppressing unnecessary experts per token can help the model find better expert combinations without getting lost in the combinatorial space.

This paper identifies a fundamental tension in modern Mixture-of-Experts models: fine-grained expert segmentation requires more exploration of the routing space, but shared experts reduce the variability needed for that exploration. The authors propose MirrorMoE, which uses token-adaptive expert suppression to balance these competing needs and improve performance across different model scales.

architecturescalingefficiency

Oct 5 – Oct 11(5)

On the estimation and validity of AI time horizons---a statistical look at the METR plot

Oct 8, 2026

Drew T. Nguyen, William Fithian

AI time horizon benchmarks need better statistical foundations: a 10x increase in human task time doesn't represent equal difficulty gains across all ranges, which matters for fairly comparing AI capabilities.

This paper examines how AI capabilities are measured using 'time horizons'—the human task completion time at which an AI succeeds 50% of the time. The authors show that the standard linear model for this relationship is flawed and propose better statistical methods using splines and item-response theory, revealing that task difficulty doesn't scale uniformly with human time.

evaluationscaling

Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoff

Oct 8, 2026

Erin Crawley, Hidenori Tanaka

Collaborative AI agents pose a population-level safety risk distinct from individual agent risks: a small group may be harmless, but once population size crosses a critical threshold, collective cyber capability can explode in a self-reinforcing cycle.

This paper applies ecological theory to AI agent safety, showing that misaligned agents collaborating to conduct cyberattacks could trigger a population explosion once they reach a critical threshold—even if individual agent capability stays constant.

Sep 28 – Oct 4(6)

Sample complexity bounds for categorical Markov random fields via Discrete Diffusions

Oct 1, 2026

Shivam Kumar, Nabarun Deb

Discrete diffusions can sample from structured categorical data with provable sample complexity guarantees, and weight-sharing neural networks learn scores more efficiently than fully-connected ones for this task.

This paper develops theoretical guarantees for sampling from high-dimensional categorical distributions using discrete diffusion models.

trainingreasoningscaling

Wasserstein Gradient Flows and Forward-Only Diffusion Are Not Enough for Multimodal Sampling

Oct 1, 2026

Daniel McBride, Pratik Khandagale, Cristina Garcia-Cardona et al.

Popular diffusion-based sampling methods have a fundamental limitation for multimodal distributions—they require exponentially long times to transition between well-separated modes, making theoretical convergence guarantees misleading about practical efficiency.

This paper challenges the claim that Wasserstein gradient flows and forward-only diffusion can efficiently sample complex multimodal distributions. Using tools from statistical physics, the authors prove these methods suffer from exponentially slow mixing times when modes are well-separated, regardless of theoretical convergence guarantees.

Sep 21 – Sep 27(2)

Multi-agent Scaling Across Disjunctive and Compensatory Tasks

Sep 25, 2026

Carolina Fortuna, Blaz Bertalanic

Multi-agent LLM scaling isn't automatic—task structure and how you combine outputs matter more than team size. On some tasks, adding agents barely helps because the underlying model's systematic biases affect all team members the same way.

This paper analyzes how multi-agent LLM teams scale based on task structure, using Steiner's taxonomy to distinguish disjunctive tasks (where one correct answer helps) from compensatory tasks (where averaging helps).

agentsscalingreasoning

Agensh: Scaling Organizational Intelligence to 1,024 Agents

Sep 22, 2026

Zhihao Zhan, Ting Song, Li Dong et al.

Decentralized multi-agent systems can outperform centralized ones by letting workers self-organize through shared workspaces and asynchronous communication, enabling practical scaling to thousands of agents for complex tasks.

Agensh is a multi-agent system that scales to 1,024 agents without needing a central coordinator. Instead of one orchestrator managing all tasks, workers self-organize by sharing a workspace, communicating asynchronously, and claiming their own work.

Sep 14 – Sep 20(1)

How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents

Sep 16, 2026

Zixi Chen, Akshay Vegesna, Samip Dahal et al.

Architectural design choices like recursive depth and boundary operators can fundamentally change scaling laws, enabling exponential compute efficiency improvements—challenging the idea that scaling laws are fixed.

This paper shows that architectural changes—specifically model growth through recursive depth (looping) and boundary operators—can improve how efficiently transformers use compute during training. A 7.4B looping model matches GPT-3 13B's performance with 20× less computation, with efficiency gains that grow larger at bigger scales.

architecturescalingefficiency

Sep 7 – Sep 13(2)

Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

Sep 10, 2026

Atindra Jha, Margaret Li, Jure Leskovec et al.

MoE models are significantly more vulnerable to data repetition than dense models—a critical concern as training data becomes scarce. Regularization helps, but the fundamental mismatch between sparsity and repeated data suggests new architectural approaches are needed.

This paper investigates how Mixture-of-Experts (MoE) models—which use sparse, specialized sub-networks—overfit more severely than dense models when training data is repeated.

trainingefficiencyscaling

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

Sep 8, 2026

Hasan Amin, Wei-Kai Chang, Rajiv Khanna

Normalized networks follow an exact mathematical law where a single scalar quantity controls the interaction between learning rate and weight decay—and this interaction is fundamentally unstable, causing training to oscillate predictably rather than settle at a fixed point.

This paper reveals that normalization in neural networks creates a hidden feedback loop between learning rate schedules and weight decay. The authors derive an exact mathematical law governing this interaction and show it produces unstable dynamics that cause training to oscillate rather than converge smoothly.

Aug 31 – Sep 6(5)

Optimal Rates for Agentic Networked Information Aggregation

Sep 4, 2026

MohammadHossein Bateni, Zahra Hadizadeh, MohammadTaghi Hajiaghayi et al.

When agents in a network each see partial data and pass only predictions forward, information loss follows a precise mathematical pattern: errors stay constant until the network depth reaches M², then shrink proportionally to M²/D.

This paper solves a theoretical problem about how information degrades when passed through networks of AI agents. Each agent sees only partial data and passes predictions forward, like a game of telephone.

agentsreasoningscaling

UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

Sep 2, 2026

Robert Hu, Carlo Luschi, Paul Balanca

Block-scaled FP4 pretraining can match or exceed current quantization methods while being simpler and faster—no randomized transforms needed, and you can keep FP4 in all layers, not just some.

This paper presents a simpler and more efficient recipe for training large language models using 4-bit floating-point (FP4) precision.

efficiencytraining

Aug 24 – Aug 30(2)

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Aug 27, 2026

Kairong Luo, Jiarui Cui, Yaorui Yin et al.

You can now train competitive small language models on a budget using consumer hardware and low-precision training—the authors prove it's possible for under $7K and share everything needed to reproduce it.

This paper demonstrates how to train a capable 1.5B language model from scratch for under $7K using consumer GPUs and FP8 precision, making LLM pretraining accessible to resource-constrained teams. The authors release their full training recipe, data, and models openly, and derive a cost scaling law showing similar performance is achievable for ~$4.4K.

efficiencytrainingscaling

Successive Capacity Growth: Task-Complexity-Driven Width and Depth Expansion for Vision Transformer Encoders in JEPA World Models

Aug 27, 2026

Frederik Berenz

Instead of pre-sizing neural network encoders at maximum capacity, you can start small and grow them incrementally as task complexity demands, achieving significant efficiency gains without sacrificing performance.

This paper introduces Successive Capacity Growth (SCG), a method that automatically expands Vision Transformer encoders in world models from minimal size upward, adding attention heads or layers only when needed to improve prediction accuracy.

Aug 17 – Aug 23(2)

Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

Aug 19, 2026

Tate Berenbaum, Muthaiah Venkatachalam

Pipeline parallelism with pre-compiled shards lets you run 70B models on clusters of consumer AI PCs at interactive speeds by splitting computation across machines and optimizing GPU utilization through careful kernel fusion and speculative decoding.

This paper shows how to run large language models (like Llama 70B) across multiple Intel AI PCs by splitting the model into layers and running each layer on a different machine. Using OpenVINO optimization, speculative decoding, and request interleaving, they achieve near-optimal speed while serving multiple users simultaneously on modest hardware.

efficiencyarchitecturescaling

Improving the matrix multiplication exponent with modern optimization and AlphaEvolve

Aug 17, 2026

Emilien Dupont, Marvin Eisenberger, Borislav Kozlovskii et al.

Machine learning and evolutionary optimization can solve previously intractable mathematical optimization problems, enabling incremental progress on long-standing theoretical computer science questions.

Researchers improved the best-known upper bound on the matrix multiplication exponent (a fundamental constant in computer science) from 2.371339 to 2.371177 by reformulating the underlying optimization problem, applying modern machine learning techniques, and using an evolutionary algorithm called AlphaEvolve to refine the solution.

Aug 10 – Aug 16(1)

On the Structural Limits of Machine Learning Decision Systems: An Information-Theoretic, Interaction-Based, and Stochastic-Dynamical Perspective

Aug 13, 2026

Nestor R. Barraza, Gabriel Pena

Machine learning performance has hard mathematical limits set by data structure, not just algorithm choice. Understanding these limits requires proper models of how data is generated, especially for systems with feedback like LLM agents.

This paper examines fundamental limits on what machine learning systems can achieve, showing that performance is constrained by the structure of the data itself rather than just algorithm design.

evaluationreasoningscaling

Aug 3 – Aug 9(2)

The Tamed Subgradient Unadjusted Langevin Algorithm beyond Convexity

Aug 6, 2026

Iosif Lytras, Nikolaos Makras, Sotirios Sabanis

SG-TULA provides a theoretically-grounded alternative to standard optimizers for non-smooth, non-convex problems with explicit convergence guarantees—useful for understanding and improving LLM pretraining when standard assumptions break down.

This paper introduces SG-TULA, a sampling algorithm for training machine learning models when the loss landscape is non-smooth, non-convex, and has steep gradients. Unlike existing methods, it works directly with subgradients without expensive smoothing, uses taming techniques for stability, and comes with theoretical guarantees on convergence speed.

trainingscaling

Representational separation between unitary and channel quantum generative models via shared classical randomness at shallow depth

Aug 5, 2026

Arunava Majumder, Marius Krumm, Hendrik Poulsen Nautrup et al.

Shallow quantum generative models become strictly more powerful when augmented with classical randomness—a weak resource that enables long-range correlations impossible for unitary-only circuits at fixed depth.

This paper proves that adding shared classical randomness to shallow quantum circuits can generate probability distributions that require much deeper purely quantum circuits to replicate. The authors show this separation is achievable with simple local operations controlled by a single random bit, and demonstrate how measurement-based quantum computation naturally implements this approach.

Jul 6 – Jul 12(3)

Scalable Visual Pretraining for Language Intelligence

Jul 10, 2026

Yiming Zhang, Zhonghan Zhao, Wenwei Zhang et al.

Language models can learn more effectively from visual documents than from text-only versions of the same content, suggesting that current pretraining pipelines waste information by converting documents to plain text.

This paper shows that training language models on visual documents (with figures, equations, and layouts intact) outperforms traditional text-only pretraining. The researchers systematically study how to extract knowledge directly from visual representations of documents and web pages, demonstrating that this visual pretraining approach scales efficiently across different model architectures.

multimodalscaling

How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

Jul 8, 2026

Xinyi Wu, Siyuan Liu, Ali Jadbabaie

RoPE frequency usage is determined by training data's dependency structure—frequencies scale inversely with dependency width. This explains why language models use mid-low frequencies and why frequency scaling enables length generalization when test contexts have similar patterns to training data.

This paper explains why transformer models use certain frequencies in Rotary Position Embeddings (RoPE) non-uniformly. The authors show that frequency selection matches the dependency structure in training data, with optimal frequencies inversely proportional to dependency width.

Jun 29 – Jul 5(6)

Will Scaling Improve Social Simulation with LLMs?

Jul 2, 2026

Caleb Ziems, William Held, Su Doga Karaca et al.

Larger LLMs will simulate most human behaviors and opinions better, but scaling alone won't fix simulations of cognitive biases, rare populations, or tasks requiring specialized human knowledge—these need targeted research beyond just bigger models.

This paper investigates whether scaling up language models improves their ability to simulate human social behavior and opinions.

evaluationscalingapplications

The Future of NLP may not be at NLP Conferences: Scholarly Migration Patterns in Natural Language Processing

Jul 2, 2026

David Jurgens

NLP research is migrating from specialized NLP conferences to general machine learning venues, driven partly by citation advantages at ML conferences—a significant shift in the field's institutional center of gravity.

This paper analyzes where NLP research is being published, finding that the field is shifting away from traditional NLP conferences like ACL toward general machine learning venues.

evaluation

Jun 22 – Jun 28(4)

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Jun 25, 2026

Josef Chen

Before building a multi-model system, measure how often all your models fail together—this sets a hard ceiling on possible gains. Standard error correlation metrics won't tell you this, but a simple statistical bound will.

This paper reveals a fundamental limit on multi-model LLM systems: their accuracy gains are capped by how often all models fail together on the same question. The authors measure this 'co-failure rate' across 67 frontier models and show that standard metrics like error correlation miss this ceiling, making it invisible to practitioners.

evaluationscalingagents

Effective Covariance Dynamics in Solvable High-Dimensional GANs

Jun 25, 2026

Andrew Bond, Zafer Doğan

Structured correlations in data can boost learning of weak features in GANs, but excessive correlation destabilizes training—there's a sweet spot determined by learning rates and noise.

This paper analyzes GAN training mathematically by studying how a linear generator learns data structure. The key innovation is handling realistic data with correlated features and class-dependent patterns—not just simple diagonal structure. The authors prove training converges to predictable equations and show that smart use of correlations can help weak features become learnable.

Jun 15 – Jun 21(2)

Variable-Width Transformers

Jun 16, 2026

Zhaofeng Wu, Oliver Sieberling, Shawn Tan et al.

Not all transformer layers need the same width—narrowing middle layers while keeping early and late layers wide improves efficiency and performance, suggesting different layers have different computational roles.

This paper proposes Variable-Width Transformers, which use wider layers at the beginning and end of the network while narrowing middle layers. This non-uniform design outperforms standard transformers of the same size on language modeling, while reducing computation by 22% and memory usage by 15%.

architectureefficiencyscaling

Looped World Models

Jun 16, 2026

Hongyuan Adam Lu, Z. L. Victor Wei, Qun Zhang et al.

Iterative refinement of latent states through looped computation offers a new scaling axis for world models—you can trade off depth for parameter efficiency without sacrificing simulation quality.

This paper introduces Looped World Models, which use parameter-shared transformer blocks that iteratively refine latent states instead of stacking deep layers. This approach achieves 100x parameter efficiency while maintaining faithful long-horizon simulation, and introduces adaptive computation that automatically adjusts depth based on prediction complexity.

architecture

Jun 8 – Jun 14(2)

Redesign Mixture-of-Experts Routers with Manifold Power Iteration

Jun 10, 2026

Songhao Wu, Ang Lv, Ruobing Xie et al.

Aligning router weights with the principal singular directions of experts improves MoE routing efficiency—a simple mathematical principle that scales from 1B to 11B parameter models.

This paper improves Mixture-of-Experts (MoE) models by redesigning how routers select which experts to use. The authors propose aligning each router with the most important direction of its expert using a mathematical technique called Manifold Power Iteration, which helps routers better match tokens to appropriate experts.

architecturescalingefficiency

Piper: A Programmable Distributed Training System

Jun 9, 2026

Megan Frisella, Shubham Tiwari, Andy Ruan et al.

By decoupling strategy declaration from runtime execution through an intermediate representation, Piper lets researchers quickly prototype and compose different parallelism strategies without reimplementing the entire distributed training system.

Piper is a distributed training system that separates how you describe a training strategy from how it actually runs. Instead of manually coding each parallelism approach (like data parallelism or pipeline parallelism), users annotate their model and declare scheduling rules.

Jun 1 – Jun 7(2)

Neuron Populations Exhibit Divergent Selectivity with Scale

Jun 2, 2026

Amil Dravid, Yasaman Bahri, Alexei A. Efros et al.

Neural networks don't just get better at tasks with scale—their internal neuron populations reorganize predictably, with interpretable neurons becoming more specialized while others remain general, offering a new lens for understanding how model size shapes network structure.

This paper reveals that as neural networks grow larger, certain neurons called Rosetta Neurons become more selective and specialized, following predictable scaling patterns. While these interpretable neurons increase in absolute number, they shrink as a percentage of total neurons, and the remaining neurons become less selective—a phenomenon the authors call Neuron Polarization.

scalingarchitecture

Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking

Jun 2, 2026

Zekun Qi, Xuchuan Chen, Dairu Liu et al.

Scaling motion data and model capacity enables a single generative model to handle complex, dynamic robot control without task-specific training, opening new possibilities for general-purpose robot learning.

Humanoid-GPT is a large Transformer model trained on 2 billion frames of motion capture data to control humanoid robots. Unlike previous shallow models that struggled with dynamic movements, this approach scales both data and model size to achieve zero-shot generalization—meaning it can control motions and tasks it never saw during training.

May 25 – May 31(1)

MobileMoE: Scaling On-Device Mixture of Experts

May 26, 2026

Yanbei Chen, Hanxian Huang, Ernie Chang et al.

MoE isn't just for giant models—on mobile devices, moderate sparsity with shared experts is both memory and compute-optimal, letting you get better performance with fewer active parameters than dense models.

MobileMoE brings Mixture-of-Experts (MoE) architecture to phones and edge devices by optimizing it for memory and compute constraints. The models use 0.3-0.9B active parameters but achieve better performance than larger dense models, running 2-4× faster on real smartphones while using less memory.

efficiencyarchitecturescaling

May 18 – May 24(6)

LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

May 22, 2026

Xu Ouyang, Deyi Liu, Yuhang Cai et al.

LLMs have a fundamental capacity limit based on signal-to-noise ratio: scaling parameters or data without maintaining sufficient signal clarity causes performance degradation, explaining phenomena like catastrophic overtraining and quantization failures that standard scaling laws can't capture.

This paper explains why large language models sometimes get worse with more training or smaller precision—not just better. Using information theory, the authors model LLM training like sending signals through a noisy channel. When you scale up the model or data without keeping the signal clear relative to noise, performance actually drops in a U-shape.

scalingtrainingevaluation

Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models

May 22, 2026

Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong et al.

You can now tune hyperparameters on a single dense model and transfer them directly to MoE models of any size or configuration, eliminating the need for expensive hyperparameter search when scaling with MoE.

Complete-muE is a framework that solves the problem of transferring hyperparameters (like learning rate and weight decay) from dense neural networks to Mixture-of-Experts (MoE) models without expensive retuning.

May 11 – May 17(2)

Eradicating Negative Transfer in Multi-Physics Foundation Models via Sparse Mixture-of-Experts Routing

May 14, 2026

Ellwil Sharma, Arastu Sharma

Sparse mixture-of-experts routing can solve the problem of conflicting physics domains in foundation models by automatically routing different physics problems to specialized experts while maintaining shared knowledge for universal principles.

This paper tackles negative transfer in multi-physics AI models—where training on different physics problems simultaneously hurts performance. The authors propose Shodh-MoE, which uses sparse expert routing to let different parts of the model specialize in different physics regimes (like fluid dynamics vs. porous media flows) while sharing knowledge where it helps.

architecturescalingefficiency

Elastic Attention Cores for Scalable Vision Transformers

May 12, 2026

Alan Z. Song, Yinjie Chen, Mu Nan et al.

You can build efficient vision transformers by routing all patch interactions through a small set of learned core tokens instead of using all-to-all attention, achieving linear complexity without sacrificing performance.

This paper proposes VECA, a vision transformer that replaces quadratic all-to-all attention with linear-time attention using learned "core" tokens as communication hubs. Instead of every patch attending to every other patch, all patches only interact through a small set of learned cores, reducing computation from O(N²) to O(N) while maintaining competitive accuracy on vision tasks.

May 4 – May 10(1)

UniPool: A Globally Shared Expert Pool for Mixture-of-Experts

May 7, 2026

Minbin Huang, Han Shi, Chuanyang Zheng et al.

You don't need separate expert sets per layer in MoE models—a shared expert pool with independent routers works better and uses fewer parameters, suggesting the standard per-layer expert allocation is unnecessarily wasteful.

UniPool replaces the standard Mixture-of-Experts design where each layer has its own expert set with a single shared pool of experts accessed by all layers. This reduces redundancy and allows expert parameters to grow sublinearly with model depth while improving performance and reducing parameter count by 30-60% compared to standard MoE.

architectureefficiencyscaling

Apr 27 – May 3(2)

Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

Apr 29, 2026

Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García et al.

Noise in transformers can synchronize token behavior and stabilize learning—a counterintuitive finding that suggests randomness plays a constructive role in how these models process sequences.

This paper proves that transformer models with finite depth and width converge to a stochastic particle system as they scale. The researchers show that token evolution follows a continuous-time process with noise-driven synchronization, meaning random perturbations actually help tokens align rather than diverge.

scalingarchitecturetraining

Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling

Apr 27, 2026

Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh et al.

You can efficiently extend pretrained LLMs to handle much longer contexts by converting them to hybrid architectures without retraining from scratch—this is more practical than building new models entirely.

This paper presents HyLo, a method to convert pretrained Transformer language models into hybrid architectures that combine Transformers with efficient linear sequence models (like Mamba2). By reusing existing model checkpoints and adding long-context training, HyLo extends context length by 32x while reducing memory use by 90%, enabling 2M-token processing on standard hardware.

Apr 20 – Apr 26(2)

A Scale-Adaptive Framework for Joint Spatiotemporal Super-Resolution with Diffusion Models

Apr 23, 2026

Max Defez, Filippo Quarenghi, Mathieu Vrac et al.

A single neural network architecture can handle multiple super-resolution scales by adapting just three hyperparameters (noise schedule, context length, and mass conservation), eliminating the need to train separate models for each upscaling factor.

This paper presents a flexible deep-learning framework for video super-resolution that works across different spatial and temporal upscaling factors without retraining from scratch.

architectureefficiencyscaling

Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

Apr 20, 2026

A. Sophia Koepke, Daniil Zverev, Shiry Ginosar et al.

Cross-modal alignment between vision and language models is much weaker than previously claimed—it only appears in small-scale experiments and reflects broad semantic overlap, not deep structural convergence.

This paper challenges the popular "Platonic Representation Hypothesis"—the idea that AI models trained on different types of data (like text and images) learn the same underlying representation of reality.

Apr 6 – Apr 12(2)

ADAPTive Input Training for Many-to-One Pre-Training on Time-Series Classification

Apr 9, 2026

Paul Quinlan, Qingguo Li, Xiaodan Zhu

ADAPT solves a critical problem in time-series AI: you can now pre-train on many diverse datasets together instead of just one, making it possible to build generalist foundation models that work across different time-series domains.

This paper introduces ADAPT, a new pre-training method that lets AI models learn from many different time-series datasets simultaneously, even when those datasets have different sizes and structures. By aligning the physical properties of diverse time-series data, the approach enables training a single foundation model on 162 datasets at once—something previous methods couldn't do well.

datascaling

PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer

Apr 7, 2026

David Picard, Nicolas Dufour, Lucas Degeorge et al.

You can replace attention with a linear-time polynomial mixer and get similar results with much faster inference—especially valuable for long sequences where attention becomes prohibitively expensive.

PoM replaces the expensive attention mechanism in transformers with a polynomial-based token mixer that runs in linear time instead of quadratic. It compresses all tokens into a learned polynomial representation, letting each token extract relevant context from this compact form.

Mar 30 – Apr 5(4)

The Compression Gap: Why Discrete Tokenization Limits Vision-Language-Action Model Scaling

Apr 3, 2026

Takuya Shiba

For robot learning systems, discrete action tokenization creates a hard ceiling on performance gains from better vision models—you need to increase action representation capacity, not just encoder quality, to see improvements.

This paper explains why upgrading vision encoders in robot learning models doesn't always improve performance. The key issue is the 'Compression Gap': when robot actions are represented as discrete tokens (like a limited vocabulary), the token codebook becomes an information bottleneck that prevents improvements from better vision encoders from helping.

architecturescalingefficiency

go-$m$HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices

Apr 2, 2026

Torque Dandachi, Sophia Diggs-Galligan

go-mHC enables efficient learned mixing of residual streams in transformers with a single tunable hyperparameter that trades off between speed and expressivity, potentially unlocking a new dimension for scaling model capacity.

This paper solves a mathematical problem in neural network design: how to efficiently mix information across different processing paths (residual streams) in transformers.

Mar 23 – Mar 29(2)

On Neural Scaling Laws for Weather Emulation through Continual Training

Mar 26, 2026

Shashank Subramanian, Alexander Kiefer, Arnur Nigmetov et al.

Neural scaling laws can predict weather model performance and guide efficient resource allocation—models trained with periodic cooldowns outperform standard approaches and enable longer, more accurate forecasts.

This paper studies how neural networks for weather forecasting improve as you scale up the model size, training data, and compute.

scalingefficiencytraining

Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction

Mar 25, 2026

Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu et al.

Foundation models can effectively predict clinical outcomes from EHR data, but scaling model size alone doesn't improve performance—you need proportionally more training data, and careful handling of repeated events is critical to avoid inflated evaluation metrics.

RAVEN is a foundation model trained on electronic health records (EHRs) from over one million patients to predict what clinical events will happen at a patient's next visit.

Mar 16 – Mar 22(3)

Optimal Splitting of Language Models from Mixtures to Specialized Domains

Mar 19, 2026

Skyler Seto, Pierre Ablin, Anastasiia Filippova et al.

You can train better domain-specific models by mathematically optimizing how many tokens to spend on general pretraining versus specialized training, rather than using a fixed two-stage recipe.

This paper shows how to efficiently train multiple specialized language models by splitting compute between general pretraining and domain-specific training. Using scaling laws, the authors predict optimal token allocation for each stage, improving performance on reasoning and knowledge tasks across different model sizes.

trainingscalingefficiency

ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws

Mar 18, 2026

Xuyang Cao, Qianying Liu, Chuan Xiao et al.

By measuring how much each language helps other languages learn during training, you can predict model performance more accurately and find better language mixture ratios than methods that ignore cross-lingual transfer effects.

This paper treats multilingual language model training as a cooperative game where each language contributes to overall performance. It uses game theory to measure how much each language helps others learn (cross-lingual transfer), then uses these insights to predict the best mix of languages for training data.

Mar 9 – Mar 15(1)

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

Mar 12, 2026

Zhoujun Cheng, Yutao Xie, Yuxiao Qu et al.

When doing RL training on LLMs, increase parallel rollouts per problem as your compute budget grows, but expect diminishing returns; this single principle helps you allocate compute efficiently across sampling and training.

This paper studies how to optimally distribute computing resources when training language models with reinforcement learning. The researchers found that the number of parallel attempts per problem should increase with total compute budget before leveling off, and this pattern holds whether problems are easy or hard—though for different reasons.

scalingtraining

Jan 20 – Jan 26, 2025(1)

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Jan 22, 2025

DeepSeek-AI, Daya Guo, Dejian Yang

RL alone can teach LLMs to reason step-by-step — DeepSeek-R1 matched o1 performance at a fraction of the training cost and proved that reasoning can be distilled into small models.

Demonstrates that pure reinforcement learning without supervised fine-tuning can produce strong reasoning capabilities. DeepSeek-R1 matches OpenAI o1 on math and coding benchmarks, and distilled versions (1.5B-70B) retain most reasoning ability.

reasoningtrainingscaling

Dec 23 – Dec 29, 2024(1)

DeepSeek-V3 Technical Report

Dec 27, 2024

DeepSeek-AI, Aixin Liu, Bei Feng

Frontier-class performance for $5.6M in compute — DeepSeek-V3 demonstrated that engineering efficiency can close the gap with models costing orders of magnitude more to train.

Presents DeepSeek-V3, a 671B MoE model (37B active) trained on 14.8T tokens for $5.6M in compute. Achieves GPT-4o and Claude 3.5 Sonnet class performance while being fully open-weight, using FP8 training and multi-token prediction.

architecturescalingefficiency

Aug 5 – Aug 11, 2024(1)

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

Aug 6, 2024

Charlie Snell, Jaehoon Lee, Kelvin Xu

Inference-time compute scaling is a viable alternative to model scaling — this paper provided the theoretical foundation for reasoning models like o1 and DeepSeek-R1.

Demonstrates that spending more compute at inference time (via search, verification, or repeated sampling) can be more efficient than training larger models. A smaller model with optimal test-time compute can match a 14x larger model.

reasoningscalingefficiency

Jul 29 – Aug 4, 2024(1)

The Llama 3 Herd of Models

Jul 31, 2024

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey

Open-weight models reached GPT-4 class performance — Llama 3 405B closed the gap between open and proprietary models, making frontier capabilities freely available.

Introduces Llama 3, a family of models up to 405B parameters trained on 15T+ tokens. Includes multimodal extensions for image, video, and speech. The 405B model approaches GPT-4 class performance while being fully open-weight.

scalingtrainingevaluation

Jul 15 – Jul 21, 2024(1)

Qwen2 Technical Report

Jul 15, 2024

An Yang, Baosong Yang, Binyuan Hui

Alibaba produced a globally competitive open model family — Qwen2 demonstrated that Chinese AI labs can match Western frontier models across languages.

Describes Qwen2, a family of models from 0.5B to 72B trained on 7T tokens across 27 languages. The 72B model matches Llama 3 70B while excelling on multilingual and code tasks.

scalingtrainingmultilingual

May 6 – May 12, 2024(1)

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

May 7, 2024

DeepSeek-AI, Aixin Liu, Bei Feng

MLA and fine-grained MoE routing enable frontier-class models at a fraction of the compute cost — DeepSeek-V2 pioneered the architecture that made DeepSeek-V3 and R1 possible.

Introduces Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, achieving strong performance while reducing inference costs dramatically. A 236B model with only 21B active parameters per token.

architectureefficiencyscaling

Jan 8 – Jan 14, 2024(1)

Mixtral of Experts

Jan 8, 2024

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux

Sparse MoE at practical scale works — Mixtral showed that routing tokens to a subset of experts gives large-model quality at small-model inference cost.

Introduces Mixtral 8x7B, a sparse MoE model that routes each token to 2 of 8 expert networks. Matches or outperforms Llama 2 70B and GPT-3.5 while using only 13B active parameters per token.

architectureefficiencyscaling

Dec 18 – Dec 24, 2023(1)

Gemini: A Family of Highly Capable Multimodal Models

Dec 19, 2023

Gemini Team, Rohan Anil, Sebastian Borgeaud

Native multimodal training from the ground up outperforms bolting vision onto language models — Gemini established Google as a direct competitor to GPT-4.

Introduces the Gemini family (Ultra, Pro, Nano), natively multimodal models trained jointly on text, images, audio, and video. Gemini Ultra is the first model to exceed human-expert performance on MMLU.

multimodalscalingevaluation

Jul 17 – Jul 23, 2023(1)

Llama 2: Open Foundation and Fine-Tuned Chat Models

Jul 18, 2023

Hugo Touvron, Louis Martin, Kevin Stone

The first commercially usable open-weight model competitive with ChatGPT — Llama 2 established the template for responsible open-source release of aligned LLMs.

Releases Llama 2, a family of 7B-70B models with a commercially permissive license. Includes Llama 2-Chat, fine-tuned with RLHF, which approaches GPT-3.5 on human evaluations. Details the safety alignment process.

trainingalignmentscaling

Mar 13 – Mar 19, 2023(1)

GPT-4 Technical Report

Mar 15, 2023

OpenAI, Josh Achiam, Steven Adler

GPT-4 reached human-expert level on professional exams and reasoning tasks — a capability threshold that shifted the conversation from whether LLMs can reason to how well.

Describes GPT-4, a large multimodal model accepting image and text inputs. Exhibits human-level performance on professional exams (bar exam 90th percentile) and substantially improves over GPT-3.5 on reasoning, factuality, and safety.

scalingevaluationsafety

Feb 27 – Mar 5, 2023(1)

LLaMA: Open and Efficient Foundation Language Models

Feb 27, 2023

Hugo Touvron, Thibaut Lavril, Gautier Izacard

You do not need the largest model to get the best results — LLaMA proved that smaller models trained on more tokens outperform larger undertrained ones, and ignited the open-source LLM ecosystem.

Introduces LLaMA, a family of foundation models from 7B to 65B parameters trained on publicly available data. LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, demonstrating that smaller models trained on more data can match much larger ones.

architecturescalingefficiency

Dec 5 – Dec 11, 2022(1)

Robust Speech Recognition via Large-Scale Weak Supervision

Dec 6, 2022

Alec Radford, Jong Wook Kim, Tao Xu

Scaling weak supervision on diverse audio data produces speech recognition that works reliably across languages and conditions — Whisper became the default open-source ASR model.

Introduces Whisper, a speech recognition model trained on 680K hours of weakly supervised web audio. Achieves robust performance across languages, accents, and noise conditions without fine-tuning.

multimodalscaling

May 2 – May 8, 2022(1)

OPT: Open Pre-trained Transformer Language Models

May 2, 2022

Susan Zhang, Stephen Roller, Naman Goyal

Meta released a full GPT-3-class model openly — OPT was one of the first major efforts to democratize access to large language model research.

Releases OPT-175B, a suite of decoder-only transformers from 125M to 175B parameters, with full model weights and training code. Aims to enable reproducible research on large language models by matching GPT-3 performance openly.

scalingtraining

Apr 4 – Apr 10, 2022(1)

PaLM: Scaling Language Modeling with Pathways

Apr 5, 2022

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin

Scale produces emergent abilities — PaLM showed that certain capabilities like multi-step reasoning appear suddenly at sufficient scale rather than improving gradually.

Presents PaLM, a 540B parameter dense transformer trained on Google Pathways infrastructure across 6144 TPUs. Demonstrates breakthrough performance on reasoning tasks and shows that scaling continues to yield discontinuous capability improvements.

scalingarchitecture

Mar 28 – Apr 3, 2022(1)

Training Compute-Optimal Large Language Models

Mar 29, 2022

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch

Previous scaling laws overemphasized model size — Chinchilla proved that training tokens should scale proportionally with parameters, reshaping how all subsequent models were trained.

Demonstrates that most large language models are significantly undertrained. For a given compute budget, the optimal strategy allocates roughly equal scaling to model size and training data. Chinchilla (70B, 1.4T tokens) outperforms Gopher (280B, 300B tokens).

scalingtraining

Aug 30 – Sep 5, 2021(1)

Finetuned Language Models Are Zero-Shot Learners

Sep 3, 2021

Jason Wei, Maarten Bosma, Vincent Y. Zhao

Instruction tuning unlocks zero-shot generalization — training on diverse tasks with natural language instructions teaches models to follow novel instructions.

Demonstrates that instruction-tuning a language model on many tasks described via natural language instructions substantially improves zero-shot performance on unseen tasks. FLAN outperforms GPT-3 on many benchmarks despite being 8x smaller.

trainingscaling

Jan 11 – Jan 17, 2021(1)

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Jan 11, 2021

William Fedus, Barret Zoph, Noam Shazeer

Sparse MoE with single-expert routing enables trillion-parameter models at practical compute costs — the architectural pattern behind modern MoE models like Mixtral.

Simplifies Mixture-of-Experts routing by using a single expert per token instead of top-k. This Switch routing enables scaling to trillion-parameter models while keeping computational cost manageable.

architecturescalingefficiency

May 25 – May 31, 2020(1)

Language Models are Few-Shot Learners

May 28, 2020

Tom B. Brown, Benjamin Mann, Nick Ryder

In-context learning emerges at scale — GPT-3 showed that sufficiently large language models can perform tasks from just a few examples in the prompt, without training.

Introduces GPT-3, a 175B parameter autoregressive language model that achieves strong few-shot performance on many NLP benchmarks by conditioning on a few examples in context, without any gradient updates or fine-tuning.

architecturescaling

Jan 20 – Jan 26, 2020(1)

Scaling Laws for Neural Language Models

Jan 23, 2020

Jared Kaplan, Sam McCandlish, Tom Henighan

Language model performance is predictably governed by power laws in parameters, data, and compute — enabling rational decisions about how to scale models.

Discovers that language model performance follows predictable power-law relationships with model size, dataset size, and compute budget. These scaling laws hold across seven orders of magnitude and enable principled allocation of training resources.

scalingtraining
scalingmultimodalapplications
efficiencyarchitecturescaling

Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

Oct 27, 2026

Andrew Szot, Samira Abnar, Omar Attia et al.

When training tool-use agents, prioritize diversity of tools and data sources over raw task quantity—this scales better and produces more capable agents than previous synthetic training approaches.

This paper studies how to train better LLM agents that use tools to complete multi-step tasks. The authors created SynthAPI, a system that generates synthetic training environments with controllable diversity across tools, tasks, and databases.

trainingdatascaling

Timestral

Oct 27, 2026

Adrien Petralia, Alexandre Gavaudan, Artjom Joosen et al.

Decoupling prediction tokens from causal context and pretraining on synthetic data enables time series models to forecast longer horizons more accurately—a practical approach for building better forecasting systems.

Timestral is a family of time series foundation models (5M–209M parameters) that achieves state-of-the-art probabilistic forecasting by combining three key techniques: Decoupled Horizon Forecasting (which trains longer predictions without masking context), native support for known-future covariates, and a two-stage pretraining pipeline on synthetic then real data.

scalingtrainingevaluation
safetyagentsscaling

Toward Joint Optimization of Circuit Depth and Training Data Size in Adaptively Grown Quantum Classifiers

Oct 8, 2026

Saeefa Rubaiyet Nowmi, Md Mahmuduzzaman Kamol, Mohammad Saidur Rahman

Quantum circuit growth algorithms don't follow predictable scaling laws with training data size, and theoretical generalization bounds, while valid, have weak predictive power for practical circuit design decisions.

This paper investigates whether quantum circuit complexity and training data requirements follow predictable scaling laws. Researchers reimplemented Q-FLAIR, an algorithm that grows quantum circuits gate-by-gate, and tested it on MNIST classification with varying dataset sizes.

trainingscalingevaluation

Learning Kilometer-Scale Weather Prediction with Global-Regional Alignment

Oct 8, 2026

Guowen Li, Yang Liu, Yujie Wang et al.

You can build accurate kilometer-scale weather forecasts by aligning global and regional model representations, allowing a single regional model to work with multiple pretrained global forecasters without retraining.

ScaleCast is a weather forecasting framework that combines global weather models with high-resolution regional predictions by aligning their representations across different grid scales. It enables kilometer-scale weather forecasts for local warnings while reusing pretrained global models, eliminating the need to retrain when switching between different global forecast sources.

architecturemultimodalscaling

A Society of Researchers: Designing Institutions for Populations of Autonomous Research Agents

Oct 7, 2026

Ali Asaria, Deep Gandhi, Tony Salomone

Large populations of AI agents need explicit organizational structures—institutions like peer review and resource allocation—to coordinate effectively and produce better research outcomes than unorganized agent swarms.

This paper proposes organizing large populations of autonomous research agents through explicit institutions inspired by academic science. Rather than managing agents individually, the authors create a 'society of researchers' where agents compete for compute resources via proposal review and grants, with a human governor allocating resources.

agentsscalingtraining
evaluationscaling

Scaling Laws for Looped Mixture of Experts

Sep 30, 2026

Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi

Looped MoE models combine two orthogonal efficiency axes: recurrence increases computational depth while sparsity expands capacity, and their scaling laws enable principled design of efficient models that match much larger dense models at the same compute budget.

This paper develops scaling laws that jointly model looped transformers (which use recurrence for computational depth) and Mixture-of-Experts (which use sparsity for capacity).

scalingefficiencyarchitecture

Looped Diffusion Transformer

Sep 30, 2026

Yong Xien Chng, Tianyi Chen, Wenwen Tong et al.

Looping computation within denoising steps is more efficient than scaling model size—you can get better images with fewer parameters by iteratively refining representations through repeated block execution.

This paper proposes Looped Diffusion Transformer, which improves text-to-image generation by repeatedly processing the same Transformer blocks within each denoising step rather than making models larger.

efficiencyarchitecturescaling

Breakdown of Local Denoising as Semantic Speciation

Sep 29, 2026

Guangkuo Liu, Mert Okyay, Yifan F. Zhang et al.

Semantic information explains why generative models transition from local to nonlocal generation at nearly the same time—this connection becomes sharper as models scale, suggesting a fundamental phase transition in how semantic structure emerges.

This paper investigates why generative models show two key moments happening nearly simultaneously: when samples commit to a semantic class (speciation) and when local context becomes insufficient for generation (nonlocality). The authors prove these windows are causally linked through semantic information distribution, showing nonlocality must occur within speciation under natural conditions.

reasoningscalingarchitecture

Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE

Sep 29, 2026

Yu Xu, Yuxin Zhang, Xiao Yang et al.

Video models need different scaling strategies than language models—splitting experts by semantic role and allowing imbalanced routing produces better video generation than forcing uniform expert usage.

This paper proposes SplitMoE, a new way to scale video generation models using a split expert architecture that avoids forcing uniform expert usage.

scalingarchitectureefficiency
agents
scaling
reasoning
trainingscaling
scaling

Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs

Sep 1, 2026

Jingtan Wang, Arun Verma, Xiaoqiang Lin et al.

You don't need to find the perfect SFT-RL budget split—a wide range of allocations work nearly equally well, and you can identify this range using small models and apply it to large ones, saving computation.

This paper solves a practical problem in LLM training: how to split your annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL). Instead of finding one perfect ratio, the authors identify a 'near-optimal region'—a range of budget splits that all perform nearly as well.

trainingefficiencyscaling

When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning

Aug 31, 2026

Hamed Babaei Giglou, Sören Auer, Jennifer D'Souza

When building ontologies with LLMs, model size matters less than you'd expect—architecture and training approach often matter more, and the best model depends heavily on your specific task (term classification vs. relationship extraction).

This paper systematically evaluates how LLM size affects ontology learning—the task of automatically extracting concepts and relationships from text to build knowledge structures.

evaluationscalingapplications

"Train classical, deploy quantum" requires rethinking generalization

Aug 31, 2026

Snehal Raj, Natansh Mathur, Alejandro Perdomo-Ortiz

Minimizing moment-matching losses on classical computers doesn't guarantee quantum generative models will generalize well—the field needs to rethink training objectives or architectures for the "train classical, deploy quantum" paradigm to work reliably.

This paper challenges a popular strategy for quantum machine learning where models are trained classically using moment-matching losses (like MMD²) and then deployed on quantum hardware. The authors show that models trained this way generalize poorly to new data, despite achieving low training loss.

trainingevaluationscaling
architectureefficiencyscaling
scaling
architecturescalingreasoning
trainingscalingreasoning

RMISC: A Large-scale Real-world Multivariate Corpus for Time Series Foundation Models

Jul 7, 2026

Qian Sun, Yong-Ming Tian, Jia-Wei Huang et al.

Real-world multivariate time series data substantially improves foundation model generalization compared to synthetic data, suggesting that practitioners should prioritize real-world datasets when pretraining time series models.

This paper introduces RMISC, a large-scale collection of 200 real-world multivariate time series datasets with 142 billion data points across diverse domains.

datascalingevaluation
scaling

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

Jul 2, 2026

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

You can now predict LLM inference efficiency on GPUs you've never tested by combining public model specs with GPU specifications—no profiling needed, and it works 4x better than physics-based estimates.

WattGPU predicts GPU power consumption and inference latency for large language models without requiring hardware profiling. Using only public LLM metadata and GPU specs, it generalizes to unseen hardware combinations, achieving 3-4x better accuracy than traditional baselines and helping operators choose efficient GPU-LLM pairings.

efficiencyevaluationscaling

TiRex-2: Generalizing TiRex to Multivariate Data and Streaming

Jul 1, 2026

Patrick Podest, Marco Pichler, Elias Bürger et al.

TiRex-2 enables efficient streaming multivariate forecasting with constant per-patch inference cost and zero-shot generalization, solving the quadratic complexity problem of Transformer-based time series models.

TiRex-2 is a time series foundation model built on xLSTM that handles multiple variables and streaming data efficiently. Unlike Transformer-based models that slow down with longer sequences, TiRex-2 uses a memory-based recurrent design that processes new data at constant cost, even when variables evolve together and some future values are known in advance.

architectureefficiencyscaling

One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

Jun 29, 2026

Philip Zmushko, Egor Petrov, Nursultan Abdullaev et al.

Asynchronous pipeline parallelism with one-step gradient delay is practical for large LLM training if you use the right optimizer; the performance gap with synchronous training can be closed with modern optimizers and error feedback corrections.

This paper shows that asynchronous pipeline parallelism for LLM training isn't fundamentally limited by stale gradients—the problem depends on which optimizer you use. Modern optimizers like Muon handle one-step gradient delays well, while older ones like AdamW struggle.

trainingefficiencyscaling

Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent

Jun 29, 2026

Lei Bai, Zongsheng Cao, Yang Chen et al.

You can match trillion-parameter model performance on complex reasoning tasks with a 35B model by focusing on longer, more complex action sequences and multi-domain expertise rather than raw parameter count.

Agents-A1 is a 35B parameter model that achieves trillion-parameter performance on complex tasks by scaling agent horizon—the length and complexity of action sequences—rather than model size.

agentstrainingscaling
trainingarchitecturescaling

Tapered Language Models

Jun 22, 2026

Reza Bayat, Ali Behrouz, Aaron Courville

You can improve language model efficiency by tapering MLP width across depth—allocating more capacity to early layers and less to later ones—a free performance gain that works across different architectures.

This paper shows that language models waste parameters by allocating them uniformly across layers. The authors propose Tapered Language Models, which gradually reduce the width of MLPs (the largest parameter-consuming components) from early to later layers. Across multiple architectures and scales, this simple change improves performance without extra cost.

architectureefficiencyscaling

Diffusion Models Adapt to Low-Dimensional Structure Under Flexible Coefficient Choices

Jun 22, 2026

Changxiao Cai, Yuchen Jiao, Gen Li

Diffusion models robustly adapt to low-dimensional structure across a wide range of coefficient choices, meaning practitioners don't need to fine-tune these hyperparameters precisely to get dimension-independent speedups on structured data.

This paper proves that diffusion models can efficiently sample from low-dimensional data structures regardless of how you set their update coefficients, as long as they fall within a broad class. The key finding is that sampling takes only O(k/ε) iterations (where k is the intrinsic dimension), independent of the ambient dimension—showing this efficiency isn't fragile to implementation details.

efficiencyscalingtraining
efficiency
scaling
trainingefficiencyscaling
scalingagents
trainingscalingefficiency

Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning

May 20, 2026

Benhao Huang, Zhengyang Geng, Zico Kolter

Iterative reasoning models work by learning task-specific attractors in their latent space; scaling test-time compute (more iterations and parallel paths) improves performance on hard problems without needing external verifiers.

This paper explains how AI models can solve hard problems by iteratively refining internal states, like a brain thinking through steps. The key insight is that models learn to create 'attractors'—stable patterns that pull the model toward correct answers.

reasoningscaling

Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

May 20, 2026

Dayal Singh Kalra, Maissam Barkeshli

When scaling up LLM training, use a higher embedding layer learning rate (scaled by model width) to stabilize training and reliably transfer hyperparameters from small to large models—this is the primary reason μP outperforms standard parameterization.

This paper explains why μP (Maximal Update) parameterization works better than standard parameterization for transferring learning rates across different model sizes. The key finding: μP's advantage mainly comes from using a higher learning rate for the embedding layer, which stabilizes training and improves hyperparameter transfer when scaling up language models.

scalingtrainingefficiency

A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability

May 18, 2026

Ruitao Liu, Xinyang Tian, Shuo Chen et al.

For distributed model training, executing tasks based on actual readiness rather than pre-committed schedules can dramatically reduce GPU idle time and improve throughput, especially when computation times vary unpredictably.

This paper introduces RRFP, a runtime system that improves GPU training efficiency by executing ready tasks immediately instead of waiting for a pre-planned order. When training large models across multiple GPUs, unpredictable delays in computation cause stages to sit idle.

trainingefficiencyscaling

Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency

May 18, 2026

Matthew L. Smith, Jonathan P. Shock, Samuel T. Segun et al.

LLM factual accuracy isn't random—it scales predictably with model size and training data frequency, meaning you can estimate what facts a model will reliably remember based on these two factors.

This paper reveals that LLM factual recall follows a predictable pattern based on two factors: model size and how often a topic appears in training data.

scalingevaluationtraining
architectureefficiencyscaling
architectureefficiencyscaling
multimodalevaluationscaling
efficiencyarchitecturescaling
architectureefficiencyscaling

Screening Is Enough

Apr 1, 2026

Ken M. Nakanishi

Screening attention removes the need for global competition among keys by using absolute relevance thresholds, achieving 40% parameter reduction and 3.2× faster inference compared to Transformers.

This paper introduces Multiscreen, a language model architecture that replaces standard softmax attention with a 'screening' mechanism. Instead of distributing attention weights across all keys, screening evaluates each key against a threshold to decide which ones are relevant, eliminating the need for keys to compete with each other.

architectureefficiencyscaling

Rethinking Language Model Scaling under Transferable Hypersphere Optimization

Mar 30, 2026

Liliang Ren, Yang Liu, Yelong Shen et al.

Hypersphere-constrained optimization enables predictable scaling of language models with a single transferable learning rate, eliminating expensive hyperparameter retuning when scaling up and improving training stability.

This paper introduces HyperP, a framework for scaling language models more efficiently by constraining weights to a hypersphere during training. The key innovation is showing that a single learning rate tuned at small scale transfers reliably across different model sizes, depths, and training amounts—achieving 1.58× better compute efficiency while maintaining training stability.

trainingscalingefficiency
applications
scaling
scalingtraining

Mixture-of-Depths Attention

Mar 16, 2026

Lianghui Zhu, Yuxin Fang, Bencheng Liao et al.

MoDA lets deep language models selectively attend to earlier layers, preventing information loss as models get deeper while adding only 3.7% computational overhead.

This paper introduces Mixture-of-Depths Attention (MoDA), a mechanism that lets attention heads skip layers by accessing key-value pairs from both the current and earlier layers. This solves a problem in very deep language models where useful information gets diluted as it passes through many layers.

architectureefficiencyscaling