Papers

Caleb Ziems, William Held, Su Doga Karaca et al.

Larger LLMs will simulate most human behaviors and opinions better, but scaling alone won't fix simulations of cognitive biases, rare populations, or tasks requiring specialized human knowledge—these need targeted research beyond just bigger models.

This paper investigates whether scaling up language models improves their ability to simulate human social behavior and opinions.

David Jurgens

NLP research is migrating from specialized NLP conferences to general machine learning venues, driven partly by citation advantages at ML conferences—a significant shift in the field's institutional center of gravity.

This paper analyzes where NLP research is being published, finding that the field is shifting away from traditional NLP conferences like ACL toward general machine learning venues.

Julius Girardin, Emanuele Troiani, Yizhou Xu et al.

Generalization doesn't scale uniformly with width and data—the relationship changes dramatically across different regimes, with the data's spectral structure determining how performance improves.

This paper analyzes how neural networks generalize as both model size and training data scale together. Using a simplified quadratic network model with structured data, the researchers derive exact formulas showing that generalization error follows different power-law patterns depending on the ratio of parameters to samples, revealing distinct phases like interpolation onset.

Josef Chen

Before building a multi-model system, measure how often all your models fail together—this sets a hard ceiling on possible gains. Standard error correlation metrics won't tell you this, but a simple statistical bound will.

This paper reveals a fundamental limit on multi-model LLM systems: their accuracy gains are capped by how often all models fail together on the same question. The authors measure this 'co-failure rate' across 67 frontier models and show that standard metrics like error correlation miss this ceiling, making it invisible to practitioners.

Jake J. Xia

Collective intelligence isn't about maximizing order—it's about finding the right balance between coordination and flexibility based on what the system needs to achieve.

This paper develops a framework for understanding multi-agent systems by analyzing how individual agent properties (power and response functions) create emergent collective behaviors. It shows that synchronization increases output but can reduce resilience, and derives an optimal balance between order and adaptability based on system objectives.

Zhaofeng Wu, Oliver Sieberling, Shawn Tan et al.

Not all transformer layers need the same width—narrowing middle layers while keeping early and late layers wide improves efficiency and performance, suggesting different layers have different computational roles.

This paper proposes Variable-Width Transformers, which use wider layers at the beginning and end of the network while narrowing middle layers. This non-uniform design outperforms standard transformers of the same size on language modeling, while reducing computation by 22% and memory usage by 15%.

Songhao Wu, Ang Lv, Ruobing Xie et al.

Aligning router weights with the principal singular directions of experts improves MoE routing efficiency—a simple mathematical principle that scales from 1B to 11B parameter models.

This paper improves Mixture-of-Experts (MoE) models by redesigning how routers select which experts to use. The authors propose aligning each router with the most important direction of its expert using a mathematical technique called Manifold Power Iteration, which helps routers better match tokens to appropriate experts.

Duc-Cuong Dang, Andre Opris, Dirk Sudholt

SPEA2's density estimation method is theoretically insufficient for maintaining solution diversity on complex problems; switching to all-pairwise distance calculations fixes this while keeping the algorithm practical.

This paper analyzes the theoretical performance of SPEA2, a popular multi-objective optimization algorithm, and identifies a weakness in how it maintains diversity among non-optimal solutions. The authors propose SPEA2+, an improved version that uses all pairwise distances instead of just nearest-neighbor distances, proving it can efficiently cover Pareto fronts like competing algorithms.

Ming Sun, Kun Yuan

For distributed machine learning without a central server, this algorithm achieves state-of-the-art communication efficiency by coupling gossip rounds with batch sizes, meaning you can train faster across networks with fewer total messages sent between nodes.

This paper presents MG-ADSGD, a decentralized learning algorithm where multiple agents optimize a shared problem by communicating only with neighbors. The algorithm combines acceleration techniques with efficient message-passing to achieve better communication efficiency than prior methods, requiring fewer total messages exchanged across the network to reach a solution.

Amil Dravid, Yasaman Bahri, Alexei A. Efros et al.

Neural networks don't just get better at tasks with scale—their internal neuron populations reorganize predictably, with interpretable neurons becoming more specialized while others remain general, offering a new lens for understanding how model size shapes network structure.

This paper reveals that as neural networks grow larger, certain neurons called Rosetta Neurons become more selective and specialized, following predictable scaling patterns. While these interpretable neurons increase in absolute number, they shrink as a percentage of total neurons, and the remaining neurons become less selective—a phenomenon the authors call Neuron Polarization.

Yanbei Chen, Hanxian Huang, Ernie Chang et al.

MoE isn't just for giant models—on mobile devices, moderate sparsity with shared experts is both memory and compute-optimal, letting you get better performance with fewer active parameters than dense models.

MobileMoE brings Mixture-of-Experts (MoE) architecture to phones and edge devices by optimizing it for memory and compute constraints. The models use 0.3-0.9B active parameters but achieve better performance than larger dense models, running 2-4× faster on real smartphones while using less memory.

Xu Ouyang, Deyi Liu, Yuhang Cai et al.

LLMs have a fundamental capacity limit based on signal-to-noise ratio: scaling parameters or data without maintaining sufficient signal clarity causes performance degradation, explaining phenomena like catastrophic overtraining and quantization failures that standard scaling laws can't capture.

This paper explains why large language models sometimes get worse with more training or smaller precision—not just better. Using information theory, the authors model LLM training like sending signals through a noisy channel. When you scale up the model or data without keeping the signal clear relative to noise, performance actually drops in a U-shape.

Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong et al.

You can now tune hyperparameters on a single dense model and transfer them directly to MoE models of any size or configuration, eliminating the need for expensive hyperparameter search when scaling with MoE.

Complete-muE is a framework that solves the problem of transferring hyperparameters (like learning rate and weight decay) from dense neural networks to Mixture-of-Experts (MoE) models without expensive retuning.

Ellwil Sharma, Arastu Sharma

Sparse mixture-of-experts routing can solve the problem of conflicting physics domains in foundation models by automatically routing different physics problems to specialized experts while maintaining shared knowledge for universal principles.

This paper tackles negative transfer in multi-physics AI models—where training on different physics problems simultaneously hurts performance. The authors propose Shodh-MoE, which uses sparse expert routing to let different parts of the model specialize in different physics regimes (like fluid dynamics vs. porous media flows) while sharing knowledge where it helps.

Alan Z. Song, Yinjie Chen, Mu Nan et al.

You can build efficient vision transformers by routing all patch interactions through a small set of learned core tokens instead of using all-to-all attention, achieving linear complexity without sacrificing performance.

This paper proposes VECA, a vision transformer that replaces quadratic all-to-all attention with linear-time attention using learned "core" tokens as communication hubs. Instead of every patch attending to every other patch, all patches only interact through a small set of learned cores, reducing computation from O(N²) to O(N) while maintaining competitive accuracy on vision tasks.

Minbin Huang, Han Shi, Chuanyang Zheng et al.

You don't need separate expert sets per layer in MoE models—a shared expert pool with independent routers works better and uses fewer parameters, suggesting the standard per-layer expert allocation is unnecessarily wasteful.

UniPool replaces the standard Mixture-of-Experts design where each layer has its own expert set with a single shared pool of experts accessed by all layers. This reduces redundancy and allows expert parameters to grow sublinearly with model depth while improving performance and reducing parameter count by 30-60% compared to standard MoE.

Nicholas Barnfield, Juno Kim, Eshaan Nichani et al.

Linear memory systems face a fundamental logarithmic penalty for top-1 retrieval but can achieve quadratic capacity if you only need the correct answer ranked highly rather than first—a distinction that matters for building efficient retrieval systems.

This paper analyzes how many key-value pairs a linear memory matrix can store, showing the answer depends on the retrieval task. For winner-take-all retrieval (finding the single best match), capacity scales as d² ≈ n log n due to extreme-value statistics. For listwise retrieval (keeping the correct answer in a top-k list), capacity improves to d² ≈ n.

Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García et al.

Noise in transformers can synchronize token behavior and stabilize learning—a counterintuitive finding that suggests randomness plays a constructive role in how these models process sequences.

This paper proves that transformer models with finite depth and width converge to a stochastic particle system as they scale. The researchers show that token evolution follows a continuous-time process with noise-driven synchronization, meaning random perturbations actually help tokens align rather than diverge.

Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh et al.

You can efficiently extend pretrained LLMs to handle much longer contexts by converting them to hybrid architectures without retraining from scratch—this is more practical than building new models entirely.

This paper presents HyLo, a method to convert pretrained Transformer language models into hybrid architectures that combine Transformers with efficient linear sequence models (like Mamba2). By reusing existing model checkpoints and adding long-context training, HyLo extends context length by 32x while reducing memory use by 90%, enabling 2M-token processing on standard hardware.

Sijie Li, Shanda Li, Haowei Lin et al.

Use active learning to strategically pick which small experiments to run when fitting scaling laws—you can predict large-scale model performance with 90% less compute by choosing experiments that reduce uncertainty about the target region you care about.

Training large AI models costs millions, and figuring out how they'll scale costs millions more. This paper proposes a smarter way to choose which smaller pilot experiments to run so you can accurately predict how a massive training run will perform, using only about 10% of the budget that naive approaches would need.

Max Defez, Filippo Quarenghi, Mathieu Vrac et al.

A single neural network architecture can handle multiple super-resolution scales by adapting just three hyperparameters (noise schedule, context length, and mass conservation), eliminating the need to train separate models for each upscaling factor.

This paper presents a flexible deep-learning framework for video super-resolution that works across different spatial and temporal upscaling factors without retraining from scratch.

Paul Quinlan, Qingguo Li, Xiaodan Zhu

ADAPT solves a critical problem in time-series AI: you can now pre-train on many diverse datasets together instead of just one, making it possible to build generalist foundation models that work across different time-series domains.

This paper introduces ADAPT, a new pre-training method that lets AI models learn from many different time-series datasets simultaneously, even when those datasets have different sizes and structures. By aligning the physical properties of diverse time-series data, the approach enables training a single foundation model on 162 datasets at once—something previous methods couldn't do well.

David Picard, Nicolas Dufour, Lucas Degeorge et al.

You can replace attention with a linear-time polynomial mixer and get similar results with much faster inference—especially valuable for long sequences where attention becomes prohibitively expensive.

PoM replaces the expensive attention mechanism in transformers with a polynomial-based token mixer that runs in linear time instead of quadratic. It compresses all tokens into a learned polynomial representation, letting each token extract relevant context from this compact form.

Takuya Shiba

For robot learning systems, discrete action tokenization creates a hard ceiling on performance gains from better vision models—you need to increase action representation capacity, not just encoder quality, to see improvements.

This paper explains why upgrading vision encoders in robot learning models doesn't always improve performance. The key issue is the 'Compression Gap': when robot actions are represented as discrete tokens (like a limited vocabulary), the token codebook becomes an information bottleneck that prevents improvements from better vision encoders from helping.

Torque Dandachi, Sophia Diggs-Galligan

go-mHC enables efficient learned mixing of residual streams in transformers with a single tunable hyperparameter that trades off between speed and expressivity, potentially unlocking a new dimension for scaling model capacity.

This paper solves a mathematical problem in neural network design: how to efficiently mix information across different processing paths (residual streams) in transformers.

Shashank Subramanian, Alexander Kiefer, Arnur Nigmetov et al.

Neural scaling laws can predict weather model performance and guide efficient resource allocation—models trained with periodic cooldowns outperform standard approaches and enable longer, more accurate forecasts.

This paper studies how neural networks for weather forecasting improve as you scale up the model size, training data, and compute.

Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu et al.

Foundation models can effectively predict clinical outcomes from EHR data, but scaling model size alone doesn't improve performance—you need proportionally more training data, and careful handling of repeated events is critical to avoid inflated evaluation metrics.

RAVEN is a foundation model trained on electronic health records (EHRs) from over one million patients to predict what clinical events will happen at a patient's next visit.

Skyler Seto, Pierre Ablin, Anastasiia Filippova et al.

You can train better domain-specific models by mathematically optimizing how many tokens to spend on general pretraining versus specialized training, rather than using a fixed two-stage recipe.

This paper shows how to efficiently train multiple specialized language models by splitting compute between general pretraining and domain-specific training. Using scaling laws, the authors predict optimal token allocation for each stage, improving performance on reasoning and knowledge tasks across different model sizes.

Xuyang Cao, Qianying Liu, Chuan Xiao et al.

By measuring how much each language helps other languages learn during training, you can predict model performance more accurately and find better language mixture ratios than methods that ignore cross-lingual transfer effects.

This paper treats multilingual language model training as a cooperative game where each language contributes to overall performance. It uses game theory to measure how much each language helps others learn (cross-lingual transfer), then uses these insights to predict the best mix of languages for training data.

Zhoujun Cheng, Yutao Xie, Yuxiao Qu et al.

When doing RL training on LLMs, increase parallel rollouts per problem as your compute budget grows, but expect diminishing returns; this single principle helps you allocate compute efficiently across sampling and training.

This paper studies how to optimally distribute computing resources when training language models with reinforcement learning. The researchers found that the number of parallel attempts per problem should increase with total compute budget before leveling off, and this pattern holds whether problems are easy or hard—though for different reasons.

DeepSeek-AI, Daya Guo, Dejian Yang

RL alone can teach LLMs to reason step-by-step — DeepSeek-R1 matched o1 performance at a fraction of the training cost and proved that reasoning can be distilled into small models.

Demonstrates that pure reinforcement learning without supervised fine-tuning can produce strong reasoning capabilities. DeepSeek-R1 matches OpenAI o1 on math and coding benchmarks, and distilled versions (1.5B-70B) retain most reasoning ability.

DeepSeek-AI, Aixin Liu, Bei Feng

Frontier-class performance for $5.6M in compute — DeepSeek-V3 demonstrated that engineering efficiency can close the gap with models costing orders of magnitude more to train.

Presents DeepSeek-V3, a 671B MoE model (37B active) trained on 14.8T tokens for $5.6M in compute. Achieves GPT-4o and Claude 3.5 Sonnet class performance while being fully open-weight, using FP8 training and multi-token prediction.

Charlie Snell, Jaehoon Lee, Kelvin Xu

Inference-time compute scaling is a viable alternative to model scaling — this paper provided the theoretical foundation for reasoning models like o1 and DeepSeek-R1.

Demonstrates that spending more compute at inference time (via search, verification, or repeated sampling) can be more efficient than training larger models. A smaller model with optimal test-time compute can match a 14x larger model.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey

Open-weight models reached GPT-4 class performance — Llama 3 405B closed the gap between open and proprietary models, making frontier capabilities freely available.

Introduces Llama 3, a family of models up to 405B parameters trained on 15T+ tokens. Includes multimodal extensions for image, video, and speech. The 405B model approaches GPT-4 class performance while being fully open-weight.

An Yang, Baosong Yang, Binyuan Hui

Alibaba produced a globally competitive open model family — Qwen2 demonstrated that Chinese AI labs can match Western frontier models across languages.

Describes Qwen2, a family of models from 0.5B to 72B trained on 7T tokens across 27 languages. The 72B model matches Llama 3 70B while excelling on multilingual and code tasks.

DeepSeek-AI, Aixin Liu, Bei Feng

MLA and fine-grained MoE routing enable frontier-class models at a fraction of the compute cost — DeepSeek-V2 pioneered the architecture that made DeepSeek-V3 and R1 possible.

Introduces Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, achieving strong performance while reducing inference costs dramatically. A 236B model with only 21B active parameters per token.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux

Sparse MoE at practical scale works — Mixtral showed that routing tokens to a subset of experts gives large-model quality at small-model inference cost.

Introduces Mixtral 8x7B, a sparse MoE model that routes each token to 2 of 8 expert networks. Matches or outperforms Llama 2 70B and GPT-3.5 while using only 13B active parameters per token.

Gemini Team, Rohan Anil, Sebastian Borgeaud

Native multimodal training from the ground up outperforms bolting vision onto language models — Gemini established Google as a direct competitor to GPT-4.

Introduces the Gemini family (Ultra, Pro, Nano), natively multimodal models trained jointly on text, images, audio, and video. Gemini Ultra is the first model to exceed human-expert performance on MMLU.

Hugo Touvron, Louis Martin, Kevin Stone

The first commercially usable open-weight model competitive with ChatGPT — Llama 2 established the template for responsible open-source release of aligned LLMs.

Releases Llama 2, a family of 7B-70B models with a commercially permissive license. Includes Llama 2-Chat, fine-tuned with RLHF, which approaches GPT-3.5 on human evaluations. Details the safety alignment process.

OpenAI, Josh Achiam, Steven Adler

GPT-4 reached human-expert level on professional exams and reasoning tasks — a capability threshold that shifted the conversation from whether LLMs can reason to how well.

Describes GPT-4, a large multimodal model accepting image and text inputs. Exhibits human-level performance on professional exams (bar exam 90th percentile) and substantially improves over GPT-3.5 on reasoning, factuality, and safety.

Hugo Touvron, Thibaut Lavril, Gautier Izacard

You do not need the largest model to get the best results — LLaMA proved that smaller models trained on more tokens outperform larger undertrained ones, and ignited the open-source LLM ecosystem.

Introduces LLaMA, a family of foundation models from 7B to 65B parameters trained on publicly available data. LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, demonstrating that smaller models trained on more data can match much larger ones.

Alec Radford, Jong Wook Kim, Tao Xu

Scaling weak supervision on diverse audio data produces speech recognition that works reliably across languages and conditions — Whisper became the default open-source ASR model.

Introduces Whisper, a speech recognition model trained on 680K hours of weakly supervised web audio. Achieves robust performance across languages, accents, and noise conditions without fine-tuning.

Susan Zhang, Stephen Roller, Naman Goyal

Meta released a full GPT-3-class model openly — OPT was one of the first major efforts to democratize access to large language model research.

Releases OPT-175B, a suite of decoder-only transformers from 125M to 175B parameters, with full model weights and training code. Aims to enable reproducible research on large language models by matching GPT-3 performance openly.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin

Scale produces emergent abilities — PaLM showed that certain capabilities like multi-step reasoning appear suddenly at sufficient scale rather than improving gradually.

Presents PaLM, a 540B parameter dense transformer trained on Google Pathways infrastructure across 6144 TPUs. Demonstrates breakthrough performance on reasoning tasks and shows that scaling continues to yield discontinuous capability improvements.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch

Previous scaling laws overemphasized model size — Chinchilla proved that training tokens should scale proportionally with parameters, reshaping how all subsequent models were trained.

Demonstrates that most large language models are significantly undertrained. For a given compute budget, the optimal strategy allocates roughly equal scaling to model size and training data. Chinchilla (70B, 1.4T tokens) outperforms Gopher (280B, 300B tokens).

Jason Wei, Maarten Bosma, Vincent Y. Zhao

Instruction tuning unlocks zero-shot generalization — training on diverse tasks with natural language instructions teaches models to follow novel instructions.

Demonstrates that instruction-tuning a language model on many tasks described via natural language instructions substantially improves zero-shot performance on unseen tasks. FLAN outperforms GPT-3 on many benchmarks despite being 8x smaller.

William Fedus, Barret Zoph, Noam Shazeer

Sparse MoE with single-expert routing enables trillion-parameter models at practical compute costs — the architectural pattern behind modern MoE models like Mixtral.

Simplifies Mixture-of-Experts routing by using a single expert per token instead of top-k. This Switch routing enables scaling to trillion-parameter models while keeping computational cost manageable.

Tom B. Brown, Benjamin Mann, Nick Ryder

In-context learning emerges at scale — GPT-3 showed that sufficiently large language models can perform tasks from just a few examples in the prompt, without training.

Introduces GPT-3, a 175B parameter autoregressive language model that achieves strong few-shot performance on many NLP benchmarks by conditioning on a few examples in context, without any gradient updates or fine-tuning.

Jared Kaplan, Sam McCandlish, Tom Henighan

Language model performance is predictably governed by power laws in parameters, data, and compute — enabling rational decisions about how to scale models.

Discovers that language model performance follows predictable power-law relationships with model size, dataset size, and compute budget. These scaling laws hold across seven orders of magnitude and enable principled allocation of training resources.

Colin Raffel, Noam Shazeer, Adam Roberts

Casting all NLP tasks as text-to-text unifies the approach to transfer learning and enables systematic comparison of design choices at scale.

Proposes T5, which frames every NLP task as a text-to-text problem. Systematically studies pre-training objectives, architectures, data, and scale, establishing best practices for transfer learning with transformers.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri

Efficient model parallelism enables training transformers at scales previously impractical — the engineering foundation for training all subsequent large language models.

Presents efficient model parallelism techniques for training transformer models up to 8.3B parameters. Introduces intra-layer parallelism that splits attention heads and MLP layers across GPUs with minimal communication overhead.

Alec Radford, Jeffrey Wu, Rewon Child

Scaling up language models unlocks zero-shot task performance — GPT-2 showed that bigger models learn general capabilities, not just better next-token prediction.

Introduces GPT-2, a 1.5B parameter language model that performs tasks zero-shot by simply conditioning on natural language descriptions. Demonstrates that language models implicitly learn to perform tasks without explicit supervision as they scale.

Ashish Vaswani, Noam Shazeer, Niki Parmar

Self-attention alone is sufficient for sequence modeling — the Transformer architecture became the foundation for virtually all modern language models.

Introduces the Transformer architecture, replacing recurrence and convolutions entirely with self-attention mechanisms. The model achieves state-of-the-art translation quality while being significantly more parallelizable and faster to train.