ThinkLLM
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
ModelsCapabilitiesUse CasesBenchmarksPapersGlossary
AboutPrivacyTermsRSS

ThinkLLM

Spot an error in our data? Let us know.

Papers

Recent AI research papers with accessible summaries. Updated daily from arXiv, summarized for developers who don't read papers regularly.

2455 papers8 this month12 topics
AllEvaluation 43Agents 42Applications 39Efficiency 29Reasoning 29Training 27Safety 23Multimodal 18Architecture 9Alignment 8Data 4scaling 2

Nov 16 – Nov 22(1)

Bio-PM-Guided Synthetic IMU Generation

Nov 19, 2026

Deepali Singh, Deepesh Parpani, Aviral Gupta

Synthetic data generated in a pretrained encoder's embedding space can effectively augment real IMU data for activity recognition, especially for hard-to-distinguish activities, without requiring raw signal synthesis.

This paper generates synthetic IMU (motion sensor) data in the representation space of a pretrained biosignal encoder to improve human activity recognition across different people. Rather than creating raw sensor signals, the authors train decoders (autoregressive and diffusion-based) to generate token sequences in the encoder's space, then use these synthetic tokens to augment training data.

dataefficiencyapplications

Nov 9 – Nov 15(2)

Agentic induction of common data elements and information extraction from sarcoma case report abstracts

Nov 13, 2026

Daniel Reichenpfader, Kwaku Appiagyei, Manovah Nadarajah et al.

LLM agents can efficiently draft data schemas from unstructured medical text and extract structured information at scale, but human expert validation remains essential for ensuring accuracy and clinical relevance.

Researchers used AI agents to automatically extract and organize medical data elements from thousands of cancer research abstracts. Starting with nearly 73,000 candidate data fields, they refined them to 53 validated fields and successfully extracted information from 79% of abstracts, with human reviewers confirming 83% were relevant.

applicationsdata

AgentSSL: Can MLE Agents Leverage Unlabeled Data?

Nov 12, 2026

Ethan Lin, Akanksha Sarkar, Ziang Liu et al.

AI agents can automatically synthesize effective semi-supervised learning pipelines, but the bottleneck isn't code generation—it's reliably selecting which generated programs actually work best.

This paper studies whether AI agents can automatically design semi-supervised learning (SSL) pipelines when labeled data is scarce. The authors show agents can discover code combining SSL techniques that outperform human baselines by up to 21.8%, but reveal a critical weakness: selecting which generated programs to use drops performance by 47%.

Nov 2 – Nov 8(5)

SWHForest: A Structural Curation and Search Layer for Large-Scale Code Corpora

Nov 8, 2026

Federico Pennino, Andrea Gurioli, Maurizio Gabbrielli et al.

For code model pre-training, you can now query structural properties (AST patterns, definitions, etc.) across massive corpora in seconds without downloading everything—enabling faster, more targeted dataset curation.

SWHForest is a framework that creates a searchable structural index of code files without storing source code, letting researchers efficiently filter and curate large code corpora before downloading. Instead of downloading millions of files to find what they need, users can query the index to identify matching files first, reducing downloads by 46-72%.

datatrainingapplications

XStack: A Benchmark for ML Repo Migration

Nov 5, 2026

Linrong Cai, Taiming Lu, Haozhe Chen et al.

Migrating ML code between frameworks and hardware is hard enough that it serves as a meaningful benchmark for AI agents, and today's best models still fail on a quarter of real-world migration tasks.

XStack is a benchmark that tests AI agents' ability to migrate machine learning repositories from PyTorch (NVIDIA GPUs) to JAX (Google TPUs). Researchers created a grading system based on published results to evaluate whether migrated code works correctly.

Oct 26 – Nov 1(7)

Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses

Oct 29, 2026

Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi et al.

Language models can sound confident about temporal patterns in data without doing proper statistical testing—this benchmark reveals they fail mainly at selecting relevant evidence, not at running tests, and a structured retrieval-then-test approach significantly improves accuracy.

This paper introduces STRATA, a benchmark for evaluating language models on temporal clinical hypothesis testing. Rather than just grading final answers, it evaluates every step: choosing which patient records matter, selecting the right statistical test, and interpreting results.

evaluationreasoningdata

IndicIPR-QA: A Benchmark for Extractive Question Answering on Indian IPR Laws

Oct 29, 2026

M. Mohaiminul Islam, Muhammad Abulaish

Building legal QA systems for low-resource languages requires domain-specific benchmarks and models—simple retrieval methods fail dramatically (20% F1) compared to fine-tuned legal BERT (85% F1) on Indian IPR law questions.

This paper introduces IndicIPR-QA, a benchmark dataset of 4,462 question-answer pairs for extractive QA on Indian intellectual property law across seven statutory acts. The dataset was validated through automatic checks, LLM evaluation, and human review, with experiments showing that fine-tuned legal BERT models significantly outperform simple keyword matching on this task.

Oct 19 – Oct 25(6)

Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox

Oct 23, 2026

Federico Lucca

Before deploying AI systems, measure ground truth carefully and optimize for the actual business metric, not just accuracy—a perfectly accurate classifier can fail catastrophically if it ignores rare but important cases.

This paper evaluates email routing systems for a personal multilingual inbox using rigorous measurement before training.

evaluationapplicationsdata

SpectralCanary: Embarrassingly Simple Low-Frequency Traces for T2I Dataset-Use Auditing

Oct 22, 2026

Sy-Tuyen Ho, Anirudh Satheesh, Soham Chawathe et al.

Low-frequency visual signals embedded in training images can reliably detect if a T2I model was fine-tuned on that dataset, enabling practical dataset-use auditing without degrading image quality or requiring complex per-image optimization.

SpectralCanary is a method for auditing whether text-to-image models were trained on specific datasets by embedding a hidden signal into images before release. It uses a smooth low-frequency mask to blend a canary image into training data, then checks if that signal appears in suspect model outputs. The approach achieves 91-92% detection rates while keeping images visually unchanged.

Oct 12 – Oct 18(2)

CoDA: Co-Evolution of Data Engines and Action Policies

Oct 15, 2026

Tianjiao Yu, Zhilin Ou, Yifan Shen et al.

Co-evolving data generation and policy training through shared failure diagnosis and staged learning signals can dramatically improve robot learning efficiency and robustness to real-world variations.

CoDA is a framework where a robot learning system and a data generation engine improve together by sharing a memory of failures and solutions. A vision-language critic diagnoses why policies fail, and this feedback guides both the creation of challenging training environments and the design of better learning signals.

trainingagentsdata

Private Retrieval Augmented Generation via Random Projection

Oct 15, 2026

Dixi Yao, Tahseen Rabbani, Tian Li

Random projection is a simple, practical alternative to differential privacy for protecting RAG datastores: it maintains retrieval quality while defending against data extraction attacks at lower computational cost.

This paper proposes using random projection to protect sensitive data in RAG systems. By applying the same random transformation to both stored knowledge embeddings and user queries, the method preserves search accuracy while preventing attackers from extracting private information from the datastore—without the computational overhead of formal privacy guarantees.

Oct 5 – Oct 11(6)

Designing Protein Structures from Natural Language

Oct 9, 2026

Joseph Lee, Yanjing Li, Bo Qiang et al.

Natural language is now a practical interface for protein design—you can describe desired protein properties in text and get back 3D structures, with experimental validation showing it works for real-world targets.

Prompteina is an AI model that generates protein structures from natural language descriptions, similar to how text-to-image models work. Trained on 1.8M protein structures with detailed captions, it can design proteins with specific structural and functional properties without needing separate input formats for each goal.

applicationsmultimodaldata

Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

Oct 9, 2026

Oktie Hassanzadeh, Dharmashankar Subramanian

LLM-based semantic query engines are non-deterministic even at temperature 0—benchmark comparisons based on single runs can be misleading, and you need multiple executions or voting strategies to get reliable results.

This paper reveals that semantic query engines using LLMs produce inconsistent results across multiple runs, even at temperature 0. By re-executing 1.2 million model calls across three benchmarks, the authors show that 0.07-10.2% of individual rows change their output between runs, and up to 66% of final answers differ.

Sep 28 – Oct 4(6)

LESSER: Post-Training Data Selection with Output-Layer Gradients

Oct 2, 2026

Lyuxin David Zhang, Eric Wong, Surbhi Goel et al.

You can select effective training data for LLMs using only output-layer gradients from forward passes, cutting computation costs by ~10× compared to full-gradient methods without sacrificing downstream performance.

This paper proposes LESSER, a method for selecting high-quality training data for large language models by using only output-layer gradients instead of full-parameter gradients. By leveraging cheaper forward passes rather than expensive backward passes, LESSER reduces computational cost by 9.7× for supervised fine-tuning while maintaining performance comparable to full-gradient selection methods.

trainingefficiencydata

Higher-Order Molecular Grammars for Generative and Foundation Models in Chemistry

Oct 1, 2026

Yiming Huang, Yujie Zeng, Vijay Prakash Dwivedi et al.

HGR enables sequence models to generate chemically valid molecules with perfect validity while capturing complex molecular topology, achieving top performance on generation and property prediction tasks without the computational cost of explicit higher-order encodings.

This paper introduces Higher-order Grammar Representation (HGR), a new way to represent molecules that captures complex structural features like ring systems by converting them into sequences of grammar rules. Unlike existing methods that struggle with computational overhead, HGR makes these structures compatible with standard sequence models while guaranteeing valid molecules.

Sep 21 – Sep 27(4)

A Living Benchmark for Information Retrieval from Electronic Health Records

Sep 24, 2026

Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani et al.

Automated benchmark generation validated by domain experts enables continuous evaluation of clinical AI systems, exposing real gaps (like multi-document synthesis) that static benchmarks miss.

Researchers created BRIE, an automatically-generated benchmark for testing how well AI systems retrieve and synthesize information from patient medical records.

evaluationapplicationsdata

ARGUS: Role-Aware Event Knowledge Graphs for U.S. Employment-Discrimination Complaints

Sep 24, 2026

Sriram Kannan, Swetha Saseendran, Vishnu Vardhan Reddy Kandi et al.

Structured event graphs from legal documents improve reasoning tasks like classification and QA, but only when you already have the right documents—they don't help with initial retrieval.

ARGUS builds structured Event Knowledge Graphs from employment-discrimination legal complaints using a 5W1H schema and LLMs. The system extracts facts, organizes events with temporal and causal relationships, and merges them into document-level graphs. Testing shows these graphs improve claim classification and help answer legal questions when relevant documents are already retrieved.

Sep 14 – Sep 20(5)

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

Sep 18, 2026

Rawan El Ghali, Umm Kulsoom, Anas Madkoor et al.

Multiple-choice scoring masks AI model failures in language understanding—QuranicMMLU shows models score 24 percentage points higher on multiple-choice than open-ended tasks, meaning benchmark design significantly impacts what we learn about model capabilities.

QuranicMMLU is a benchmark for testing how well AI models understand Quranic Arabic across five linguistic areas: sound, word structure, grammar, meaning, and context.

evaluationdata

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

Sep 17, 2026

Sho Kawano, Zehang Richard Li, Paul A. Parker

When evaluating AI systems across domains with limited labels, borrowing information across domains through prediction-powered smoothing gives more accurate performance estimates than evaluating each domain independently.

This paper addresses how to accurately evaluate AI systems across different domains (like task types or conversation types) when you can only label a small sample. It proposes prediction-powered smoothing, which combines AI predictions with limited labels to get better estimates for each domain, and a validation method to choose between different estimation approaches.

Sep 7 – Sep 13(6)

Generative Marketing Mix Modeling: A Causal Inference Framework Linking GEO and GEM to Business Impact

Sep 10, 2026

Masahiro Kato, Daiki Honma, Taka Kato

Businesses can now quantify the ROI of appearing in generative AI outputs by combining visibility data with causal inference, enabling marketing decisions in an AI-driven world where traditional metrics don't apply.

This paper introduces a framework to measure how generative AI systems (like ChatGPT) affect business outcomes when companies optimize their visibility in AI-generated answers.

applicationsevaluationdata

CausalArena: Benchmarking Causal Discovery in the Foundation Model Era

Sep 10, 2026

Zi-Rong Li, Si-Yang Liu, Tian-Zuo Wang et al.

No single causal discovery method generalizes across different benchmark types; evaluation results depend heavily on which test scenario you use, making diverse benchmarks essential for fair comparison.

CausalArena is a comprehensive benchmark for evaluating causal discovery methods—tools that learn cause-and-effect relationships from data. It addresses a critical problem: existing benchmarks use different test scenarios, making it hard to compare methods fairly.

Aug 31 – Sep 6(12)

A Deep Generative Model for Synthesizing Labeled Wireless Signals

Sep 4, 2026

Yuxiao Li, Keke Hu, Santiago Mazuelas et al.

Synthetic wireless signals generated by GANs can effectively replace expensive real-world data collection for training wireless sensing models, reducing labeling costs while maintaining physical realism.

This paper introduces IIns-GAN, a deep learning method that generates realistic labeled wireless signals for training machine learning models.

datatrainingapplications

Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe

Sep 4, 2026

Dain Kim, Eungi Cho, Kyumin Kim et al.

Open-source models can match much larger models at multi-step API calling by training on execution-verified data—showing that smart data synthesis matters more than model size for tool-use tasks.

This paper addresses a real-world problem: open-source AI models struggle when they need to call multiple government APIs in sequence to complete tasks. The authors create KOPA-Bench, a benchmark of 145 real Korean government API tasks, and introduce EDGE, a method that learns which API outputs can feed into other APIs by actually testing them live.

Aug 24 – Aug 30(4)

SWE-Prime: Fewer Trajectories, Better Performance

Aug 27, 2026

Dewu Zheng, Ruizhe Ye, Yanlin Wang et al.

Filtering training data for code-fixing tasks at both trajectory and step levels beats training on all successful examples—quality and relevance of supervision matter more than quantity.

SWE-Prime improves how AI models learn to fix software bugs by being smarter about which training examples to use. Instead of training on all successful bug fixes, it filters trajectories (problem-solving paths) at two levels—first selecting high-quality complete solutions, then identifying which individual steps within those solutions are actually worth learning from.

trainingdataapplications

RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

Aug 27, 2026

Maayan Sharon, Tom Hope

Retrieval systems need to understand different types of scientific inspiration—not just finding similar papers, but finding papers that generalize ideas, concretize them, or solve stated problems.

RATIO is a benchmark for retrieving scientific papers that can inspire research at different levels of abstraction. It defines three types of retrieval: finding approaches to solve problems, generalizing to broader concepts, or narrowing down to concrete implementations.

Aug 17 – Aug 23(7)

Inducing Task Models from Computer-Use Traces

Aug 20, 2026

Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen et al.

By converting raw computer activity into structured task models with goal hierarchies and control flow, TMI enables AI agents to learn realistic work procedures and organizations to audit and reuse task knowledge from employee activity traces.

This paper presents Task Model Induction (TMI), a method that automatically discovers and structures how people actually work on computers by analyzing screenshots and input logs.

agentsdataapplications

MidTool: Mid-training Data Synthesis for Agentic Tool Use

Aug 20, 2026

Fengqing Jiang, Yite Wang, Boyi Liu et al.

Tool-use capabilities in language models improve significantly when trained during mid-training with targeted synthetic data, rather than waiting until post-training—similar to how math and reasoning skills benefit from dedicated training phases.

MidTool is a data synthesis pipeline that creates training data for teaching language models to use tools effectively during mid-training (the stage between pretraining and fine-tuning).

training

Aug 10 – Aug 16(10)

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Aug 14, 2026

Zhelun Wu

When combining evidence from multiple sources, separate the task of interpreting each source from aggregating those interpretations—use structured tuples and calibrated scoring rather than simple concatenation and vote counting.

This paper separates evidence interpretation from decision aggregation in multi-source reasoning systems. Instead of concatenating sources into one prompt, the authors propose a structured evidence tuple (hypothesis, reliability, rationale, provenance) and show how to properly combine interpretations using calibrated log-likelihood ratios.

reasoningevaluationdata

Generating Benchmark Health Data Using a Tabular Diffusion Transformer

Aug 14, 2026

Hao Yan, Lisa Pilgram, Dan Liu et al.

You can now generate realistic synthetic tabular health data from multiple heterogeneous sources by standardizing them statistically first, then using diffusion transformers to learn and reproduce their patterns.

This paper presents a method for generating synthetic health data from multiple different database tables with varying structures. It works in two stages: first converting diverse tables into a standardized statistical format, then using a diffusion transformer to learn patterns and generate new synthetic tables.

Aug 3 – Aug 9(6)

Tytan: Interactive Neurosymbolic Construction of Analytic Semantic Schemas from Relational Data

Aug 6, 2026

Donna Hooshmand, Shubham Shahi, Cameron Barrie et al.

Automating semantic schema construction lets non-technical users query databases without expert help, and TYTAN's hybrid symbolic-LLM approach achieves production-ready accuracy by knowing when to ask humans for clarification.

TYTAN automatically builds semantic schemas for relational databases by combining symbolic analysis with LLM inference to identify entities, relationships, and data roles. It asks targeted questions when ambiguous, achieving 100% coverage and correctness on real-world databases—eliminating the manual work that currently bottlenecks data analysis tools.

dataapplicationsreasoning

RxnCLF: Contrastive Transformation-Aware Reaction Foundation Model for Improved Reactivity Prediction

Aug 6, 2026

Yiting Zheng, Cheng Fang, Anthony Donofrio et al.

Using a unified graph representation of reactions instead of separate reactant/product graphs enables better learning of chemical transformations, leading to more accurate yield predictions and a foundation model that could generalize across diverse reaction types.

RxnCLF is a self-supervised learning framework that represents chemical reactions as unified graphs (condensed reaction graphs) to better capture how molecules transform.

Jul 27 – Aug 2(9)

AskChem: Claim-Centered Infrastructure for Chemistry Literature Synthesis

Jul 30, 2026

Bing Yan, Gregory Wolfe, Stefano Martiniani et al.

Instead of searching papers, search claims: AskChem lets you find specific findings with built-in source verification, reducing manual work for literature synthesis and improving AI agent reliability by 12% on citation accuracy.

AskChem is a search infrastructure that converts chemistry papers into atomic, provenance-tracked claims rather than returning full documents. Scientists and AI agents can search across 2.4M claims from 147K papers, with each claim linked to its source DOI and exact evidence location, enabling faster synthesis of findings across multiple papers.

applicationsdata

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

Jul 30, 2026

Haomin Qi, Xingliang Wang, Xuanqi Gao et al.

By mining repository history and reconstructing code states, you can generate verified, executable coding tasks at scale—reducing the manual effort of creating training data for code-generation agents while maintaining realistic development scenarios.

Change2Task automatically converts pull requests from repository history into executable coding tasks for training AI agents.

Jul 20 – Jul 26(2)

Complexity Bounds and Approaches to Learning Projected Gradient Descent Solver Iterates

Jul 24, 2026

Anjian Li, Ryne Beeson

Collecting intermediate solver iterations during optimization—not just final solutions—significantly increases training data for learning-based optimization without additional computational cost, with theoretical guarantees on generalization.

This paper addresses data scarcity in training models that generate initial guesses for optimization problems. The authors propose collecting intermediate solver steps (k-neighborhoods) alongside final solutions to increase training data without extra computation.

trainingdata

LKValues: Aligning Large Language Models with Sri Lankan Societal Values

Jul 22, 2026

Nethmi Muthugala, Supryadi, Surangika Ranathunga et al.

LLMs trained primarily on Western data misalign with non-Western cultural values; this work provides a replicable framework for embedding country-specific values in low-resource languages through survey-grounded datasets and targeted fine-tuning.

This paper introduces LKValues, a resource suite for aligning large language models with Sri Lankan cultural values. The authors surveyed 205 Sri Lankans to identify 40 key societal values, created a 150k-instance instruction dataset in Sinhala and English, and built an evaluation benchmark.

agentstrainingdata
evaluationapplicationsdata

MedGraph: Maintaining Patient State with Graph Memory for Long-Term Medical Dialogue

Nov 5, 2026

Donghua Zhang, Zizheng Wang, Ze Dong et al.

For medical AI systems handling long-term patient care, structured graph memory that tracks temporal changes and explicitly maintains safety facts outperforms generic retrieval methods by a significant margin.

MedGraph is a graph-based memory system for medical chatbots that track patients across multiple visits. It solves three key problems: outdated information, mixing up different patients' facts, and missing safety-critical details.

applicationsdatasafety

MOMASPLAT: Dynamic Novel-View Demonstration Augmentation for Mobile Manipulation

Nov 5, 2026

Sateesh Kumar, Arnav Balaji, Arpit Bahety et al.

By reconstructing 3D scenes and manipulated objects separately, then re-rendering demonstrations from novel robot base poses, MOMASPLAT makes visuomotor policies robust to viewpoint changes—doubling success rates on unseen positions without collecting more real-world data.

MOMASPLAT augments robot manipulation demonstrations by synthesizing them from unseen camera viewpoints using 3D reconstruction and novel-view synthesis. This solves a key problem in mobile manipulation: robots rarely execute tasks from the exact same position as during training, causing policy failures.

dataapplicationsmultimodal

GUILD: Guided Iterative Data Collection for Imitation Learning

Nov 3, 2026

Shivin Dass, Kevin Nguyen, Rohan Yelandur et al.

Active data collection guided by predictive attribution can reduce the amount of real-world robot teleoperation needed by 2-3x compared to unguided collection, because it focuses on collecting data that actually improves policy performance rather than just semantically relevant data.

GUILD is a method for collecting better training data for robot learning by actively deciding what to collect rather than passively gathering data. It uses vision-language models to suggest relevant tasks and environments, then evaluates which collected data actually helps the robot policy perform better, creating a feedback loop that guides future data collection toward more useful examples.

datatrainingapplications
evaluationdataapplications

How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates

Oct 29, 2026

Max Neuwinger, Hanna Yukhymenko, Imanol Schlag

Safety alignment evaluated only in English can hide serious vulnerabilities in other languages—culturally localized attacks often succeed where English ones fail, and template-based data augmentation is an effective way to scale safety training across low-resource languages.

This paper addresses the gap in multilingual safety training for large language models by creating MSafeTemp, a dataset of 60 languages built using template-based augmentation. The authors extract safe patterns from harmful prompts, translate them, and fill slots with culturally authentic content to generate diverse refusal and benign examples.

safetydata

FIRME: Registry-Anchored Memory Graphs for Evolving, Heterogeneous Investor Knowledge

Oct 29, 2026

Yunbai Zhang, Yunyi Gao, Han-Tai Shiao

For AI systems that need to remember and reuse knowledge over time, maintaining a canonical registry with persistent aliases and source tracking prevents fragmented relationship histories and lets evidence accumulate reliably.

FIRME is a memory system for financial AI that tracks investor knowledge across time and sources by maintaining a canonical registry of relationships. It reconciles different ways of describing the same observation (e.g., "Apple stock" vs "AAPL") and links investor discussions to cash-flow data, while preserving where each piece of information came from and when.

dataapplications

Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

Oct 27, 2026

Andrew Szot, Samira Abnar, Omar Attia et al.

When training tool-use agents, prioritize diversity of tools and data sources over raw task quantity—this scales better and produces more capable agents than previous synthetic training approaches.

This paper studies how to train better LLM agents that use tools to complete multi-step tasks. The authors created SynthAPI, a system that generates synthetic training environments with controllable diversity across tools, tasks, and databases.

trainingdatascaling

What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

Oct 27, 2026

Chang Su, Bin Wu, Emine Yilmaz et al.

Knowledge that appears inconsistently across web snapshots is learned faster during later pretraining stages and achieves better performance, suggesting that temporal variability in training data is a useful signal for understanding and predicting learning dynamics.

This paper investigates how language models learn different types of knowledge during pretraining by analyzing models trained on web data from different years.

trainingevaluationdata

Pneuma: A Large-Scale Real-World Time Series Benchmark for Commercial Aircraft Predictive Maintenance

Oct 27, 2026

Jiafeng Lin, Yutian Zhang, Huan Zhang et al.

Real-world predictive maintenance is harder than lab benchmarks suggest—even strong models fail on subtle faults in actual aircraft data, and the dataset's class imbalance and cross-aircraft variance are critical challenges developers must address.

Pneuma is the first public large-scale benchmark dataset for aircraft predictive maintenance, containing over 920 million time series measurements from real commercial aircraft.

evaluationdataapplications
safetyevaluationdata

Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

Oct 22, 2026

Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan

Agentic auditing can efficiently find corpus-specific data quality issues by treating an LLM as a debugger that compiles findings into reusable tests, costing $2-3 per run versus $12k-90k for full-corpus LLM classification.

Researchers used an LLM-powered agent to audit the Slovak portion of FineWeb2, a multilingual web corpus. Instead of classifying documents, the agent explored the corpus through an API, discovered data quality issues, and compiled them into executable tests.

dataevaluationagents

How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

Oct 22, 2026

Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias et al.

Retrieval evaluation scores are highly sensitive to annotation completeness—expanding ground-truth evidence annotations improved reported metrics without changing what the system actually retrieved, suggesting evaluation protocols must account for annotation coverage to avoid misleading perform...

This paper evaluates a retrieval system for a clinical AI tool that finds relevant medical evidence about breast lesions. The researchers tested different ways to chunk documents and rank results, finding that a hybrid approach combining multiple retrieval methods achieved strong performance.

evaluationapplicationsdata

Contestable Agents for Data Governance: Recording Human Stewardship of AI-Generated Metadata Across Structured and Unstructured Systems

Oct 22, 2026

Humza Akhtar, Brian Freeman

When AI assists with data governance, require agents to show their reasoning with citations, let humans review before committing changes, and record what humans actually verified—this prevents errors from propagating while capturing institutional knowledge.

This paper presents a system where AI agents propose metadata decisions for industrial data (like equipment classifications) with cited evidence, but humans review and approve each decision before it's recorded. The system keeps a full audit trail of what was checked, changed, and why—preventing AI errors from silently spreading through business systems while documenting human expertise.

agentsdatasafety

Benchmarking Arabic Dialects for Math Problems

Oct 20, 2026

Noor Abo Mokh, Teresa Lynn, Nizar Habash et al.

Math reasoning in low-resource languages isn't just about translation—cultural localization of problems changes how models solve them internally, even when overall accuracy stays similar.

This paper introduces DAR-MGSM, the first benchmark for evaluating math reasoning in Dialectal Arabic across five major dialects plus Modern Standard Arabic. The benchmark includes both translated and culturally localized versions of math word problems, revealing that cultural adaptation significantly impacts model performance and internal reasoning patterns.

evaluationdata
safetyefficiencydata
evaluationdata

TabMF: Learning Dataset Representations with Tabular Foundation Models

Oct 9, 2026

Rafiq Kamel, Nicolas Atienza, Aziz Fouché et al.

Pretrained tabular models already capture dataset-level structure useful for representation learning; adding a simple readout layer and contrastive training makes them effective dataset encoders without task-specific training.

TabMF repurposes pretrained tabular foundation models as dataset encoders by adding a dataset-level readout and contrastive post-training. This enables creating compact numerical representations of entire datasets for tasks like dataset retrieval and monitoring distribution shifts, outperforming hand-crafted features and other learned approaches.

dataapplications

Long Text to Predictive Features: LLM-Guided Blockwise Feature Engineering via Executable Program Search

Oct 8, 2026

Ziming Dai, Dabiao Ma, Ziheng Guo et al.

You can use LLMs to design feature extraction code offline, then deploy the resulting programs without needing LLM calls at inference time—achieving better predictions while meeting real-time deployment constraints.

This paper presents LLM-BlockFE, a system that automatically converts long text into reusable feature programs for risk prediction models. Instead of calling an LLM for every prediction, the system uses an LLM offline to design feature extraction code once, then deploys frozen programs for fast inference.

dataefficiencyapplications

Base Models Can Reason By Taking a Cue From Training Data

Oct 5, 2026

Sophie L. Wang, Amil Dravid, Rulin Shao et al.

Base models already contain reasoning capabilities encoded in their training data—you can unlock them by conditioning on the right token cues, without needing expensive RL fine-tuning.

This paper shows that base language models can achieve reasoning performance comparable to RL-trained models by using specific starting tokens (like "Okay" or "Alright") that trigger learned associations from training data. The authors demonstrate they can create new reasoning cues through data interventions and trace these effects back to specific document types in the training set.

trainingreasoningdata

PlotGround: Grounding Plot Digitization in Real Scientific Figures and Their Source Data

Oct 5, 2026

Yaohui Zhang, Binxu Li, Haoyi Duan et al.

Current multimodal models can approximate values from scientific figures but struggle with precision; providing source data instead of figures dramatically improves accuracy (90% to 97.4%) while reducing computational cost, suggesting a practical path for scientific data extraction.

PlotGround is a benchmark for evaluating how well AI models can extract numerical values from scientific figures.

evaluationmultimodaldata
architecturedataapplications

Finetuning with Sampling: SFT Learns Better Than You Think

Oct 1, 2026

Aayush Karan, Sitan Chen, Yilun Du

SFT isn't inherently worse than RL for posttraining—the gap comes from data distribution mismatch. By reshaping data to be more on-policy before training, SFT can generalize better and forget less than strong RL baselines.

This paper shows that supervised finetuning (SFT) can match or beat reinforcement learning for posttraining if you transform the training data first. The authors use an MCMC sampling algorithm to gradually shift off-policy expert demonstrations toward on-policy trajectories that a reference model can actually learn from.

trainingdata

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Oct 1, 2026

Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing et al.

Current frontier AI models struggle with real enterprise data work: the best model scores 95+ on only 35% of tasks, revealing a major gap between text-to-SQL benchmarks and actual data agent capabilities needed for production systems.

Argo-Bench is an evaluation framework with 210 realistic data science tasks that test AI agents on enterprise-scale workflows.

evaluationagentsdata

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes

Oct 1, 2026

Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc et al.

Spatially grounded self-distillation with synthetic data can teach multimodal models better visual reasoning that transfers to real-world tasks, without requiring human annotations or external teachers.

This paper improves multimodal AI models by having them learn from a smarter version of themselves that receives spatial hints about where to look in images. Using synthetic scenes with automatic object labels, the approach trains models to understand spatial relationships without human annotation, and surprisingly, these improvements transfer to real-world vision tasks.

trainingmultimodaldata

Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)

Oct 1, 2026

Zilin Du, Bowen Yang, Boyang Albert Li

When scaling data selection with neural networks, the standard loss function causes poor generalization—a new loss function (PVM) that matches predicted values pointwise solves this and transfers better across datasets and model scales.

This paper addresses data selection for training large language models by proposing TESS, a framework that uses a neural network to score and select training examples. Unlike existing meta-learning approaches that assign per-sample weights, TESS uses a novel loss function (Pointwise Value Matching) that avoids optimization instability and improves generalization to new datasets and model sizes.

trainingdataefficiency
dataapplicationsreasoning

EquivSVA: A Formally Verified Dataset of Behavioral Assertions Across Equivalent RTL Implementations

Sep 22, 2026

FNU Aditi

When evaluating LLMs that generate hardware assertions, using equivalent implementations reveals that the same behavior can produce different numbers of correct assertions—showing that assertion quality depends on implementation details, not just the intended behavior.

EquivSVA is a dataset of 120 behavior families with 480 RTL implementations and 914 formally verified assertions, designed to test whether AI-generated hardware assertions capture true behavior or just implementation details. Each behavior family has four structurally different implementations of the same functionality, enabling controlled studies of assertion robustness.

evaluationdata

onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

Sep 21, 2026

Lei Yang, Mengyin Liu, Jia Wang et al.

Token-level correction during annotation is significantly faster than full rewriting and produces on-policy training data that preserves the model's natural generation patterns while providing precise supervision signals.

onPanda is an interactive annotation tool that helps create training data for AI models by letting annotators correct responses token-by-token. Instead of rewriting entire outputs, annotators find the first mistake, fix it, and let the model regenerate from that point.

trainingdataalignment
evaluationdata

Q&A on Any Spreadsheet Requires Interpreting Its Grid Structure

Sep 17, 2026

Zofia Smoleń

Labeling spreadsheet cells by role helps, but hits a hard limit because spreadsheets are inherently 2D with infinite possible relationships. Better approaches should flatten spreadsheets into 1D text rather than trying to classify cells into fixed categories.

This paper tackles how to make spreadsheets queryable by AI systems. The key insight is that spreadsheets are fundamentally 2D structures that don't fit neatly into categories—so instead of trying to classify each cell's role, the field should develop better ways to convert spreadsheets into readable text that language models can work with.

dataevaluationapplications

PAA: The Probabilistic Allen Algebra: A Generative and Complete Probabilistic Extension of Allen's Interval Relations

Sep 17, 2026

Julian Eggert

Probabilistic Allen algebra lets you reason about uncertain temporal relations from language and data by deriving relation probabilities from distributions over interval boundaries, rather than assigning scores arbitrarily—making temporal reasoning more grounded in uncertainty.

This paper extends Allen's interval algebra—a system for reasoning about temporal relationships—to handle uncertainty. Instead of crisp yes/no answers about whether one event is "before" another, it models time points and interval boundaries as probability distributions (Gaussians), allowing expressions like "roughly during" or "just before" to have graded meanings.

reasoningdata

Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation

Sep 16, 2026

Guanhua Ji, Tianyu Li, Dayoon Suh et al.

Combining generated video with generated audio allows robots to infer not just motion but also the forces needed for contact-heavy tasks—something video alone cannot provide.

This paper shows how robots can learn contact-rich manipulation tasks by generating both video and audio together. The system uses the loudness of generated contact sounds to create force profiles that guide the robot's movements, enabling tasks like pushing and grasping that require precise force control. The approach works without task-specific training data.

agentsmultimodaldata
evaluationreasoningdata

Nuha-Speech: Building General-Purpose Arabic Speech-LLMs

Sep 10, 2026

Yingzhi Wang, Reem Alhazzani, Muhammad Alqurishi

Arabic speech AI was severely underrepresented in multilingual models—this project provides the dataset, trained models, and evaluation tools needed to build general-purpose Arabic speech systems from scratch.

Nuha-Speech addresses the lack of Arabic speech AI models by creating a comprehensive framework: a 1.5M-sample Arabic speech question-answering dataset, fine-tuned models based on Qwen-Omni, and evaluation benchmarks. This work establishes foundational infrastructure for building Arabic-capable speech AI systems despite limited available resources.

multimodaldataevaluation

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

Sep 10, 2026

Pruthwik Mishra, Rudra Trivedi, Avi Patel et al.

Contextual transformer models like MuRIL and XLM-RoBERTa can effectively identify which language each token belongs to in code-mixed text, outperforming traditional monolingual approaches.

This paper tackles language identification in code-mixed text (where people mix multiple languages in one message) by treating it as a sequence labeling task. Researchers fine-tuned transformer models designed for Indian languages on three language pairs and released benchmark datasets with manually annotated examples to help future work on this social media problem.

datatraining

Dynamic language model representations for multi-objective reaction optimisation

Sep 10, 2026

Joshua W. Sin, David Ming Segura, Bojana Ranković et al.

Language models can learn task-specific representations of chemical reactions directly from text, enabling more efficient multi-objective optimization than hand-crafted chemical descriptors—achieving high-yield synthesis conditions with under 3% of the design space explored.

Researchers developed a method to optimize chemical reactions across multiple goals (yield, selectivity, safety) by using a fine-tuned language model to learn dynamic representations of reaction components from text descriptions.

applicationsdata

Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

Sep 10, 2026

Chibuzor Okocha, Christan Earl Grant

Standard speech recognition metrics mask critical failures on code-switched speech—you need switch-aware diagnostics to understand where systems actually break down, especially for low-resource languages like Yoruba.

This paper evaluates how well modern speech recognition systems handle code-switched speech (mixing English and Yoruba). Standard metrics like word error rate hide the real problems: systems struggle much more with Yoruba than English, and fail especially at language switches.

evaluationdata
agentsdatatraining

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

Sep 4, 2026

Ankit Goyal, Jaideep Ray

Memory portability isn't automatic: compressed formats couple tightly to specific models and fail asymmetrically during upgrades, while structured knowledge graphs and raw histories are more robust. Always test migrations in both directions and keep original source data for recovery.

When you upgrade an AI agent's model, its memory often breaks in unexpected ways. This paper tests four memory storage formats—raw text, chunked retrieval, compressed notes, and structured knowledge graphs—to see which survive model swaps. Fixed schemas stay reliable, but compressed notes and retrieval systems lose accuracy unpredictably depending on which direction you upgrade.

agentsevaluationdata

LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics

Sep 4, 2026

Gaurab Baral

Standard semantic similarity metrics are unreliable for legal text simplification because they can't distinguish between preserving words and preserving legal meaning—a problem that requires new evaluation approaches beyond token overlap.

This paper exposes a critical flaw in how we measure whether simplified legal text preserves meaning. Current metrics fail because they conflate surface-level word overlap with actual legal meaning.

evaluationdata

Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views

Sep 3, 2026

Joseph Lee, Yidi Huang, Dokyoon Kim et al.

LLMs learn knowledge more effectively when exposed to multiple reformulations of the same concept rather than just repeated text, suggesting that data diversity—not just quantity—is fundamental to pre-training success.

This paper investigates how LLMs learn knowledge during pre-training by studying the role of 'auxiliary views'—different reformulations of the same knowledge. Through controlled experiments, the researchers show that diverse representations of knowledge improve learning better than simple repetition, even when the total number of tokens stays the same.

trainingdata

Last Translation Benchmark

Sep 3, 2026

Vilém Zouhar, Niyati Bafna, Mukund Choudhary et al.

Standard translation benchmarks are saturating and automatic metrics are unreliable—this benchmark uses human-authored hard cases with explicit failure rules to provide reproducible evaluation that actually identifies what models get wrong.

This paper introduces the Last Translation Benchmark, a curated collection of challenging translation examples (text, images, audio, video) designed to expose weaknesses in state-of-the-art machine translation models.

evaluationdata

Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments

Sep 3, 2026

Jie Wu, Zhenru Zhang, Beichen Zhang et al.

Instead of building environments from scratch, you can extract them from existing agent trajectories, then use those environments to generate diverse training tasks at scale—turning single frozen demonstrations into many verifiable, interactive learning scenarios.

Terminal-Universe reconstructs executable coding environments from agent trajectories by replaying file operations and synthesizing missing dependencies. It then generates new tasks from these environments—both variations of original tasks and cross-workspace queries—and extends them into multi-turn interactions with simulated user feedback.

trainingagentsdata

AI Contextual Measurement for Recovering Individual and Group-Level Effects: Validation Against Survey Measures and an Occupational Application

Sep 2, 2026

Wenxin Jiang, Xuyang Wang, Yuxiao Wu

AI can reliably measure hard-to-survey characteristics at the individual level, but only when you have rich background data available—it's most useful for filling gaps in existing datasets rather than replacing surveys entirely.

Researchers developed AICOME, a framework for validating whether AI-generated measures of occupational and social characteristics can accurately recover both individual and group-level effects in statistical models.

evaluationdata

Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics

Sep 1, 2026

Maksim Evdokimov, Matvey Ivanov, Dmitrii Tsiupin et al.

Smart data curation and cost-aware deployment economics can make small VLMs competitive with much larger models for document processing—the key is selecting training examples strategically and measuring real-world costs including verification and correction.

This paper presents a practical document understanding system that extracts structured data from documents at a fraction of the cost of human annotation or larger models.

efficiencydataapplications

Configurable Semantic Chunking for Biomedical Information Extraction in Retrieval-Augmented Generation

Aug 31, 2026

Riya Ahuja, Tim Kacprowski, Roya Shiasi Sardoabi

Semantic chunking that respects entity and relationship boundaries significantly outperforms fixed-size chunking for biomedical RAG, especially when relation cues are explicit in the text.

This paper improves biomedical information extraction in RAG systems by replacing fixed-size text chunking with a configurable semantic chunking framework. The approach preserves important entities and relationships by using trigger-centered chunking and hierarchical relation resolution, achieving 8.4 F1 points improvement on biomedical relation extraction benchmarks.

dataapplications

OntoAligner-Ensemble: Voting-Based Fusion across Heterogeneous Ontology Alignment Techniques

Aug 31, 2026

Hamed Babaei Giglou, Sören Auer, Peio Popov et al.

Combining different types of ontology alignment methods through voting-based ensemble learning reliably improves results, with the best approach depending on whether you prioritize precision (mix different paradigms) or overall F1-score (use multiple LLMs).

This paper presents OntoAligner-Ensemble, a framework that combines predictions from different ontology alignment methods (string-based, knowledge graph embeddings, and LLM-based) using voting and selection strategies. Testing across biomedical and other domains shows that mixing diverse alignment approaches consistently improves precision-recall balance and often beats individual methods.

evaluationdata

LLM Post-Training as Brownfield Maintenance: An Industrial Perspective on Dataware Engineering

Aug 31, 2026

Gopi Krishnan Rajbahadur, Amir M. Ebrahimi, Boyuan Chen et al.

Post-training in production is about engineering discipline around data mixtures and yield metrics, not algorithmic breakthroughs—small improvements in converting training data into usable supervision can compound into significant model gains.

This paper treats LLM post-training as industrial maintenance work, not research. Teams inherit a deployed model and must improve it within strict compute budgets without breaking existing capabilities.

trainingdataapplications
evaluationdataapplications

CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases

Aug 27, 2026

Sil Hamilton, Albert Yu Sun, Oscar J. Romero et al.

LLMs perform much worse on corporate Q&A tasks at realistic scale (thousands of documents) than on smaller benchmarks, and CorporateBench provides a standardized way to measure this performance gap without exposing real company data.

CorporateBench is a large-scale Q&A benchmark for testing how well language models handle real-world corporate document collections. It includes 230,000+ human-validated documents organized into synthetic companies of varying sizes, with questions requiring both information extraction and knowledge base querying.

evaluationdataapplications

RCMN: Understanding Misleadingness in Influential Public Discourse

Aug 27, 2026

Peiling Yi

Misleadingness in public discourse is diverse and often subtle (exaggeration, omission, unsupported inference), and while AI can sometimes predict how readers will interpret misleading content, identifying *how* it misleads requires rich contextual and evidential grounding.

This paper introduces RCMN, a framework for understanding how public discourse misleads readers through framing, omission, and context rather than just false claims.

evaluationdatasafety
agents
data

OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models

Aug 20, 2026

Nikita Khudov

Domain-specific benchmarks built from verified sources reveal that LLM performance gaps aren't just about model size—reasoning capabilities, self-preference bias, and access to reference materials dramatically change accuracy, with some models gaining 33+ percentage points when given context.

OenoBench is a wine-domain benchmark with 3,266 multiple-choice questions built from verified facts extracted from government registries and peer-reviewed sources.

evaluationdata

Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training

Aug 19, 2026

Zachary Speck, Asa Shepard

Individual training examples can be learned and then completely forgotten during pre-training, leaving virtually no measurable impact on final model behavior or internal structure, suggesting that what matters for final performance is the aggregate signal, not individual data points.

Researchers trained 32 GPT-2 models from scratch and injected a single training example at peak learning rate to measure its actual impact. The example was learned immediately but completely forgotten by the end of training, leaving no detectable trace in the final model's weights, geometry, or performance—despite moving the model within its loss basin during training.

trainingdata

Comment-level Topic Drift Analysis in the Reddit Corpus

Aug 19, 2026

Steven Morse, Daniel Runfola, Trenton W. Ford

Political and social topics show measurable semantic drift in online discourse over time, detectable through embedding-space analysis—a technique that could help track how language and meaning evolve around contentious issues.

This paper analyzes how topics shift and evolve in Reddit discussions over 16 years by tracking semantic embeddings of 12.7 billion comments. Using pretrained language models and unsupervised clustering, the researchers show that politically charged topics drift significantly in meaning over time, while stable domains like sports remain consistent.

dataevaluationapplications

An Analytical-Prior Framework for Data-Efficient Prediction of Sound-Reduction Frequencies in Rectangular Side-Branch Helmholtz Resonators

Aug 17, 2026

Jiaming Li

When you have a fast but imperfect analytical model and limited expensive simulation data, teaching a neural network to correct the analytical model's errors—or pre-training it on the analytical model first—can cut your data requirements dramatically.

This paper shows how to make machine learning models more data-efficient by combining cheap analytical equations with expensive high-fidelity simulations. Using Helmholtz resonators as a test case, the authors demonstrate two approaches: learning to correct analytical predictions, or distilling analytical knowledge into a neural network before fine-tuning with limited simulation data.

dataefficiencytraining

GEO-Flag: Detecting and Measuring GEO-Optimized Web Content

Aug 17, 2026

Junjie Chu, Ye Leng, Mingjie Li et al.

Generative search engines are vulnerable to content optimization tactics that inflate the visibility of low-authority or false information; systematic detection methods are now possible but require careful design to avoid relying on author-based shortcuts.

This paper introduces GEO-Flag, a system for detecting web pages optimized for generative search engines (like Google's AI Overviews).

safetyevaluationdata
dataarchitectureevaluation

LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

Aug 13, 2026

Fanfei Li, Jana Zeller, Manuel Prada-Corral et al.

Training on a carefully curated, grade-level-appropriate curriculum creates a sandbox for studying knowledge acquisition with clear boundaries—useful for understanding how models learn and what happens when you try to teach them new concepts.

Researchers created LittleLeaner, a 5B-parameter language model trained on an 88B-token curriculum limited to U.S. Grade 5 material, to study how models acquire knowledge under controlled conditions. Unlike models trained on messy web data, LittleLeaner has clear, interpretable knowledge boundaries, making it easier to understand what the model knows and how it learns new information.

trainingdataevaluation

DFM Mimir v1: An Open HRM Delivering Frontier Performance at 1B Parameters Using Only Permissible Post-Training Data

Aug 13, 2026

Peter Schneider-Kamp, Jacob Nielsen, Gianluca Barmina et al.

You can build competitive frontier-level language models at 1B parameters using only openly licensed data, making it feasible for researchers and organizations to develop ethical AI without relying on scraped or restricted datasets.

Mimir v1 is a 1-billion-parameter language model trained entirely on permissible (legally and ethically sourced) data that achieves competitive performance with much larger models. It uses a Hierarchical Reasoning Model architecture and excels at English, math, code, and Danish tasks—showing that high-quality open-source models don't require massive proprietary datasets.

trainingdataefficiency

Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

Aug 13, 2026

Yuto Nishida, Hirokazu Kiyomaru, Yusuke Oda et al.

You can measure training data importance during pretraining by tracking parameter trajectories, revealing that different data types matter at different training stages—literature early, STEM later—without needing task-specific validation sets.

This paper introduces a new method to measure how much training data influences language model development without needing to pick specific downstream tasks. Instead of testing on particular benchmarks, the researchers measure influence by tracking how each piece of training data pushes the model toward its final parameters.

trainingdataevaluation

Wasserstein Filtering: A Sample Selection Method for Robust Distribution Learning

Aug 13, 2026

Yikai Xu, Zhao Chen, Jian Huang

You can robustly learn from contaminated data by selecting the subset of samples that maximizes Wasserstein distance from the full dataset—this works as a model-agnostic preprocessing tool before training any model.

This paper introduces Wasserstein Filtering, a method to clean contaminated datasets by selecting samples whose distribution is most different from the full dataset. The approach uses optimal transport theory to identify and remove outliers, with theoretical guarantees and practical algorithms that work as a preprocessing step for any downstream task.

dataevaluation

Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages

Aug 12, 2026

Avijit Roy, Proma Roy

AI systems for underrepresented languages fail at the infrastructure level—in data collection, tokenization, and deployment design—long before model training begins. Fixing this requires treating offline-first design and linguistic diversity as core infrastructure priorities, not afterthoughts.

This paper reveals how AI infrastructure systematically disadvantages speakers of underrepresented languages like Bengali before models are even trained.

dataapplications

A Cascaded Unsupervised-Supervised NLP Pipeline for Detecting Accusatory Language in Public Procurement

Aug 12, 2026

Bryan Torres, Daniel Riofrío, José Vega-Sánchez et al.

Lightweight, domain-specific NLP models can effectively detect fraud signals in real-world government data without expensive infrastructure—useful for building practical oversight tools in resource-constrained settings.

This paper develops a hybrid NLP pipeline to detect accusatory language in public procurement comments from Ecuador's procurement system.

applicationsdata

ConVAWG: A Retrieval-Grounded Framework for Controlled Synthetic Dialogue Generation in Violence Against Women and Girls

Aug 11, 2026

Chen Lyu, Xingwei Tan, Simon Cullen et al.

This work demonstrates how to generate sensitive, realistic training data for abuse detection by modeling VAWG as temporally unfolding multi-turn conversations rather than isolated toxic sentences, enabling better downstream safety systems.

ConVAWG is a framework for generating synthetic multi-turn dialogues depicting Violence Against Women and Girls scenarios. Using retrieval-grounded methods, persona seeds, crime definitions, and real case reviews, it creates realistic abuse scenarios with controlled toxicity while respecting privacy constraints that prevent releasing real conversation data.

datasafetyapplications

MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment

Aug 11, 2026

Changhao Xiang, Shangyu Xing, Zhen Wu et al.

By interleaving visual objects directly into text during pretraining, you can teach multimodal models object-level grounding 12x more efficiently than traditional image-text pair training.

This paper introduces MultiModal Code-Switching (MMCS), a new way to train vision-language models by replacing words in text with their corresponding visual objects. Instead of just pairing whole images with descriptions, MMCS explicitly shows the model which objects match which words, making training much more efficient—achieving the same performance with 12x less data.

multimodaltrainingdata
trainingapplicationsdata

Reasoning Core: Designing Broad Procedural Data for Completion-Supervised Reasoning Training

Aug 5, 2026

Damien Sileo, Valentin Lacombe, Dimitri Kachler

Procedural datasets for training reasoning models need careful design beyond correctness: difficulty calibration, compact targets, and rigorous auditing (combining model review, human judgment, and testing) significantly impact training utility.

This paper introduces Reasoning Core, a collection of 50 procedural generators that create verifiable reasoning problems across diverse domains (math, logic, planning, code, etc.). The authors compare their dataset with three alternatives using completion-supervised fine-tuning on 3B models, showing Reasoning Core achieves better performance on reasoning benchmarks.

trainingdatareasoning

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

Aug 5, 2026

Indraneil Paul, Falko Helm, Goran Glavaš et al.

Training language models on code contexts that span multiple related files—rather than isolated snippets—dramatically improves their ability to handle long contexts and understand complex codebases, even when this data makes up a small fraction of total training.

OctoLong is a pipeline that creates long, dependency-rich code contexts by automatically retrieving related code files across repositories.

trainingdata

Agogic: Performance-Timed Music Tokens for LLM-Native Text-to-Symbolic-Music Generation

Aug 4, 2026

Junhao Chen, Mingjin Chen, Jingjia Mao et al.

Music tokenization design is more important than model size for text-to-music generation—a small model with the right token representation outperforms massive models with poor representations, challenging the field's scaling assumptions.

This paper investigates how music tokenization—the way music is converted into discrete symbols for language models—affects text-to-music generation quality. By fixing model size, data, and training approach while swapping only the tokenization scheme, researchers found that representation choice matters far more than model scale.

architecturedataevaluation

Assessment of Conditional Diffusion Model for Synthetic Histopathology Image Generation

Aug 4, 2026

Seyed Kahaki, Shijie Li, Weijie Chen et al.

When generating synthetic medical images, use domain-specific evaluation metrics rather than generic ones—and prioritize generating diverse data over pixel-perfect realism for downstream medical AI tasks.

This paper evaluates how well synthetic histopathology images generated by diffusion models work for medical AI tasks. The authors show that standard image quality metrics (FID, IS) designed for natural images fail for medical images, and propose using pathology-specific metrics instead.

evaluationdataapplications
training
data
agents

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

Jul 30, 2026

Manyi Wang, Junjielong Xu, Pinjia He

When using SWE-bench benchmarks to evaluate LLM coding abilities, be aware that over 1 in 10 test cases have misaligned problem statements and solutions—PAIChecker can automatically identify these problematic cases to improve benchmark reliability.

This paper identifies a critical quality issue in SWE-bench-like benchmarks used to evaluate AI coding abilities: 13.6% of PR-Issue pairs are misaligned, meaning the issue description doesn't actually match the code changes.

evaluationdata

ScaFE: Data-Efficient Scar Classification with LLM-Generated Clinical Feature Programs

Jul 30, 2026

Ruman Wang, Hangting Ye

LLMs can transfer medical knowledge into auditable, locally-executable feature programs instead of making direct predictions—improving data efficiency, privacy, and reproducibility in medical image classification.

ScaFE uses an LLM to generate executable programs that measure clinical scar features from photos, rather than asking the model to diagnose directly. These programs run locally, protecting patient data, and feed structured features into a simple Random Forest classifier.

applicationsefficiencydata

AI systems and the reproduction of (standard) language ideologies in World Englishes

Jul 30, 2026

Kingsley Ugwuanyi

AI systems aren't neutral—they embed and amplify existing power structures around which English dialects are considered 'legitimate,' with real consequences for speakers of non-standard varieties who are increasingly mistaken for AI.

This paper examines how AI language models reflect and reinforce biases toward 'standard' English while marginalizing non-dominant varieties spoken globally. Using examples from training data, model design, and public discourse, it shows how AI systems reproduce language ideologies that privilege English from wealthy countries while treating other Englishes as suspect or AI-like.

safetydataalignment

Beyond Sentiment: Structured Information Extraction from Financial News

Jul 30, 2026

Daohan Zhu, Sitong Ge, Ruofei Wang et al.

Financial sentiment analysis misses critical predictive signals; extracting structured semantic dimensions (event type, impact scope, temporal horizon, confidence) alongside sentiment improves stock prediction accuracy and reveals that sentiment-return relationships are highly nonlinear.

This paper shows that financial news contains multiple independent information dimensions beyond sentiment—like event type, impact scope, and time horizon—that together predict stock movements better than sentiment alone.

applicationsdata

Towards Autonomous Aircraft Surveillance from Nanosatellites through On-Board Inference and Generative Data Augmentation

Jul 30, 2026

Antonio Delgado-Rosa, David Muñoz-Valero, Enrique Adrian Villarrubia-Martin et al.

On-device inference on resource-constrained satellites combined with synthetic data generation can enable autonomous, real-time aircraft detection without overwhelming downlink capacity—shifting satellites from passive data collectors to active decision-makers.

This paper tackles satellite-based aircraft detection by running AI inference directly on small satellites (CubeSats) instead of sending raw images to Earth, and uses AI-generated synthetic images to train better detection models. The approach balances limited satellite bandwidth with scarce training data, enabling real-time autonomous surveillance from orbit.

efficiencyapplicationsdata

A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports

Jul 30, 2026

Jia Yu, Yan Zhu, Yili He et al.

Routine clinical reports contain rich supervision signals—you can extract finding-to-frame correspondence at scale to train medical vision-language models without expensive frame-level annotation, turning existing documentation into training data.

EndoCLIP is a vision-language model trained on 280,000 colonoscopy reports to link clinical findings with individual video frames.

multimodaldata

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

Jul 27, 2026

Zhen Huang, Yikun Wang, Shijie Xia et al.

Instead of applying uniform data processing rules, adapting the cleaning strategy per example—deciding what operation each piece of data needs—improves LLM pretraining efficiency and downstream performance.

DataOrchestra is a framework that customizes data processing for each example in pretraining, rather than applying one fixed strategy to all data. An orchestrator decides whether to drop, keep, or clean each data chunk, and if cleaning is needed, selects specific operations like editing or rewriting.

trainingdataefficiency
alignmentdata