Current agent harness methods for continual learning underperform simple memory-based baselines on repeated tasks, but weight-based learning shows potential for generalizing to new domains—suggesting future work should focus on training models to better use external tools and memory systems.
This paper introduces Continual-ARC, a benchmark for measuring how AI agents learn over time by solving tasks from the ARC-AGI dataset. Tasks repeat with new instances and new tasks arrive continuously, mimicking real-world learning.