Smart memory management—not just model compression—can unlock 200K-token context on consumer hardware, making local AI development practical for reasoning tasks.
JustFit enables running large language models with 200K-token context on a 24GB laptop by intelligently managing memory. It uses three techniques: compressing key-value cache data, swapping model components between storage and RAM, and preserving state across requests. Tests show it can handle 6.93x more context than existing methods while maintaining reasonable speed.