Migrating ML code between frameworks and hardware is hard enough that it serves as a meaningful benchmark for AI agents, and today's best models still fail on a quarter of real-world migration tasks.
XStack is a benchmark that tests AI agents' ability to migrate machine learning repositories from PyTorch (NVIDIA GPUs) to JAX (Google TPUs). Researchers created a grading system based on published results to evaluate whether migrated code works correctly.