Generating training data in parallel using older model checkpoints rather than waiting for the latest model weights.