Training and fine-tuning run simulator

Price a training run the way it actually happens: with failures, checkpoints, and lost work between them.

A training run is a project with a start, an end, and a budget — and at cluster scale it is interrupted constantly. Every failure throws away all work since the last checkpoint, and every checkpoint costs time to write. This simulator prices the run as scheduled, the run as it really goes, and the fine-tuning jobs that follow.

01: The run, as scheduled

02: Failures and checkpoints

Interruptions are scale-dependent, and every one costs the work since the last checkpoint.

Where the run's GPU-time goes

Blue is productive training. Amber is checkpoint writing. Red is work lost to failures and restarts.

  • Productive training
  • Checkpoint writing
  • Lost to failures

More Info

Training is a synchronous project: thousands of accelerators advance in lockstep, so one failure stalls all of them, and recovery means rolling every GPU back to the last checkpoint. That makes two storage numbers into training-speed numbers—how fast a checkpoint writes, and how fast it reloads—and it makes the failure interval the most expensive statistic nobody puts on the datasheet.

Frequently asked questions

Where this leads

From AI factory math to AI factory control

Checkpoint write and restore speed is a data-delivery problem: the whole cluster stalls at the pace of the storage path. The same is true of loading the training data itself—a starved data pipeline shows up as idle accelerators, not as a storage alert.