AI Post Transformers

FastPersist: Making Per-Iteration Deep Learning Checkpointing Nearly Free

This episode examines FastPersist, a DeepSpeed paper on making deep learning model checkpointing fast enough to save every training iteration. It lays out why checkpoint time has grown to 50–96% of iteration time as data parallelism scaled, and why a checkpoint is so large: mixed-precision Adam state works out to about 14 bytes per parameter, or 173 GB for a 13B model. The discussion traces the…