When Trusting 11 Nines Nearly Broke a Production ML Pipeline

https://www.phone-bookmarks.win/how-to-move-hot-data-to-cold-storage-automatically-during-seasonal-traffic-spikes

When Trusting 11 Nines Nearly Broke Training: A Production Incident We had a model training cluster that churned through terabytes of image tiles every week