– Addressing data-loading performance bottlenecks in ML training workflows
– Utilizing Cloud Storage feature Compose Objects to combine smaller objects into larger ones for faster data fetching
– Employing high-throughput parallel-listing to speed up metadata retrieval for datasets
Dataflux has achieved significant performance gains for Dataflux by addressing data-loading bottlenecks in ML training workflows. Data is loaded in batches from storage during training runs, and if the reading and processing of data takes longer than GPU computation, the GPU becomes underutilized, leading to longer training times. By using Cloud Storage features like Compose Objects, Dataflux can dynamically combine smaller objects into larger objects, reducing the number of objects that need to be fetched and improving data loading speeds.
Dataflux also employs high-throughput parallel-listing to speed up the initial metadata needed for datasets, using a work-stealing algorithm to optimize listings even on datasets with millions of objects. These optimizations help ensure minimal GPU stalls during ML training with Dataflux, leading to reduced training times and increased accelerator utilization. Fast-listing and dynamic-composition are implemented in the Dataflux Client Libraries available on GitHub.
Dataflux Dataset for PyTorch and the Dataflux Python client library are available for users to try out and see how they can boost their workflows. These tools are part of Dataflux’s effort to provide storage AI-related capabilities and improve ML training efficiencies. Users can learn more about Dataflux and its offerings from the Google Cloud Next ’24 recorded session “How to define a storage infrastructure for AI and analytical workloads.”