modelzoo.transformers.data_processing.h5_map_dataset.dataset#

Classes

HDF5Dataset

Dynamically read samples from disk for using mapping paradigms.

RestartableDataLoader

The state we care about for allowing deterministic restart of instances of HDF5Dataset is the total number of samples streamed globally, which gets consumed by the sampler.