cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.read_checkpoint#

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.read_checkpoint(checkpoint_path, resume_from_checkpoint=True)[source]#

Checkpoint reader for execution.

Parameters

  • checkpoint_path (str) – Path to read checkpoint data from

  • resume_from_checkpoint (bool) – Resume from checkpoint for execution. Defaults to True.

Returns

Tuple containing number of files processed and the count of tfrecords/HDF5 files

written to output directory.

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.process_dataset

next

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.set_defaults