cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.verify_saved_hdf5_files#

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.verify_saved_hdf5_files(params)[source]#

This function is used to do sanity checks at the end of the creation of hdf5 files. This function loads every .h5 files generated and checks:

  1. The data type

  2. Shape of the dataset

  3. Fact that labels and inputs are as expected

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.validate_tokens

next

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.verify_saved_hdf5_files_mp