cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.get_verification_args

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.get_verification_args(processes, data_processor)

Get arguments for verifying HDF5 dataset. :param params: Dictionary containing parameters for verifying HDF5 dataset. :type params: dict :param data_processor: Class containing methods that specify how the dataset

will be processed and written into HDF5 files.

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.get_tokenizer_vocab

next

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.handle_bos_token_default