modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.wikitext_detokenizer

modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.wikitext_detokenizer(string)

Detokenizer for wikitext. Used for special handling of data for substrings.

Parameters

string (str) – String to detoknize before tokenization.

Returns

Detokenized string

previous

modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.verify_saved_hdf5_files_mp

next

modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.DatasetStats