cerebras.modelzoo.data_preparation.huggingface#

CSDataCollatorForLanguageModeling

HF_converter_example_BookCorpus

Example script to convert HF BookCorpus dataset to HDF5 dataset

HF_converter_example_Eli5

Example script to convert HF Eli5 dataset to HDF5 dataset

HuggingFaceDataProcessor

Pytorch HuggingFace Dataloader

HuggingFace_BookCorpus

HuggingFace BookCorpus Dataset

HuggingFace_Eli5

HuggingFace Eli5 Dataset

previous

cerebras.modelzoo.data_preparation.hdf5_shuffling.h5_dataset_shuffle.shuffle_dataset

next

cerebras.modelzoo.data_preparation.huggingface.CSDataCollatorForLanguageModeling