modelzoo.transformers.data_processing.huggingface#

CSDataCollatorForLanguageModeling

HF_converter_example_BookCorpus

Example script to convert HF BookCorpus dataset to HDF5 dataset

HF_converter_example_Eli5

Example script to convert HF Eli5 dataset to HDF5 dataset

HuggingFaceDataProcessor

Pytorch HuggingFace Dataloader

HuggingFace_BookCorpus

HuggingFace BookCorpus Dataset

HuggingFace_Eli5

HuggingFace Eli5 Dataset