cerebras.modelzoo.data_preparation.huggingface.HF_converter_example_BookCorpus

Example script to convert HF BookCorpus dataset to HDF5 dataset

Functions

main

previous

cerebras.modelzoo.data_preparation.huggingface.CSDataCollatorForLanguageModeling.CSDataCollatorForLanguageModeling

next

cerebras.modelzoo.data_preparation.huggingface.HF_converter_example_BookCorpus.main