cerebras.modelzoo.data_preparation.data_preprocessing.preprocess_data

Script to generate an HDF5 dataset for GPT Models.

Functions

main	Main function for execution.

previous

cerebras.modelzoo.data_preparation.data_preprocessing.nlg_token_generator.NLGTokenGenerator

next

cerebras.modelzoo.data_preparation.data_preprocessing.preprocess_data.main