cerebras.modelzoo.data_preparation.huggingface.CSDataCollatorForLanguageModeling

Classes

CSDataCollatorForLanguageModeling

Overrides DataCollatorForLanguageModeling from HF to shift the inputs/labels in the dataloader

previous

cerebras.modelzoo.data_preparation.huggingface

next

cerebras.modelzoo.data_preparation.huggingface.CSDataCollatorForLanguageModeling.CSDataCollatorForLanguageModeling