modelzoo.transformers.data_processing.huggingface.CSDataCollatorForLanguageModeling.CSDataCollatorForLanguageModeling

class modelzoo.transformers.data_processing.huggingface.CSDataCollatorForLanguageModeling.CSDataCollatorForLanguageModeling

Bases: transformers.DataCollatorForLanguageModeling

Overrides DataCollatorForLanguageModeling from HF to shift the inputs/labels in the dataloader

Methods

torch_call

__call__(*args: Any, **kwargs: Any) → Any

Call self as a function.

__init__(*args: Any, **kwargs: Any) → None

static __new__(cls, *args: Any, **kwargs: Any) → Any

previous

modelzoo.transformers.data_processing.huggingface.CSDataCollatorForLanguageModeling

next

modelzoo.transformers.data_processing.huggingface.HF_converter_example_BookCorpus