cerebras.modelzoo.data_preparation.data_preprocessing.dpo_token_generator#

Classes

DPOTokenGenerator

Initialize the DPTokenGenerator class with dataset parameters, tokenizer, and token IDs.

previous

cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.Reader

next

cerebras.modelzoo.data_preparation.data_preprocessing.dpo_token_generator.DPOTokenGenerator