cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.dpo_data_token_generator#

Classes

DPOTokenGenerator

Initialize the DPTokenGenerator class with dataset parameters, tokenizer, and token IDs.

previous

cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.data_reader.Reader

next

cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.dpo_data_token_generator.DPOTokenGenerator