cerebras.modelzoo.data_preparation.data_preprocessing.custom_tokenizer_example.CustomLlama3Tokenizer#
Classes
Custom implementation of Llama3 Tokenizer, which overrides compute_offsets of the HuggingFace (which is buggy - https://github.com/huggingface/tokenizers/issues/1553). |