modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.split_text_and_tokenize#
- modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.split_text_and_tokenize(text, tokenizer, max_tok_len=2000, remove_bos_in_chunks=True)[source]#
Function to split the text into smaller sequences of length max_tok_len and then tokenize each of the smaller sequences. This is done to avoid performance issues with tokenizers like LlamaTokenizer which are slow for long sequences.
- Parameters
text (str) – text to be tokenized
tokenizer (Tokenizer) – tokenizer to be used
max_tok_len (int, optional) – max length of each sequence. Defaults to 2000.
remove_bos_in_chunks (bool, optional) – whether to ignore bos token id in chunks. Defaults to True.
- Returns
list of token ids for the text
- Return type
tok_ids (list)