modelzoo.transformers.data_processing.utils.whitespace_tokenize

modelzoo.transformers.data_processing.utils.whitespace_tokenize(text, lower=False)

Splits a piece of text based on whitespace characters

previous

modelzoo.transformers.data_processing.utils.text_to_tokenized_documents

next

modelzoo.transformers.data_processing.utils.maskedLmInstance