cerebras.modelzoo.data_preparation.utils.text_to_tokenized_documents(data, tokenizer, multiple_docs_in_single_file, multiple_docs_separator, single_sentence_per_line, spacy_nlp)[source]#

Convert the input data into tokens :param str data: Contains data read from a text file :param tokenizer: Tokenizer object which contains functions to

convert words to tokens

  • multiple_docs_in_single_file (bool) – Indicates whether there are multiple documents in the given data string

  • multiple_docs_separator (str) – String used to separate documents if there are multiple documents in data. Separator can be anything. It can be a new blank line or some special string like “—–” etc. There can only be one separator string for all the documents.

  • single_sentence_per_line (bool) – Indicates whether the data contains one sentence in each line

  • spacy_nlp – spaCy nlp module loaded with spacy.load() Used in segmenting a string into sentences

Return List[List[List]] documents

Contains the tokens corresponding to sentences in documents. List of List of Lists [[[],[]], [[],[],[]]] documents[i][j] -> List of tokens in document i and sentence j