cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.set_doc_idx#

cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.set_doc_idx(df, file_idx, start_doc_idx, end_doc_idx)[source]#

This is used to set metadata for a given dataframe

Parameters

  • file_idx – The file index of the current dataframe

  • start_doc_idx – The starting doc index of the current dataframe

  • end_doc_idx – The ending doc index of the current dataframe

previous

cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.optional_lock

next

cerebras.modelzoo.data_preparation.data_preprocessing.data_reader.split_entry_by_paragraph_or_sentence