modelzoo.transformers.data_processing.scripts.chunk_preprocessing.data_reader#

This module contains helper functions and classes to read data from different formats, process them, and save in HDF5 format. It supports JSONL, GZipped JSON, Parquet, ZST compressed JSONL, and TAR archives of ZST compressed JSONL files.

Classes:
DataFrame:

An object to hold and process data with the ability to serialize itself into an HDF5 format.

Reader:

Provides a mechanism to read data from multiple file formats, process it, and yield in manageable chunks.

Functions

find_last_paragraph_or_sentence_end

Find the last end of a paragraph (denoted by '

get_data_size

Compute the size of the given data.

set_doc_idx

split_entry_by_paragraph_or_sentence

Split a large entry into chunks by sentence or paragraph end.

Classes

DataFrame

Initialize the DataFrame object.

Reader

Initialize the Reader instance.