modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils#

Functions

add_common_args

For the argparse to parse arguments for subcommands, we add common command line arguments to each subcommand parser here.

collect_stats

Collect statistics of the dataset.

create_features_auto_lm

Given a list of token_ids, generate input sequence and labels.

create_features_summarization

Given a list of prompt_ids and completion_ids, generate input sequence and labels.

dump_args

Write the input params to file.

dump_result

Write outputs of execution

get_files

Get all files of given filetypes from input directory.

get_params

Retrieve configuration parameters :returns:

get_parser

Argparser definition for command line arguments from user.

get_verification_args

Get arguments for verifying HDF5 dataset. :param params: Dictionary containing parameters for verifying HDF5 dataset. :type params: dict :param data_processor: Class containing methods that specify how the dataset will be processed and written into HDF5 files.

handle_jsonl

process_dataset

Process a dataset and write it into HDF5 format.

read_checkpoint

Checkpoint reader for execution.

set_defaults

update_params

Update config parameters with CLI arguments

validate_tokens

verify_saved_hdf5_files

This function is used to do sanity checks at the end of the creation of hdf5 files. This function loads every .h5 files generated and checks: 1. The data type 2. Shape of the dataset 3. Fact that labels and inputs are as expected.

verify_saved_hdf5_files_mp

Verify the generated HDF5 dataset.

wikitext_detokenizer

Detokenizer for wikitext.

Classes

DatasetStats

DatasetStats(num_sequences: int, num_tokens: int, detokenized_bytes: int, detokenized_chars: int, non_pad_tokens: int, loss_valid_tokens: int)

Reader

VerificationArgs

VerificationArgs(processes: int, files_per_record: int, max_seq_length: int, tokenizer_obj: object, eos_id: int, pad_id: int)