Shuffling Samples for HDF5 dataset of GPT models

Input files format

The script expects the dataset files to be in .h5 file format. They can be located either in the input_dir or one of its subdirectories. It can be found here.

Generating Shuffled HDF5 Dataset

bash launch_h5_shuffle.sh <path/to/input_dir> <path/to/output_dir> <num_chunks> <num_workers> <multi_modal_flag>

Here num_chunks is the number of output HDF5 files and multi_modal_flag denotes if the dataset is multimodal or not. This flag should be either empty or --multi_modal.

Output files structure

The output directory will contain a bunch of h5 files as shown below:

<path/to/output_dir>
├── logs/
├── shuf_split/
├── workers/
├── 0/
├── 1/
├── 2/
├── ⋮
├── data-00000.h5
├── data-00001.h5
├── data-00002.h5
└── ⋮

- The numbered directories (`0/`, `1/`, ...) are temporary directories that were created while shuffling the samples and can be removed at the end of the run. 
- `logs/` contains the logs of each worker. 
- `shuf_split/` contains `.shuf` files for each HDF5 input file to indicate it has completed reading all samples from that file.
- `workers/` contains `.txt` files for each worker to indicate that it has completed reading all samples from the HDF5 files that were assigned to it. They are used to sync between workers.