modelzoo.transformers.data_processing.slimpajama.preprocessing.datasets.RedPajamaReplication

class modelzoo.transformers.data_processing.slimpajama.preprocessing.datasets.RedPajamaReplication

Bases: modelzoo.transformers.data_processing.slimpajama.preprocessing.datasets.Dataset

Methods

already_shuffled	Datasets where the source is already shuffled should override this to return True so that it isn't shuffled again.
dir_path	Path to the directory
documents	A generator producing all documents in the dataset.
name	Human-readable name of tfhe dataset
num_docs	Return an estimate of the dataset number of documents.
sample_documents
short_documents_path	Path to the file with short documents
size	Return an estimate of the dataset size.

__init__(datasets, duplicates, short_docs)

already_shuffled()

Datasets where the source is already shuffled should override this to return True so that it isn’t shuffled again.

dir_path()

Path to the directory

documents(queues)

A generator producing all documents in the dataset.

name()

Human-readable name of tfhe dataset

num_docs()

Return an estimate of the dataset number of documents. Implementations may use a faster, less accurate estimate.

short_documents_path()

Path to the file with short documents

size()

Return an estimate of the dataset size. Implementations may use a faster, less accurate estimate.

previous

modelzoo.transformers.data_processing.slimpajama.preprocessing.datasets.RedPajamaGithubDataset

next

modelzoo.transformers.data_processing.slimpajama.preprocessing.datasets.RedPajamaStackExchangeDataset