modelzoo.transformers.data_processing.scripts.pile.download.get_urls_from_split

modelzoo.transformers.data_processing.scripts.pile.download.get_urls_from_split(split)

Get urls given split of dataset.

Parameters

split (str) – Split of dataset to get urls for.

Returns

List of urls, containing jsonl.zst file names for downloading.

previous

modelzoo.transformers.data_processing.scripts.pile.download.get_urls_for_tokenizer_files

next

modelzoo.transformers.data_processing.scripts.pile.download.main