modelzoo.transformers.data_processing.scripts.pile.download.download_tokenizer_files#

modelzoo.transformers.data_processing.scripts.pile.download.download_tokenizer_files(args)[source]#

Download files needed for tokenization for dataset creation.

Parameters

args (argparse namespace) – Arguments for downloading the tokenizer files.

previous

modelzoo.transformers.data_processing.scripts.pile.download.download_pile

next

modelzoo.transformers.data_processing.scripts.pile.download.get_urls_for_tokenizer_files