cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.get_files#

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.get_files(input_dir=None, filetypes=None, metadata_files=None)[source]#

Get all files of given filetypes from input directory.

Parameters

  • input_dir (str) – Input directory to read files from.

  • filetypes (list) – File types to fetch from the given input directory. Defaults to None.

  • metadata_files (str) – Comma separated string of metadata files.

Returns

List of lists containing all file paths as strings

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.format_fim

next

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.get_params