cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.chunk_data_preprocessor.get_compression_factor#
- cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.chunk_data_preprocessor.get_compression_factor(filename: str) int[source]#
Calculate and return the compression factor based on a file’s extension.
- Parameters
filename (str) – The name of the file.
- Returns
- Compression factor. Returns 3 for all compressed and parquet formats,
otherwise returns 1 for uncompressed formats.
- Return type
int