cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.chunk_data_preprocessor.update_progress#

cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.chunk_data_preprocessor.update_progress(pbar: tqdm.tqdm, progress_counter: multiprocessing.context.BaseContext.Value, total_chunks: int, start_time: float, stop_event: threading.Event) → None[source]#

Update the progress bar based on the current progress.

Parameters

  • pbar (tqdm) – The progress bar instance.

  • progress_counter (Value) – A shared counter to track progress across processes.

  • total_chunks (int) – Total chunks to process.

  • start_time (float) – The start time of the process.

  • stop_event (Event) – Event to signal when to stop updating progress.

Returns

None

previous

cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.chunk_data_preprocessor.get_compression_factor

next

cerebras.modelzoo.data_preparation.nlp.chunk_data_processing.chunk_data_preprocessor.ChunkDataPreprocessor