cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingCSWriter#
- class cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingCSWriter(checkpoint_file)[source]#
Bases:
cerebras.modelzoo.tools.checkpoint_converters.streaming_checkpoints.StreamingCSWriterViewWrites a Cerebras H5 checkpoint in a streaming (incremental) manner rather than accumulating the full checkpoint into memory and then writing all weights at the end.
It is essential that self.save() is called in order to flush the required metadata (state’s spec). Without this call, the resulting checkpoint will not be able to be loaded with cstorch.load(…).
The StreamingCSWriter class supports re-accessing and even updating keys that have already been written. There are two restrictions: 1. An existing key that stores a dict/list/tuple cannot be replaced. 2. An existing key storing any type cannot be replaced by a dict/list/tuple
- Parameters
checkpoint_file – Path to new H5 checkpoint. A file cannot already exist at this location.
Methods
getitemskeyspopsavevalues