Measure throughput of your model#

It is often desirable to measure the throughput of a model. In order to provide this information out of the box, Cerebras Model Zoo runs print a couple of throughput metrics to the console as well as sending them to events files that are then viewable in TensorBoard. This section describes what these metrics are, how they are calculated, and intricacies to be aware of when interpreting them.

Out-of-the-box throughput metrics from Cerebras Model Zoo#

There are two throughput metrics that are printed to the console by default when running a model using Cerebras Model Zoo. These are Rate and GlobalRate.

It is important to note that these metrics are what’s measured by the user node. While they are useful for getting an overall picture of throughput, they are not exact measurements of throughput seen by the Cerebras Wafer-Scale Cluster. This is due to the asynchronous nature of execution on Cerebras Wafer-Scale, where input workers stream data to the wafer quasi-independently of the user node that’s receiving the outputs.


GlobalRate measures the average throughput of the entire training run. It does so by dividing total samples for which outputs have been received by total time since the executable was fully loaded to the Wafer-Scale Engine.


GlobalRate is also logged to events files and is viewable in TensorBoard as avg_samples_per_sec.


Rate measures a smoothed out version of GlobalRate, where at each sampling interval (i.e. logging step), a smoothing factor (default of 0.4) is applied to the previously calculated Rate and added to the local throughput since the last sampling point.


Rate is also logged to events files and is viewable in TensorBoard as local_samples_per_sec.


While Rate is more susceptible to spikes than GlobalRate, it is more representative of the current throughput measured by the user node.

Ephemeral throughput spikes after checkpointing#

During a checkpointing step, wafer stops processing samples until a checkpoint is taken. Once checkpointing on the Wafer-Scale Cluster is complete, streaming samples from input workers is immediately resumed. However, while the WSE is processing samples post the checkpoint step, user node may still be downloading the checkpoint from the Wafer-Scale Cluster, which could take some time, especially for large checkpoints. As such, the WSE may have computed outputs for a number of samples but user node will only fetch those outputs once it has fully downloaded the checkpoint. As a result, once checkpointing on the user node is complete and it starts fetching outputs from the Wafer-Scale Cluster, a large number of outputs may be readily available for fetching, which could result in a large spike in local throughput (i.e., Rate) seen. Once user node catches up to the wafer, throughput will stabalize and return to normal.


This effect is less pronounced in GlobalRate when doing long training runs since it’s amortized over the entire training duration.

Throughput in Weight Streaming execution#

In Weight Streaming execution, outputs (such as losses, summaries, etc.) are received as soon their values have been computed by the wafer (except after a checkpointing step, as described above). As such, there’s a close one-to-one correspondence between the throughput achieved by the wafer vs. what the user node sees (i.e., Rate and GlobalRate).

Having said that, the first few logging steps may present outlier throughputs due to difference in when the clock is started on the user node vs. when the wafer actually starts processing data. This effect is short-lived and steady-state throughput is achieved quickly thereafer.