TensorFlow: Quick-Start Guide

TensorFlow: Quick-Start Guide

If you are new to Cerebras, begin with this quick-start guide. Before you get into in-depth development, follow this quick-start guide to familiarize yourself at a high level with the Cerebras system workflow.

Cerebras Wafer-Scale Clusters are composed of CS-2 systems, MemoryX and SwarmX nodes, input pre-processing servers and associated internal network switches. These clusters support two types of execution modes to enable ML models of different sizes:

  • Layer pipelined: In this mode all the layers of the network are loaded altogether onto the Cerebras WSE. This mode is selected for neural network models which fit entirely on the WSE, approximately up to 1B parameters.

  • Weight streaming: In this mode, one layer of the neural network model is loaded at a time. This layer-by-layer mode is used to run extremely large models (>1B parameters).

Cerebras Wafer-Scale Clusters rely on Kubernetes internally to manage various resources.

The weight streaming mode is supported only on Cerebras Wafer-Scale Clusters and via the appliance model. The appliance model is based on Kubernetes and enables you to submit a job to the cluster as if it were a single device; you only have to indicate how many CS-2 systems you would like to use for your run. To run your large-scale models in weight streaming mode, follow steps provided in Weight Streaming Appliance Workflow.

To run models in pipeline mode on Cerebras Wafer-Scale Clusters, Cerebras also leverages Kubernetes for internal resource allocation (instead of Slurm, which was used in the past releases). Pipelined jobs can be launched via similar scripts used in the past with a Slurm-based workflow. To run your small- to medium-sized models on a Wafer-Scale Cluster in Pipeline mode, follow the steps provided in Pipeline K8s Workflow.

Note

If you like to run PL models only and have not yet upgraded to the Wafer-Scale Cluster, you can still use Slurm-based workflow on your existing cluster. Note that this will not support weight streaming.

If you are ready to start developing / adapting your own TF code for CS System

Skip to Workflow for TensorFlow on CS for an in-depth development guide using TensorFlow for Cerebras.