Cerebras Model Zoo supported config parameters#

The Cerebras Model Zoo supports an extensive range of models, each with its own set of parameters. This document provides a comprehensive list of these parameters as defined in the Model Zoo.

With the introduction of Config classes in the Model Zoo, each parameter is defined in the model’s config file. These classes, implemented as Python dataclasses, organize and validate the parameters necessary for model definition and training.

Common parameters across models#

1. RunConfig Parameters:

RunConfig

2. Sparsity Parameters:

SparsityBaseConfig

3. Optimizer Parameters:

OptimizerConfig

Model Specific Parameters#

Large Language Model (LLM) parameters#

  1. BERT: BertConfig

  2. Bloom: BloomConfig

  3. BTLM: BTLMConfig

  4. DPO: DPOConfig

  5. DPR: DPRConfig

  6. Falcon: FalconConfig

  7. GPT2: GPT2Config

  8. GPT3: GPT3Config

  9. GPTJ: GPTJConfig

  10. Llama: LlamaConfig

  11. Mistral: MistralConfig

  12. MPT: MPTConfig

  13. SantaCoder: SantaCoderConfig

  14. StarCoder: StarCoderConfig

  15. T5: T5Config

  16. Transformer: TransformerConfig

Vision model parameters#

  1. DiT: DiTConfig

  2. Vision Transformer: VisionTransformerConfig

Multimodal model parameters#

  1. LLaVA: LlavaConfig

Understanding Config classes structure#

Each of these parameters are present as part of a Config class. A Config class, implemented as a Python dataclass, serves as a container for storing essential settings and parameters needed for defining and training a model.

Each of these class atributes correspond to the respective section in a YAML file which is used to define the parameters for a training run.

A config class looks like this:

class <ConfigClass>:
    train_input = Optional[DataConfig] = None

    eval_input = Optional[DataConfig] = None

    model = <ModelConfigClass> = required

    sparsity: Optional[SparsityConfig] = None

    optimizer: OptimizerConfig = required

    runconfig: RunConfig = required

For more information about Config classes, refer to the Model Zoo config classes documentation.

Additional notes#

Each model has a designated ModelConfigClass. In cases where a model is a variant of another, it may inherit the ModelConfigClass from the parent model. To understand more about this inheritance and the hierarchy of config classes, visit the Config class hierarchy documentation.

previous

Model Zoo usage examples

next

Convert checkpoints and model configs