Seamlessly import user-specific dependency packages to Cerebras environment#

Users with more advanced data loader requirements might need to provide their specific dependency packages in their Python environments to support their data loader functions. Before 1.8.0, users can store these dependency packages in the shared storage that Cerebras appliance can access to, and specify --mount-dir and --python-path when invoking to allow the workers inside the appliance to find these dependency packages. This approach is hard to use and can be error-prone. More info in Setup Cerebras virtual environment.

In 1.8.0, we introduce a Custom Worker Container Workflow to provide seamless support for importing user-specific dependency packages in their Python environments into Cerebras appliance. With this feature, users do not need any special handling. The script will automatically find all pip-installed dependency packages on the user node, and apply them on Cerebras appliance when the data loader functions are deployed.

In 1.8.0, this feature is not enabled by default and is gated behind a Cerebras-specific option. We call these options debug_args. For this feature, the option is named debug_args.debug_usr.build_custom_worker. Setting this option to True will enable this feature.

How to Enable Custom Worker Container Workflow#

Save the following script as on any accessible directory on the user node:

from cerebras_appliance.run_utils import write_debug_args
from cerebras_appliance.pb.workflow.appliance.common.common_config_pb2 import DebugArgs

debug_args = DebugArgs()
debug_args.debug_usr.build_custom_worker = True

out_file = "debug_args.out"
write_debug_args(debug_args, out_file)

Run it inside a Cerebras virtual environment you have setup on the user node, see Setup Cerebras virtual environment:


This script will produce a file debug_args.out. You can then add the additional option when invoking as follows:

python <other args> --debug_args_path debug_args.out

This applies to both TensorFlow and PyTorch.


There are a few limitations in the custom worker container workflow in 1.8.0.

  • Currently, the custom worker container workflow only accounts for Python dependencies that were pip installed on the user node and assumes PyPI access from the workers in Cerebras appliance. If workers do not have access to PyPI, the workflow will fail.

  • The error propagation back to the end users is not perfect. Some errors might only be discovered by looking through the logs stored inside the appliance. The appliance logs can be extracted using csctl log-export command as described in Log Export.