Import user-specific dependencies in Cerebras environment#

If you are developing your own data loader, you might need to provide specific dependency packages in the virtual Python environments to support your data loader functions. Before 1.8.0, you could store these dependency packages in the shared storage that Cerebras appliance can access to, and specify --mount_dir and --python_path when invoking to allow the workers inside the appliance to find these dependency packages. This approach is hard to use and can be error-prone.

As of release 1.9.0, the Custom Worker Container Workflow provides seamless support for importing user-specific dependency packages in the Python environments into Cerebras appliance. With this feature, you do not need any special handling. The script will automatically find all pip-installed dependency packages on the user node, and apply them on Cerebras appliance when the data loader functions are deployed.

In the event where the custom worker container failed to be created, a fallback policy would take into effect by mounting the site packages from the user node Python virtual environment to the worker environment via a predefined NFS-based cluster volume.

In 1.9.0, both the custom worker container feature and the venv mounting fallback policy has been enabled by default. If you would like to disable the features, there are two Cerebras-specific options that can support that. We call these options debug_args.

  • To disable the custom worker container feature, the option is named debug_args.debug_usr.skip_image_build.

Setting this option to True will disable this feature.

  • To disable the venv mounting fallback policy, the option is named

debug_args.debug_usr.skip_user_venv_mount. Setting this option to True will disable the fallback policy.

How to Disable Custom Worker Container Workflow#

Save the following script as on any accessible directory on the user node:

from cerebras_appliance.run_utils import write_debug_args
from cerebras_appliance.pb.workflow.appliance.common.common_config_pb2 import DebugArgs

debug_args = DebugArgs()
debug_args.debug_usr.skip_image_build = True
debug_args.debug_usr.skip_user_venv_mount = True

out_file = "debug_args.out"
write_debug_args(debug_args, out_file)

Run it inside a Cerebras virtual environment you have setup on the user node, see Set up a Cerebras virtual environment:

(venv_cerebras_pt) $ python

This script will produce a file debug_args.out. You can then add the additional option when invoking as follows:

(venv_cerebras_pt) $ python <other args> --debug_args_path debug_args.out


Currently, the custom worker container workflow only accounts for Python dependencies that were pip installed on the user node and assumes Internet and PyPI access from the workers in Cerebras appliance. If workers do not have access to the Internet or PyPI, the workflow will fail and the fallback policy will take into effect.