Seamlessly import user-specific dependency packages to Cerebras environment#
Users with more advanced data loader requirements might need to provide their specific
dependency packages in their Python environments to support their data loader functions. Before 1.8.0,
users can store these dependency packages in the shared storage that Cerebras appliance can access to,
and specify --mount-dir and --python-path when invoking run.py to allow the workers inside the appliance to
find these dependency packages. This approach is hard to use and can be error-prone. More info in Setup Cerebras virtual environment.
In 1.8.0, we introduce a Custom Worker Container Workflow to provide seamless support for importing
user-specific dependency packages in their Python environments into Cerebras appliance. With this feature,
users do not need any special handling. The run.py script will automatically find all
pip-installed dependency packages on the user node, and apply them on Cerebras appliance when the data loader
functions are deployed.
In 1.8.0, this feature is not enabled by default and is gated behind a Cerebras-specific option. We call these
options debug_args. For this feature, the option is named debug_args.debug_usr.build_custom_worker. Setting
this option to True will enable this feature.
How to Enable Custom Worker Container Workflow#
Save the following script as debug_args_writer.py on any accessible directory on the user node:
from cerebras_appliance.run_utils import write_debug_args
from cerebras_appliance.pb.workflow.appliance.common.common_config_pb2 import DebugArgs
debug_args = DebugArgs()
debug_args.debug_usr.build_custom_worker = True
out_file = "debug_args.out"
write_debug_args(debug_args, out_file)
Run it inside a Cerebras virtual environment you have setup on the user node, see Setup Cerebras virtual environment:
python3 debug_args_writer.py
This script will produce a file debug_args.out. You can then add the additional option when invoking
run.py as follows:
python run.py <other args> --debug_args_path debug_args.out
This applies to both TensorFlow and PyTorch.
Limitations#
There are a few limitations in the custom worker container workflow in 1.8.0.
Currently, the custom worker container workflow only accounts for Python dependencies that were pip installed on the user node and assumes PyPI access from the workers in Cerebras appliance. If workers do not have access to PyPI, the workflow will fail.
The error propagation back to the end users is not perfect. Some errors might only be discovered by looking through the logs stored inside the appliance. The appliance logs can be extracted using csctl log-export command as described in Log Export.