Adding Custom Packages To cbcore Container#
You can add custom packages to the Cerebras cbcore container. This section describes how to add two specific packages: the spektral
and networkx
, to your Cerebras cbcore container.
Adding spektral
and networkx
packages#
Requirements#
Read this before you proceed further
Make sure that your system administrator has enabled the user control of binds for Singularity. See here for more: Bind Paths and Mounts — Singularity container 3.0 documentation .
Know that you are responsible to maintain the directory of packages you are installing in the Cerebras container (the
~/Cerebras-1.6.1-extra-python-packages
directory in the example installation instructions below).Ensure that you map the directory containing these packages to all the Slurm nodes in your cluster, for example via NFS.
Installation#
Create a directory shared between and accessible by all the hosts. For example, a directory with the name:
~/Cerebras-1.6.1-extra-python-packages
.Run the following
singularity exec
command with-B
option to perform the following two tasks:Access the
~/Cerebras-1.6.1-extra-python-packages
directory as$HOME/.local
directory from within the Cerebras container, andInstall the
spektral
andnetworkx
packages in$HOME/.local
. In reality these packages are installed in~/Cerebras-1.6.1-extra-python-packages
directory but they are accessible within the Cerebras container as$HOME/.local
directory.TensorFlow
singularity exec -B ~/Cerebras-1.6.1-extra-python-packages:$HOME/.local /<path>/cbcore-1.6.1-<identifier-string>.sif pip install --user spektral==1.0.7 networkx==2.6.2
PyTorch
singularity exec -B ~/Cerebras-1.6.1-extra-python-packages:$HOME/.local /<path>/cbcore-1.6.1-<identifier-string>.sif python-pt -m pip install --user spektral==1.0.7 networkx==2.6.2
Accessing the packages#
Run the following command to test that the modules in the installed packages are accessible from within the Cerebras container:
TensorFlow
singularity exec -B ~/Cerebras-1.6.1-extra-python-packages:$HOME/.local /<path>/cbcore-1.6.1-<identifier-string>.sif python -c 'import spektral; print(spektral.__file__)'
PyTorch
singularity exec -B ~/Cerebras-1.6.1-extra-python-packages:$HOME/.local /<path>/cbcore-1.6.1-<identifier-string>.sif python-pt -c 'import spektral; print(spektral.__file__)'
The output should resemble the following, if the $HOME
value is /my/home/username
:
/my/home/username/.local/lib/python3.7/site-packages/spektral/__init__.py
.