Adding Custom Packages To cbcore Container#

You can add custom packages to the Cerebras cbcore container. This section describes how to add two specific packages: the spektral and networkx, to your Cerebras cbcore container.

Adding spektral and networkx packages#

Requirements#

Read this before you proceed further

  • Make sure that your system administrator has enabled the user control of binds for Singularity. See here for more: Bind Paths and Mounts — Singularity container 3.0 documentation .

  • Know that you are responsible to maintain the directory of packages you are installing in the Cerebras container (the ~/Cerebras-1.6.1-extra-python-packages directory in the example installation instructions below).

  • Ensure that you map the directory containing these packages to all the Slurm nodes in your cluster, for example via NFS.

Installation#

  1. Create a directory shared between and accessible by all the hosts. For example, a directory with the name: ~/Cerebras-1.6.1-extra-python-packages.

  2. Run the following singularity exec command with -B option to perform the following two tasks:

    • Access the ~/Cerebras-1.6.1-extra-python-packages directory as $HOME/.local directory from within the Cerebras container, and

    • Install the spektral and networkx packages in $HOME/.local. In reality these packages are installed in ~/Cerebras-1.6.1-extra-python-packages directory but they are accessible within the Cerebras container as $HOME/.local directory.

      • TensorFlow

      singularity exec -B ~/Cerebras-1.6.1-extra-python-packages:$HOME/.local /<path>/cbcore-1.6.1-<identifier-string>.sif pip install --user spektral==1.0.7 networkx==2.6.2

      • PyTorch

      singularity exec -B ~/Cerebras-1.6.1-extra-python-packages:$HOME/.local /<path>/cbcore-1.6.1-<identifier-string>.sif python-pt -m pip install --user spektral==1.0.7 networkx==2.6.2

Accessing the packages#

Run the following command to test that the modules in the installed packages are accessible from within the Cerebras container:

  • TensorFlow

    singularity exec -B ~/Cerebras-1.6.1-extra-python-packages:$HOME/.local /<path>/cbcore-1.6.1-<identifier-string>.sif python -c 'import spektral; print(spektral.__file__)'

  • PyTorch

    singularity exec -B ~/Cerebras-1.6.1-extra-python-packages:$HOME/.local /<path>/cbcore-1.6.1-<identifier-string>.sif python-pt -c 'import spektral; print(spektral.__file__)'

The output should resemble the following, if the $HOME value is /my/home/username:

/my/home/username/.local/lib/python3.7/site-packages/spektral/__init__.py.