Apptainer

In Summer 2024, the CS department has made available Apptainer software (formally named Singularity) for containers in the environment. This is available as a software module and can be used throughout the environment (including on SLURM nodes).

There are also three general purpose servers, apptainer[01-03], that may be used for building and running containers. These servers are accessible via SSH.

Apptainer is available as a module. To load and use it simply execute the following

~$ module load apptainer

By default for Apptainer containers, mountpoints such as home, project, and bigtemp directories are available along with certain system directories. This means that changes to files you make when using an image will persist outside of the image as well.

System directories examples (local to the server):

• /tmp
• /proc

Mounted Directories (remote storage):

• /u
• /l
• /s
• /p
• /bigtemp

To disable such directories from becoming available within a container, add the flag -c Apptainer commands such as exec or run.

Apptainer provides built-in compatibility for Docker container images. For example, container images from Docker Hub can often be ran and built directly using Apptainer without modification. In such instances, Apptainer will automatically convert the container image to an Apptainer SIF image when using the build command

~$ apptainer build mylolcow.sif docker://ghcr.io/apptainer/lolcow

Further, Docker Hub images can also be run directly

~$ apptainer run docker://sylabsio/lolcow:latest

Dockerfiles can be rewritten to a syntactically similar Apptainer Definition file. For comparisons between the two, see the Apptainer page for Apptainer Definition file vs. Dockerfile.

At times, if a Docker image has a user with a /home/username, this may not function as expected since Apptainer containers run as a user process. In such instances, it's recommended to reconfigure the Dockerfile (or Apptainer def file) to install software to /opt within the container.

More information on Apptainer's support for Docker and OCI containers can be found here.

Apptainer image files are named with the suffix Singularity Image Format (.sif). To create your own image, an Apptainer Definition file must be written first. A full guide on def files can be found on the official Apptainer website.

Once a def file has been created, the following can be done (replace <imagename> with the name of the resulting SIF image, and <defname> with the name of your def file)

~$ apptainer build <imagename>.sif <defname>.def

Apptainer Definition files are similar to a Dockerfile, and comparisons can be found here.

Note, when building images, be sure to check the file size. By default, Apptainer will create the SIF file in the current directory where the command was executed. For large images, it's recommended to use project directories for persistent storage or bigtemp for temporary image storage.

The following example Apptainer Def File provides a baseline structure to modify as needed.

Bootstrap: docker
From: ubuntu:{{ UBUNTU_VER }}

%arguments
    UBUNTU_VER=22.04

%files
    /u/userid/file0 /opt/file0

%environment
    export MY_ENV_VAR=/opt/my_custom_dir/bin

%post
    apt -y update && apt install -y python3
    mkdir /opt/my_custom_dir

%runscript
    echo "Arguments received: $*"

%startscript
    ...

%test
    if ! command -v python3 2>&1 > /dev/null; then
        echo "ERROR: command 'python3' not found!"
        exit 1
    fi
    
    echo "INFO: All tests passed"

%labels
    Author myusername
    Version v0.0.1

%help
    General information about running the container can go here

• Bootstrap: and From: are headers that must always be included at the start of a def file
  • Common bootstraps
    • docker
    • arch
  • Further details on these headers can be found (here)

• %arguments
  • This section defines variables that are accessible at build time
  • Variables can be accessed by using the notation {{ variable_name }}

• %files
  • This section defines copying files into the container from the current working directory, occurring before the post section
  • General form for copying files is <src> [<dest>]
  • <src> is either a path on the system, or a path from a previous stage of the build
  • <dest> is always a path into the current container

• %environment
  • This section defines environment variables that are set at runtime. This can be helpful if you compile software into a directory within the container

• %post
  • This section executes during the build process. This is where you would install necessary software packages, download from sources such as git, create directories, etc.

• %runscript
  • This section executes when a container is run using one of the following commands
    • ~$ apptainer run container_name.sif
    • ~$ apptainer exec container_name.sif
  • Parameters can be passed to this section. For example, ~$ apptainer run container_name.sif arg0 arg1

• %startscript
  • This section executes when a container is started using the command ~$ apptainer instance start container_name.sif

• %test
  • This section executes at the end of the build process. This can be used to validate compilation of software, or the presence of commands as shown in the example

• %labels
  • Simple labels to associate to a container. This section can be viewed by executing the command ~$ apptainer inspect container_name.sif after the container has been built

• %help
  • This section can be viewed by executing the command ~$ apptainer run-help container_name.sif after the container has been built

For further reading, please see the official documentation for Apptainer Def Files

At times, it can be difficult to predict all commands necessary to fully configure an image at build time. A sandbox can be used to interactively build a container. However, a sandbox should be used to create a complete list of build commands necessary to transition to a def file for a complete image build.

To create an Apptainer Sandbox Directory

~$ apptainer build --sandbox /u/userid/apptainer_sandbox docker://ubuntu

Then, you can use commands such as shell or run

~$ apptainer shell /u/userid/apptainer_sandbox

Further, the --writable can also be used to write to files within /u/userid/apptainer_sandbox

~$ apptainer shell --writable /u/userid/apptainer_sandbox

Note, if you need to run apt install ... commands within the sandbox, be sure to use the --fakeroot option

~$ apptainer shell --fakeroot --writable /u/userid/apptainer_sandbox

For errors such as the following are encountered when using --writable

FATAL:   container creation failed: mount hook function failure: ... destination /u doesn't exist in containe

Be sure to make the directory within the sandbox

~$ mkdir /u/userid/apptainer_sandbox/u

At times, some complex builds require CLI input that can't be automatically passed with environment variables or by setting DEBIAN_FRONTEND=noninteractive. In such cases, you can build an SIF image using a sandbox with

apptainer build <imagename>.sif /u/userid/apptainer_sandbox

To open an interactive shell within a container

~$ apptainer shell <imagename>.sif
Apptainer>

Note, after opening a shell, commands such as ls are available. As mentioned previously, home directories are available by default within an image

Apptainer> ls -al

To run commands within a given container, for example when a container is made for running a python program

~$ apptainer exec <imagename>.sif python3 mypythonprogram.py 

When an image is created from a def file, a runscript can also be defined to turn the image into more of an executable program. Read more about runscripts here.

~$ apptainer run <imagename>.sif
or
~$ ./<imagename>.sif

When using a GPU image, the host system's GPU drivers can be loaded into an image by including the flag --nv when running an Apptainer command

For example,

~$ apptainer exec --nv <imagename>.sif python3 mygpuprogam.py 

Be sure to reference the CS wiki page about slurm for additional information here.

After initializing an interactive job in slurm

~$ module purge
~$ module load apptainer
~$ apptainer shell /bigtemp/example/myimage.sif
Apptainer>

Below is a simple example of a possible sbatch script for running an image on a GPU system

#!/bin/bash

#SBATCH --mem=4000
#SBATCH --gpus=1
#SBATCH -t 02:00:00
#SBATCH -p gpu

module purge
module load apptainer
apptainer exec --nv myimage.sif python3 mygpuprogam.py