日本語版はこちら
Container virtualization is utilized in various systems, and HPC is no exception. For instance, some supercomputers in Japanese universities will introduce Docker, and AWS Batch is also a Docker-baed service.
While Docker is becomimg a standard of container virtualization, it has some issues to solve in order to use for HPC. For example, it is difficult to communicate among containers with MPI. In scientific simulation such as CFD, multiple processes are run in parallel with multiple servers (nodes) and CPUs, and then MPI is utilized. It is difficult to configure MPI to work among Docker containers, and this is a big hurdle to introduce container virtualization to HPC.
Then Singularity is released as a container focused on HPC. It has advantages as shown below.
- Integratable with MPI
- It is possible to convert Docker images to Singularity images. Investment to Docker images is effectively utilized even after you move to Singularity.
- Containers can be shared with Singularity Hub and private repositories.
In this article, Singularity is installed on an Amazon EC2 instance and used. As a software to execute in the container, OpenFOAM (OpenFOAM v4.1 for Ubuntu) is chosen. Firstly a public image on Singularity Hub is used, and a custom image is built after that. Some tips are shared at the end of the article. Excuse me in advance, but MPI run is out of coverage.
Let's Try
Configuration of the EC2 instance
I confirmed the OpenFOAM container can be run in the EC2 instance with the follownig configuration.
| Item | Configurated value |
|---|---|
| Region | ap-northeast-1(Tokyo) |
| AMI | Amazon Linux AMI 2017.09.1.20171120 x86_64 HVM GP2(ami-da9e2cbc) |
| Instance Type | t2.micro |
| EBS | 16GiB |
| Security Group | Allow SSH (TCP22) from my IP |
Installation of Singularity
Install required packages, and make directory.
$ sudo yum -y update
$ sudo yum -y groupinstall "Development Tools"
$ mkdir ~/git
$ cd ~/git
Install Singularity while referring to the Official Quick Start.
$ git clone https://github.com/singularityware/singularity.git
$ cd singularity
$ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install
Execute $ singularity --help. Please confirm Singularity is successfully installed by displaying Usage message like bellow.
$ singularity --help
USAGE: singularity [global options...] <command> [command options...] ...
GLOBAL OPTIONS:
-d|--debug Print debugging information
-h|--help Display usage summary
-s|--silent Only print errors
-q|--quiet Suppress all normal output
--version Show application version
-v|--verbose Increase verbosity +1
-x|--sh-debug Print shell wrapper debugging information
[...]
For any additional help or support visit the Singularity
website: http://singularity.lbl.gov/
Then, configure the following two:
(1) Install squashfs-tools to use simg format images.
$ sudo yum -y install squashfs-tools
(2) Change configuration while referring to Error running Singularity with sudo in this page.
$ sudo visudo
# Edit the line which begins with "Defaults secure_path = " as below.
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
That's all for preparation.
Utilization of Public Images
Let's execute a tutorial of OpenFOAM with a Singularity container.
I published a Singurality image of OpenFOAM on Singularity Hub. A command $ singularity run downloads images from Singularity Hub, and run as a container immediately. The OpenFOAM image is configured to execute simpleFoam by $ singularity run. The standard output of simpleFoam is shown by executing the following command.
(Although the output says "ERROR" because there is no congiguration files for simpleFoam, download and execution of the container are succeeded)
$ cd /tmp
$ singularity run shub://TakahisaShiratori/openfoam:4
Progress |===================================| 100.0%
/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : 4.1
Exec : simpleFoam
[...]
--> FOAM FATAL IO ERROR:
cannot find file
file: //tmp/system/controlDict at line 0.
From function regIOobject::readStream()
in file db/regIOobject/regIOobjectRead.C at line 72.
FOAM exiting
To execute commands other than simpleFoam, use $ singularity exec. For example blockMesh is executed by the follownig command.
$ singularity exec shub://TakahisaShiratori/openfoam:4 blockMesh
Progress |===================================| 100.0%
/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : 4.1
Exec : blockMesh
[...]
--> FOAM FATAL IO ERROR:
cannot find file
file: //tmp/system/controlDict at line 0.
From function regIOobject::readStream()
in file db/regIOobject/regIOobjectRead.C at line 72.
FOAM exiting
Save the image by the following command to reduce time to download from Singularity Hub.
$ singularity pull --name openfoam4.simg shub://TakahisaShiratori/openfoam:4
Once the image was saved, the container can be run by specifying .simg files instead of images which begin with shub://.
$ singularity exec openfoam4.simg blockMesh
Let's execute analysis with OpenFOAM. The following commands clone and run pitzDaily tutorial.
$ cd ~/git
$ git clone https://github.com/OpenFOAM/OpenFOAM-4.x.git
$ cd /tmp
$ cp -r ~/git/OpenFOAM-4.x/tutorials/incompressible/simpleFoam/pitzDaily /tmp/
$ cd /tmp/pitzDaily
$ ls
0 constant system
$ sudo singularity exec /tmp/openfoam4.simg blockMesh
$ sudo singularity exec /tmp/openfoam4.simg simpleFoam
$ ls
0 100 200 298 constant postProcessing system
Directories such as 100, 200 are created if the analysis by blockMesh and simpleFoam is succeeded. The result of the analysis is saved in these directories. In the figure below, the result is visualized by paraFoam in another EC2 instance.
Building Own Image
If you cannot find the image you want to use, or want a image of your in-house software, you can build your own image. There are two ways for the build: (1) manual execution of install commands with Shell, and (2) automated build with Recipe.
with Shell
Here I show how to log-in the shell. I don't mention about how to install OpenFOAM with shell because it was hard to install manually. You can log-in the shell of Ubuntu container, pulled from Docker, as below.
$ cd /tmp
$ cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2017.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.09"
PRETTY_NAME="Amazon Linux AMI 2017.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
$ sudo singularity build --writable ubuntu.img docker://ubuntu
$ sudo singularity shell --writable ubuntu.img
Singularity: Invoking an interactive shell within container...
Singularity ubuntu.img:/tmp> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
The first cat /etc/os-release replys information of host Amazon Linux, and the second one replys that of Ubuntu in the container. Now you can use the shell in the container. Please execute commands for the installation to create your image. Use $ exit to go back to the host.
Singularity ubuntu.img:/tmp> exit
with Recipe
You can build your image by describing the installation commands to Recipe (Here .def is used as its filename extension). The same image of OpemFOAM v4.1 as shub://TakahisaShiratori/openfoam:4 can be built by the following procedure. First, save this file as ubuntu1404_openfoam4.def.
$ cd /tmp
$ wget https://raw.githubusercontent.com/TakahisaShiratori/openfoam/master/Singularity.4 -O ubuntu1404_openfoam4.def
The folowing command builds the image accorning to the Recipe file.
$ sudo singularity build openfoam4_recipe.simg ubuntu1404_openfoam4.def
This command builds the same image as openfoam4.simg, and the built image is saves as the file named openfoam4_recipe.simg. Please edit the .def file by referring here to create your own image.
Things I failed
I failed the following three while configuring Singularity.
File sharing between hosts and containers
According to Working with Files part in this page, files in /home/$USER, /tmp and $PWD are shared by default. In this article, I used /tmp as working directory because it is independent on environment variables. This feature is useful because we don't have to mount a disk explicitly. Please be careful not to modify files in the host unintentionally.
Use of bashrc
Configurations of environment variables have to be described on ~/.bashrc after the installation of OpenFOAM. However, bashrc is not executed when Singularity containers are started to run. The environment variables should be substituted to $SINGULARITY_ENVIRONMENT as Line 16 of this recipe shows.
Containerization of OpenFOAM v5
I tried to build an image with this recipe to use the latest OpenFOAM(v5) but didn't work well. An image was built successfully, but run and exec were failed because commands of OpenFOAM cannot be called.
This problem is still under investigation.
Summary
In this article, I wrote a procedure to install Singularity, a container software for HPC, on Amazon EC2, and executed a tutorial of OpenFOAM. By utilizing Singularity, we can make software portable among various environments such as AWS, other cloud services, and on-premises. Execution with MPI, the main advantage of Singularity, is my future work.
Click here if you want to contact us.
