トップ > AWS > Running OpenFOAM on AWS with Singularity containers

Running OpenFOAM on AWS with Singularity containers

AWS
記事タイトルとURLをコピーする

日本語版はこちら

Container virtualization is utilized in various systems, and HPC is no exception. For instance, some supercomputers in Japanese universities will introduce Docker, and AWS Batch is also a Docker-baed service.

While Docker is becomimg a standard of container virtualization, it has some issues to solve in order to use for HPC. For example, it is difficult to communicate among containers with MPI. In scientific simulation such as CFD, multiple processes are run in parallel with multiple servers (nodes) and CPUs, and then MPI is utilized. It is difficult to configure MPI to work among Docker containers, and this is a big hurdle to introduce container virtualization to HPC.

Then Singularity is released as a container focused on HPC. It has advantages as shown below.

  • Integratable with MPI
  • It is possible to convert Docker images to Singularity images. Investment to Docker images is effectively utilized even after you move to Singularity.
  • Containers can be shared with Singularity Hub and private repositories.

In this article, Singularity is installed on an Amazon EC2 instance and used. As a software to execute in the container, OpenFOAM (OpenFOAM v4.1 for Ubuntu) is chosen. Firstly a public image on Singularity Hub is used, and a custom image is built after that. Some tips are shared at the end of the article. Excuse me in advance, but MPI run is out of coverage.

Let's Try

Configuration of the EC2 instance

I confirmed the OpenFOAM container can be run in the EC2 instance with the follownig configuration.

Item Configurated value
Region ap-northeast-1(Tokyo)
AMI Amazon Linux AMI 2017.09.1.20171120 x86_64 HVM GP2(ami-da9e2cbc)
Instance Type t2.micro
EBS 16GiB
Security Group Allow SSH (TCP22) from my IP

Installation of Singularity

Install required packages, and make directory.

$ sudo yum -y update
$ sudo yum -y groupinstall "Development Tools"
$ mkdir ~/git
$ cd ~/git

Install Singularity while referring to the Official Quick Start.

$ git clone https://github.com/singularityware/singularity.git
$ cd singularity
$ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

Execute $ singularity --help. Please confirm Singularity is successfully installed by displaying Usage message like bellow.

$ singularity --help
USAGE: singularity [global options...] <command> [command options...] ...

GLOBAL OPTIONS:
    -d|--debug    Print debugging information
    -h|--help     Display usage summary
    -s|--silent   Only print errors
    -q|--quiet    Suppress all normal output
       --version  Show application version
    -v|--verbose  Increase verbosity +1
    -x|--sh-debug Print shell wrapper debugging information

[...]

For any additional help or support visit the Singularity
website: http://singularity.lbl.gov/

Then, configure the following two:

(1) Install squashfs-tools to use simg format images.

$ sudo yum -y install squashfs-tools

(2) Change configuration while referring to Error running Singularity with sudo in this page.

$ sudo visudo
# Edit the line which begins with "Defaults    secure_path = " as below.
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

That's all for preparation.

Utilization of Public Images

Let's execute a tutorial of OpenFOAM with a Singularity container.

I published a Singurality image of OpenFOAM on Singularity Hub. A command $ singularity run downloads images from Singularity Hub, and run as a container immediately. The OpenFOAM image is configured to execute simpleFoam by $ singularity run. The standard output of simpleFoam is shown by executing the following command.
(Although the output says "ERROR" because there is no congiguration files for simpleFoam, download and execution of the container are succeeded)

$ cd /tmp
$ singularity run shub://TakahisaShiratori/openfoam:4
Progress |===================================| 100.0%
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.1                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 4.1
Exec   : simpleFoam

[...]

--> FOAM FATAL IO ERROR:
cannot find file

file: //tmp/system/controlDict at line 0.

    From function regIOobject::readStream()
    in file db/regIOobject/regIOobjectRead.C at line 72.

FOAM exiting

To execute commands other than simpleFoam, use $ singularity exec. For example blockMesh is executed by the follownig command.

$ singularity exec shub://TakahisaShiratori/openfoam:4 blockMesh
Progress |===================================| 100.0%
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.1                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 4.1
Exec   : blockMesh

[...]

--> FOAM FATAL IO ERROR:
cannot find file

file: //tmp/system/controlDict at line 0.

    From function regIOobject::readStream()
    in file db/regIOobject/regIOobjectRead.C at line 72.

FOAM exiting

Save the image by the following command to reduce time to download from Singularity Hub.

$ singularity pull --name openfoam4.simg shub://TakahisaShiratori/openfoam:4

Once the image was saved, the container can be run by specifying .simg files instead of images which begin with shub://.

$ singularity exec openfoam4.simg blockMesh

Let's execute analysis with OpenFOAM. The following commands clone and run pitzDaily tutorial.

$ cd ~/git
$ git clone https://github.com/OpenFOAM/OpenFOAM-4.x.git
$ cd /tmp
$ cp -r ~/git/OpenFOAM-4.x/tutorials/incompressible/simpleFoam/pitzDaily /tmp/
$ cd /tmp/pitzDaily
$ ls
0  constant  system
$ sudo singularity exec /tmp/openfoam4.simg blockMesh
$ sudo singularity exec /tmp/openfoam4.simg simpleFoam
$ ls
0  100  200  298  constant  postProcessing  system

Directories such as 100, 200 are created if the analysis by blockMesh and simpleFoam is succeeded. The result of the analysis is saved in these directories. In the figure below, the result is visualized by paraFoam in another EC2 instance.

Building Own Image

If you cannot find the image you want to use, or want a image of your in-house software, you can build your own image. There are two ways for the build: (1) manual execution of install commands with Shell, and (2) automated build with Recipe.

with Shell

Here I show how to log-in the shell. I don't mention about how to install OpenFOAM with shell because it was hard to install manually. You can log-in the shell of Ubuntu container, pulled from Docker, as below.

$ cd /tmp
$ cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2017.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.09"
PRETTY_NAME="Amazon Linux AMI 2017.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
$ sudo singularity build --writable ubuntu.img docker://ubuntu
$ sudo singularity shell --writable ubuntu.img
Singularity: Invoking an interactive shell within container...

Singularity ubuntu.img:/tmp> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

The first cat /etc/os-release replys information of host Amazon Linux, and the second one replys that of Ubuntu in the container. Now you can use the shell in the container. Please execute commands for the installation to create your image. Use $ exit to go back to the host.

Singularity ubuntu.img:/tmp> exit

with Recipe

You can build your image by describing the installation commands to Recipe (Here .def is used as its filename extension). The same image of OpemFOAM v4.1 as shub://TakahisaShiratori/openfoam:4 can be built by the following procedure. First, save this file as ubuntu1404_openfoam4.def.

$ cd /tmp
$ wget https://raw.githubusercontent.com/TakahisaShiratori/openfoam/master/Singularity.4 -O ubuntu1404_openfoam4.def

The folowing command builds the image accorning to the Recipe file.

$ sudo singularity build openfoam4_recipe.simg ubuntu1404_openfoam4.def

This command builds the same image as openfoam4.simg, and the built image is saves as the file named openfoam4_recipe.simg. Please edit the .def file by referring here to create your own image.

Things I failed

I failed the following three while configuring Singularity.

File sharing between hosts and containers

According to Working with Files part in this page, files in /home/$USER, /tmp and $PWD are shared by default. In this article, I used /tmp as working directory because it is independent on environment variables. This feature is useful because we don't have to mount a disk explicitly. Please be careful not to modify files in the host unintentionally.

Use of bashrc

Configurations of environment variables have to be described on ~/.bashrc after the installation of OpenFOAM. However, bashrc is not executed when Singularity containers are started to run. The environment variables should be substituted to $SINGULARITY_ENVIRONMENT as Line 16 of this recipe shows.

Containerization of OpenFOAM v5

I tried to build an image with this recipe to use the latest OpenFOAM(v5) but didn't work well. An image was built successfully, but run and exec were failed because commands of OpenFOAM cannot be called. This problem is still under investigation.

Summary

In this article, I wrote a procedure to install Singularity, a container software for HPC, on Amazon EC2, and executed a tutorial of OpenFOAM. By utilizing Singularity, we can make software portable among various environments such as AWS, other cloud services, and on-premises. Execution with MPI, the main advantage of Singularity, is my future work.

Click here if you want to contact us.