Running OpenFOAM on AWS with Singularity containers

日本語版はこちら

Container virtualization is utilized in various systems, and HPC is no exception. For instance, some supercomputers in Japanese universities will introduce Docker, and AWS Batch is also a Docker-baed service.

While Docker is becomimg a standard of container virtualization, it has some issues to solve in order to use for HPC. For example, it is difficult to communicate among containers with MPI. In scientific simulation such as CFD, multiple processes are run in parallel with multiple servers (nodes) and CPUs, and then MPI is utilized. It is difficult to configure MPI to work among Docker containers, and this is a big hurdle to introduce container virtualization to HPC.

Then Singularity is released as a container focused on HPC. It has advantages as shown below.
  • Integratable with MPI
  • It is possible to convert Docker images to Singularity images. Investment to Docker images is effectively utilized even after you move to Singularity.
  • Containers can be shared with Singularity Hub and private repositories.
In this article, Singularity is installed on an Amazon EC2 instance and used. As a software to execute in the container, OpenFOAM (OpenFOAM v4.1 for Ubuntu) is chosen. Firstly a public image on Singularity Hub is used, and a custom image is built after that. Some tips are shared at the end of the article. Excuse me in advance, but MPI run is out of coverage.

Let's Try

Configuration of the EC2 instance

I confirmed the OpenFOAM container can be run in the EC2 instance with the follownig configuration.
Item Configurated value
Region ap-northeast-1(Tokyo)
AMI Amazon Linux AMI 2017.09.1.20171120 x86_64 HVM GP2(ami-da9e2cbc)
Instance Type t2.micro
EBS 16GiB
Security Group Allow SSH (TCP22) from my IP

Installation of Singularity

Install required packages, and make directory. Install Singularity while referring to the Official Quick Start. Execute $ singularity --help. Please confirm Singularity is successfully installed by displaying Usage message like bellow. Then, configure the following two:

(1) Install squashfs-tools to use simg format images. (2) Change configuration while referring to Error running Singularity with sudo in this page. That's all for preparation.

Utilization of Public Images

Let's execute a tutorial of OpenFOAM with a Singularity container.

I published a Singurality image of OpenFOAM on Singularity Hub. A command $ singularity run downloads images from Singularity Hub, and run as a container immediately. The OpenFOAM image is configured to execute simpleFoam by $ singularity run. The standard output of simpleFoam is shown by executing the following command.
(Although the output says "ERROR" because there is no congiguration files for simpleFoam, download and execution of the container are succeeded) To execute commands other than simpleFoam, use $ singularity exec. For example blockMesh is executed by the follownig command. Save the image by the following command to reduce time to download from Singularity Hub. Once the image was saved, the container can be run by specifying .simg files instead of images which begin with shub://. Let's execute analysis with OpenFOAM. The following commands clone and run pitzDaily tutorial. Directories such as 100, 200 are created if the analysis by blockMesh and simpleFoam is succeeded. The result of the analysis is saved in these directories. In the figure below, the result is visualized by paraFoam in another EC2 instance.

Building Own Image

If you cannot find the image you want to use, or want a image of your in-house software, you can build your own image. There are two ways for the build: (1) manual execution of install commands with Shell, and (2) automated build with Recipe.

with Shell

Here I show how to log-in the shell. I don't mention about how to install OpenFOAM with shell because it was hard to install manually. You can log-in the shell of Ubuntu container, pulled from Docker, as below. The first cat /etc/os-release replys information of host Amazon Linux, and the second one replys that of Ubuntu in the container. Now you can use the shell in the container. Please execute commands for the installation to create your image. Use $ exit to go back to the host.

with Recipe

You can build your image by describing the installation commands to Recipe (Here .def is used as its filename extension). The same image of OpemFOAM v4.1 as shub://TakahisaShiratori/openfoam:4 can be built by the following procedure. First, save this file as ubuntu1404_openfoam4.def. The folowing command builds the image accorning to the Recipe file. This command builds the same image as openfoam4.simg, and the built image is saves as the file named openfoam4_recipe.simg. Please edit the .def file by referring here to create your own image.

Things I failed

I failed the following three while configuring Singularity.

File sharing between hosts and containers

According to Working with Files part in this page, files in /home/$USER, /tmp and $PWD are shared by default. In this article, I used /tmp as working directory because it is independent on environment variables. This feature is useful because we don't have to mount a disk explicitly. Please be careful not to modify files in the host unintentionally.

Use of bashrc

Configurations of environment variables have to be described on ~/.bashrc after the installation of OpenFOAM. However, bashrc is not executed when Singularity containers are started to run. The environment variables should be substituted to $SINGULARITY_ENVIRONMENT as Line 16 of this recipe shows.

Containerization of OpenFOAM v5

I tried to build an image with this recipe to use the latest OpenFOAM(v5) but didn't work well. An image was built successfully, but run and exec were failed because commands of OpenFOAM cannot be called. This problem is still under investigation.

Summary

In this article, I wrote a procedure to install Singularity, a container software for HPC, on Amazon EC2, and executed a tutorial of OpenFOAM. By utilizing Singularity, we can make software portable among various environments such as AWS, other cloud services, and on-premises. Execution with MPI, the main advantage of Singularity, is my future work.

Click here if you want to contact us.

AWS運用自動化サービス「Cloud Automator」無料トライアルはこちらから

COMMENT ON FACEBOOK