Installation¶

First step to install nextSIM-DG is to download this repository :

git clone https://github.com/nextsimhub/nextsimdg.git

You will get the main branch of the code, if you need a specific version :

git clone -b v1.0 https://github.com/nextsimhub/nextsimdg.git

It may be easier to use either the docker file (see below), or the spack installation instructions.

Dependencies¶

To compile nextSIM-DG, you need to install first some libraries :

`NetCDF`_

`Boost`_

`CMake`_

Installing dependencies on on MAC OS

If your package manager is `Homebrew`_ :

brew install netcdf
brew install netcdf-cxx
brew install boost
brew install cmake
brew install eigen

Installing dependencies on a Debian-based Linux distribution (Debian, Ubuntu, etc)

You must have root privilege :

sudo apt-get update
sudo apt-get install netcdf-bin libnetcdf-c++4-dev libboost-all-dev cmake subversion libeigen3-dev
svn checkout http://forge.ipsl.fr/ioserver/svn/XIOS3/trunk xios
cd xios
./make_xios --arch <your_architecture> --job <number_of_jobs>

There is also a --debug option to compile in debug mode. For example, to compile XIOS in debug mode on a Linux operating system and the GCC compiler using 8 parallel jobs, use:

./make_xios --arch GCC_LINUX --job 8 --debug

Installing dependencies via conda

Install conda via anaconda or miniconda (no root privileges required)

conda create --name nextsimdg
conda activate nextsimdg
conda install -c conda-forge netcdf-cxx4
conda install -c conda-forge eigen
conda install -c conda-forge boost
conda install -c anaconda cmake

Installing dependencies via spack

spack is a package manager which was designed for HPC software. It is language agnostic and compiles binaries from source. To install spack please follow instruction from their documentation (see install instructions).

In the root directory of the repository there is a spack.yaml file which can be used to install all required dependencies. The following command will create a spack environment (called nextsim in this example), load the spack environment and install the dependencies:

spack env create nextsim spack.yaml
spack env activate nextsim
spack install

Note

This will install by building from source. This can take a significant amount of time, depending on packages available on your machine and how your spack install is configured. If you use a clean install of spack this could take 15-50 minutes on a modern CPU with 4-8 cores. It is hard to give an accurate measurement but the idea is to be patient. This only needs to be done once however, and then it can be loaded in seconds whenever you need it.

To use the installed dependencies, you will need to load them first. If you are in the same bash session as the install they will be already loaded, but in future you can use the following command to activate the environment.

spack env activate nextsim

To unload an environment you can use the handy (and very fun) alias of despacktivate. For more information on using spack environments please see using environments.

Python dependencies¶

To install the Python dependencies, first ensure that you have a Python environment set up. You can use conda or virtualenv for this purpose. Once you have created and activated your Python environment, you can install the required Python packages with .. code:

pip install -r requirements.txt

Building domain_decomp¶

NextSIM-DG uses the domain_decomp library for generating domain decompositions for MPI parallel simulations. This library is included in the nextsimdg repository as a submodule, so you need to initialize and update the submodule before building it. You can do this with the following commands:

cd nextsimdg
git submodule init
git submodule update
cd domain_decomp
cmake -Bbuild -S.
cmake --build build --config Release

The domain_decomp library is not required to build the nextSIM-DG model but it is required for running the tests and for generating domain decompositions for application case studies. See the domain_decomp repo for further details on how to build and use the library.

Building nextSIM-DG¶

After all dependencies have been installed, we can build the code:

cd nextsimdg
mkdir -p build
cd build
cmake ..
make

Configuring the dynamics¶

The dynamics for nextSIM-DG are chosen at the point of configuring CMake. This is in contrast to most of the model configuration, which is done at model run time. The dynamics are set through the configuration option DynamicsType. The available options for the dynamics are

DG1: First order discontinuous Galerkin dynamics on a 2D rectangular grid. Advection calculations are performed with 3 DG components.
- This is the default option if no other option is provided to CMake.
DG2: Second order discontinuous Galerkin dynamics on a 2D rectangular grid. Advection calculations are performed with 6 DG components.

The syntax for chosing the dynamics via CMake is the standard method of providing options to CMake. For example, to compile the model with second order discontinuous Galerkin dynamics (DG2), the CMake command line with the dynamics argument would be

cmake -DDynamicsType=DG2 ..

Dependencies and Build for MPI Parallelisation¶

To build the code with MPI support, we need to install the respective compiler as well as parallel NetCDF support.

For example, on Debian-based Linux we need to also do:

sudo apt-get install libnetcdf-mpi-dev
sudo apt-get install openmpi-bin libopenmpi-dev

The cmake call has to enable MPI support:

cmake .. -DENABLE_MPI=ON

You might need to tell cmake which compiler to use, e.g.

cmake .. -DCMAKE_CXX_COMPILER=/usr/bin/mpicxx -DENABLE_MPI=ON

Dependencies and Build for GPU¶

The GPU version of nextSIM-DG uses Kokkos to support different devices. In principle, every target that Kokkos supports should work for nextSIM-DG as well. However, development and testing is mostly done with the CUDA backend, so expect more friction when targeting non NVIDIA GPUs. The library can be acquired through a package manager, e.g.

sudo apt-get install kokkos

In case that no suitable version of Kokkos is available on the system, a copy of the library can be placed in lib/kokkos. To build with GPU support, it is necessary to enable both the feature itself and an appropriate kokkos backend:

cmake .. -DWITH_KOKKOS=ON -DKokkos_ENABLE_CUDA=ON -DWITH_THREADS=ON

While it is currently not possible to use Kokkos and MPI together, OpenMP (WITH_THREADS) can be enabled to run modules that have not been ported to the device in parallel on the CPU. The OpenMP backend of Kokkos can be enabled as well, however it will only be used if no device backend is enabled. Same as the device backends, it should be used in concert with the basic OpenMP parallelisation and may have different performance characteristics then WITH_THREADS=ON alone.

Using Dockerfiles for Development or Production Runs¶

In the Dockerfiles directory we provide two Dockerfile’s are provided.

Dockerfile.devenv - This is the Dockerfile used to build the development image (ghcr.io/nextsimhub/nextsimdg-dev-env), that is used in the GitHub CI.
Dockerfile.production - This Dockerfile is based off of the development image and it additionally installs nextsim so that you can run on any machine with docker installed.

Development Dockerfile¶

A development image is provided on the nextsimhub GitHub container registry because it is needed for the CI.

If in future, this needs to be replaced. Please see instructions on the GitHub website.

To build the docker image, please use these instructions:

docker build --file Dockerfile.devenv . -t ghcr.io/nextsimhub/nextsimdg-dev-env:latest

Note

The formatting of the image name is important. The format is ghcr.io/NAME_OF_REPOSITORY/NAME_OF_IMAGE:TAG

If you want to test or use the image locally, use the following command:

docker pull ghcr.io/nextsimhub/nextsimdg-dev-env:latest

Production Dockerfile¶

The production image is not stored on the nextsimhub GitHub container registry because it is not needed for the CI. Users of the code may be interested in building their own. The instructions are as follows:

docker build --file Dockerfiles/Dockerfile.production . -t nextsim-production:latest

This will build a local image of the nextsim code. The production Dockerfile supports additional build arguments (--build-arg that can be specified at build time). For example, to build with MPI enabled, using 4 processes to compile, use the following command,

docker build --file Dockerfiles/Dockerfile.production --build-arg mpi=ON --build-arg jobs=4 . -t nextsim-production:latest

For a full list of options, please see Dockerfile.production. By default MPI and xios options are disabled and the number of build jobs is 1.

Running Tests¶

After building the code, you can run the tests provided with the repository using ctest. This ensures that the installation and build process was successful.

To run all tests, navigate to the build directory and execute:

ctest

If you want to see more detailed output for each test, use the -V option:

ctest -V

If you want to run a subset of tests, you can use the -R option with a regular expression to match the test names. For example, to run only the tests related to XIOS, use:

ctest -R Xios

For more information on ctest options, you can refer to the official ctest documentation.

Building the Documentation (Locally)¶

If you would like to build the documentation locally, follow the instructions below.

A Dockerfile is provided in the Dockerfiles directory to build the documentation i.e., Dockerfile.sphinx.

To build the docker image run the following command from the root of the repository:

docker build --file Dockerfiles/Dockerfile.sphinx . -t nextsim-docs:latest

This should create a local docker image called nextsim-docs:latest.

To build the documentation, run the following command from the root of the repository:

docker run --rm -v $PWD:/docs nextsim-docs:latest

Optionally, you can specify the number of jobs to use for building the documentation. For example, to use 4 jobs, run:

docker run --rm -v $PWD:/docs nextsim-docs:latest 4

Finally, to view the built documentation, open the file docs/_build/html/index.html in your web browser e.g.,

xdg-open docs/_build/html/index.html  # Linux
open docs/_build/html/index.html      # MacOS
start docs\_build\html\index.html     # Windows