Run JOREK on the TOK-I Cluster at IPP Garching

This page was recently updated after the update of the cluster to SLES15. If you find anything that is incorrect with the new system, please correct it.

ssh toki01.bc.rzg.mpg.de
...
ssh toki08.bc.rzg.mpg.de

Modules required 

module load git intel/2023.1.0.x impi/2021.9 mkl fftw-mpi gnuplot hdf5-mpi

Add to your ~/.bashrc or ~/.alias file.

module load impi-interactive

It is recommended not to add impi-interactive to your .bashrc, as this may stop simulations on the TOK cluster from working.

Compiling JOREK and Libraries 

module load pastix

Warning: There seems to be an issue when JOREK is run on clusters at IPP (TOK-I, TOK, DRACO, and COBRA) with more than one MPI task per harmonic. The run hangs at the first PASTIX call during the very first time-step. In such a situation, the combination of Pastix_5.2.3 and Scotch_5.1.12b has been seen to fix this issue.

The following Makefile.inc file should allow you to compile JOREK:

# JOREK configuration file for compilation on the TOK cluster at IPP Garching
# see also https://www.jorek.eu/wiki/doku.php?id=draco
 
MODEL             = model307
 
FC                = mpiifort
CC                = mpiicc
CXX               = mpiicpc
 
COMPILER_FAMILY   = intel
 
USE_PASTIX        = 1
USE_STRUMPACK     = 0
USE_MUMPS         = 1
USE_PASTIX_MURGE  = 0
USE_BLOCK         = 1
USE_HDF5          = 1
USE_FFTW          = 1
USE_MKL           = 0
 
 
LIBDIR = /tokp/work/ihol/bin/intel
 
ifeq (1, $(USE_MUMPS))
 MKLLIB += -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
         -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64
 MKLINC = -I${MKL_HOME}/include
 
 MUMPSDIR = $(LIBDIR)/MUMPS_5.2.1
 LIB_MUMPS = -L$(MUMPSDIR)/lib -ldmumps -lmumps_common -lpord
 INC_MUMPS = $(MUMPSDIR)/include
 
 LIB_MUMPS += $(MKLLIB)
 INC_MUMPS += $(MKLINC)
endif
 
 
ifeq (1, $(USE_PASTIX))
 MKLLIB += -L${MKL_HOME}/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
 MKLINC = -I${MKL_HOME}/include
 
 PASTIX_DIR = $(LIBDIR)/pastix_5.2.3/install
 SCOTCH_DIR = $(LIBDIR)/scotch_5.1.12
 
 LIB_PASTIX = -L$(PASTIX_DIR) -lpastix -L$(SCOTCH_DIR)/lib -lscotch -lscotcherrexit
 INC_PASTIX = -I$(PASTIX_DIR)/include
 
 INC_PASTIX += $(MKLINC)
 LIB_PASTIX += $(MKLLIB)
 
endif
 
ifeq (1, $(USE_STRUMPACK))
 MKLLIB = -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
         -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64
 MKLINC = -I${MKLROOT}/include
 
 METIS_HOME=$(LIBDIR)/metis_git/install
 PARMETIS_HOME=$(LIBDIR)/ParMETIS_git/install
 GKLIB_HOME=$(LIBDIR)/GKlib_git/install
 
 STRUMPACK_HOME=$(LIBDIR)/STRUMPACK-7.1.3/install
 
 STRUMPACKINC = -I$(STRUMPACK_HOME)/include
 STRUMPACKLIB = -L$(STRUMPACK_HOME)/lib64 -lstrumpack
 
 STRUMPACKINC += -I$(METIS_HOME)/include -I$(GKLIB_HOME)/include
 STRUMPACKLIB += -L$(METIS_HOME)/lib -lmetis -L$(GKLIB_HOME)/lib -lGKlib
 
 STRUMPACKINC += -I$(PARMETIS_HOME)/include
 STRUMPACKLIB += -L$(PARMETIS_HOME)/lib -lparmetis
 
 STRUMPACKINC += $(MKLINC)
 STRUMPACKLIB += $(MKLLIB)
 
 DEFINES += -DNEWSPK
endif
 
# --- FFTW Library
#LIBFFTW           = -Wl -rpath -L$(FFTW_HOME)/lib -lfftw3
LIBFFTW           = $(FFTW_HOME)/lib/libfftw3.a
INC_FFTW          = -I$(FFTW_HOME)/include
 
# --- HDF5 Library
HDF5INCLUDE = $(HDF5_HOME)/include
#HDF5LIB     = -Wl -rpath -L$(HDF5_HOME)/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lz
HDF5LIB     = $(HDF5_HOME)/lib/libhdf5hl_fortran.a  $(HDF5_HOME)/lib/libhdf5_hl.a  $(HDF5_HOME)/lib/libhdf5_fortran.a $(HDF5_HOME)/lib/libhdf5.a -L$(HDF5_HOME)/lib  -lz

File systems

  • Best is to use /tokp/work/<user>/ for your runs
  • Also possible:
    • /tokp/scratch/<user>/ — not backed up
    • /ptmp1/work/<user>/
    • /ptmp1/scratch/<user>/ — not backed up

Batch submission

Not available.

Interactive Runs

  • Check with top beforehand that the machine is not already heavily used by others
  • Do not run very long simulations
  • Since each node has 24 cores, a rule of thumb is that OpenMP threads × MPI tasks should not exceed 20
export OMP_NUM_THREADS=<number-of-OpenMP-threads>
nice mpirun -n <mpi-tasks> ./jorek_model199 < ./input | tee logfile

When running on TOK-I, if you get the error below, specify the location of the JOREK binary:

env: ‘jorek_model199’: No such file or directory

Without MPI

To run axisymmetric simulations, use only one MPI task:

mpirun -n 1 ./jorek_model199 < ./input | tee logfile

Since MPI is always included in JOREK, this is the closest one can get to running without MPI.

Interactive regression tests

To run regression tests interactively, set:

export MPIRUN="mpirun -n 1"

and then perform them in the usual way using:

./non_regression_tests/run_test.sh