Raven system at MPCDF

Basic information

Documentation:

Compute nodes:

  • currently 360 nodes
  • 2×36 cores per node
  • memory per node: 240 GB

Accessing the cluster

First log into the gate machine:

ssh gate.mpcdf.mpg.de

From the gate machine:

ssh raven[01-02]i

File systems

AFS is accessible from the login nodes.

Both /u and /ptmp file systems are accessible from the login and compute nodes.

  • Your data can be stored under /u/<user>; a quota of 2.5 TB applies
    (check via '/usr/lpp/mmfs/bin/mmlsquota raven_u')
  • /ptmp/<user> should be used as temporary storage

No automatic backup is done for either file system.

Note that files on /ptmp/ are automatically deleted if they have not been touched for 12 weeks.

Compiling JOREK

  • Add the following lines to your .bashrc file:
module purge
module load intel/2025.1 impi/2021.15
module load mkl/2025.1
module load fftw-mpi
module load hdf5-mpi
module load hwloc/2.2

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${INTEL_HOME}/compiler/latest/linux/compiler/lib/intel64/
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MKLROOT}/lib/intel64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${HDF5_ROOT}/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${FFTW_HOME}/lib

  • In order to use pre-compiled external solver libraries, please send a request to ihor.holod@ipp.mpg.de indicating your Raven user name.

  • Example Makefile.inc
    (note that PaStiX and STRUMPACK should not be chosen simultaneously):

MODEL             = model303
FC                = mpiifx
CC                = mpiicx
CXX               = mpiicpx
 
FFLAGS += -O3 -mcmodel=medium
EXTRA_FLAGS += -lstdc++
COMPILER_FAMILY   = intel
# Never use STRUMPACK and PASTIX at the same time 
USE_PASTIX        = 0
USE_PASTIX_MURGE  = 0
USE_PASTIX6       = 0
USE_STRUMPACK     = 1
USE_MUMPS         = 1
USE_BLOCK         = 1
USE_HDF5          = 1
USE_FFTW          = 1
USE_MKL           = 0
USE_DIRECT_CONSTRUCTION=0
USE_BICGSTAB=0
LIBS += -liomp5 -pthread -ldl -lm
 
LIBDIR = /raven/u/ihol/lib/intel

ifeq (1, $(USE_PASTIX))

 MKLLIB += -L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
 MKLINC = -I${MKLROOT}/include
 PASTIX_DIR = $(LIBDIR)/pastix_5.2.3/install
 SCOTCH_DIR = $(LIBDIR)/scotch_5.1.12

 LIB_PASTIX = -L$(PASTIX_DIR) -lpastix -L$(SCOTCH_DIR)/lib -lscotch -lscotcherr
 INC_PASTIX = -I$(PASTIX_DIR)/include

 INC_PASTIX += $(MKLINC)
 LIB_PASTIX += $(MKLLIB)

endif

ifeq (1, $(USE_MUMPS))
# --- MUMPS
 MKLLIB += -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
         -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64
 MKLINC = -I${MKLROOT}/include
 MUMPS_DIR = $(LIBDIR)/MUMPS_5.4.1
 LIB_MUMPS = -L$(MUMPS_DIR)/lib -ldmumps -lmumps_common -lpord
 INC_MUMPS = $(MUMPS_DIR)/include

 LIB_MUMPS += $(MKLLIB)
 INC_MUMPS_EXTRA += $(MKLINC)

endif

ifeq (1, $(USE_STRUMPACK))
 MKLLIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
         -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64
 MKLINC = -I${MKLROOT}/include
 GKLIB_HOME = $(LIBDIR)/GKlib/install
 METIS_HOME = $(LIBDIR)/METIS/install
 PARMETIS_HOME = $(LIBDIR)/ParMETIS/install
 STRUMPACK_HOME=$(LIBDIR)/STRUMPACK-7.2.0/install

 STRUMPACKINC = -I$(STRUMPACK_HOME)/include
 STRUMPACKLIB = -L$(STRUMPACK_HOME)/lib64 -lstrumpack

 STRUMPACKINC += -I$(METIS_HOME)/include -I$(GKLIB_HOME)/include -I$(PARMETIS_HOME)/include
 STRUMPACKLIB += -L$(METIS_HOME)/lib -lmetis -L$(GKLIB_HOME)/lib -lGKlib -L$(PARMETIS_HOME)/lib -lparmetis

 STRUMPACKINC += $(MKLINC)
 STRUMPACKLIB += $(MKLLIB)

 DEFINES += -DNEWSPK
endif

# --- FFTW Library
LIBFFTW           = -L$(FFTW_HOME)/lib -lfftw3
INC_FFTW          = -I$(FFTW_HOME)/include

# --- HDF5 Library
HDF5INCLUDE = $(HDF5_HOME)/include
HDF5LIB     = -L$(HDF5_HOME)/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lz

Submitting jobs

Below is an example SLURM job script:

#!/bin/bash -l
#SBATCH -D ./
#SBATCH -J jorek
#SBATCH -C icelake
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=36
#SBATCH --time=00:10:00

source $HOME/.bashrc

export OMP_NUM_THREADS=36
export OMP_PLACES=cores

srun ./jorek_model303 < intear > logout.out

Running interactively

For running interactively on Raven, go to the interactive nodes by:

ssh raven-i

or

ssh raven[03-06]

There are limits on:

  • cores (<= 8)
  • memory (<= 256000M)
  • time (<= 2 hours)

(My jobs also started with more than the recommended number of cores, but the user guide warns against it.)

srun -n N_TASKS -p interactive --time=TIME_LESS_THAN_2HOURS --mem=MEMORY_LESS_THAN_32G ./EXECUTABLE

Example:

export OMP_NUM_THREAD=4
nice srun -n 2 -p interactive --time=00:30:00 --mem=16GB ./jorek_model600 < input_jorek | tee logfile

It can take a short time to allocate the resources.