Viper supercomputer system at MPCDF

Login via gateway 

ssh <user>@gate.mpcdf.mpg.de

ssh <user>@viper.mpcdf.mpg.de

File systems

The file system /u (a symbolic link to /viper/u) is designed for permanent user data (source files, config files, etc.). The size of /u is 1.2 PB. Your home directory is in /u. The default disk quota in /u is 1.0 TB, and the file quota is 256K files. You can check your disk quota in /u with:

/usr/lpp/mmfs/bin/mmlsquota viper_u

The file system /ptmp (a symbolic link to /viper/ptmp) is designed for batch job I/O (12 PB, no system backups). Files in /ptmp that have not been accessed for more than 12 weeks will be removed automatically. The period of 12 weeks may be reduced if necessary, with prior notice.

As a current policy, no quotas are applied on /ptmp.

The /r file system (a symbolic link to /ghi/r) stages archive data. It is available only on the login nodes viper.mpcdf.mpg.de and on the interactive nodes viper-i.mpcdf.mpg.de.

Each user has a subdirectory /r/<initial>/<userid> to store data. For efficiency, files should be packed into tar files, with a size of about 1 GB to 1 TB, before archiving them in /r. Please avoid archiving many small files. When the file system /r becomes filled above a certain threshold, files will be transferred from disk to tape, beginning with the largest files that have not been used for the longest time.

Please do not use the file system /tmp or $TMPDIR for scratch data. Instead, use /ptmp, which is accessible from all Viper cluster nodes.

In cases where an application really depends on node-local storage, please use the directories from the environment variables JOB_TMPDIR and JOB_SHMTMPDIR, which are set individually for each Slurm job.

Compiling JOREK

Add the following lines to your .bashrc file:

module purge
module load intel/2024.0 impi/2021.11 mkl/2024.0
module load hdf5-mpi/1.14.1
module load fftw-mpi/3.3.10

export LD_LIBRARY_PATH=${FFTW3_ROOT}/lib:${HDF5_ROOT}/lib:${MKLROOT}/lib:${LD_LIBRARY_PATH}

Example Makefile.inc:

MODEL             = model303

FC                = mpiifx
CC                = mpiicx
CXX               = mpiicpx

FLAGS += -O2 -march=znver4
COMPILER_FAMILY   = intel

USE_PASTIX        = 0
USE_PASTIX_MURGE  = 0
USE_PASTIX6       = 0
USE_STRUMPACK     = 1
USE_MUMPS         = 0
USE_BLOCK         = 1
USE_HDF5          = 1
USE_FFTW          = 1
USE_MKL           = 0

LIBDIR = /u/ihol/lib/intel

ifeq (1, $(USE_STRUMPACK))
 MKLLIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 \
         -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64
 MKLINC = -I${MKLROOT}/include

 GKLIB_HOME = $(LIBDIR)/GKlib/install
 METIS_HOME = $(LIBDIR)/METIS/install
 PARMETIS_HOME = $(LIBDIR)/ParMETIS/install
 STRUMPACK_HOME=$(LIBDIR)/STRUMPACK-7.2.0/install

 STRUMPACKINC = -I$(STRUMPACK_HOME)/include
 STRUMPACKLIB = -L$(STRUMPACK_HOME)/lib64 -lstrumpack

 STRUMPACKINC += -I$(METIS_HOME)/include -I$(GKLIB_HOME)/include -I$(PARMETIS_HOME)/include
 STRUMPACKLIB += -L$(METIS_HOME)/lib -lmetis -L$(GKLIB_HOME)/lib -lGKlib -L$(PARMETIS_HOME)/lib -lparmetis

 STRUMPACKINC += $(MKLINC)
 STRUMPACKLIB += $(MKLLIB)

 DEFINES += -DNEWSPK
endif

# --- FFTW Library
LIBFFTW           = -L$(FFTW_HOME)/lib -lfftw3
INC_FFTW          = -I$(FFTW_HOME)/include

# --- HDF5 Library
HDF5INCLUDE = $(HDF5_HOME)/include
HDF5LIB     = -L$(HDF5_HOME)/lib -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lz

Running JOREK

Example batch job script:

#!/bin/bash -l
#SBATCH -D ./
#SBATCH -J JOREK
#SBATCH --nodes=1
#SBATCH --tasks-per-node=4
#SBATCH --cpus-per-task=32
#SBATCH --time=01:00:00

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export OMP_PLACES=cores

export EXECUTABLE="./jorek_model303"

srun ${EXECUTABLE} <inxflow |& tee logfile.out