We take the first step of downloading, compiling and testing NEMO4 code on ARCHER2.
Pre-requisites:
Suggested reading:
Installing NEMO
Get NEMO4, in my case I’m using a fork of Pierre’s custom build.
This uses the latest stable version as it’s base, namely: NEMO4.0.4.
svn co -r 13653 https://forge.ipsl.jussieu.fr/nemo/svn/NEMO/releases/r4.0/r4.0.4 NEMO4
#optional
#copy code from Pierre into NEMO folder, something like:
#git clone git@github.com:chrisb13/NEMOCFG.git
#cd NEMOCFG
#git reset --hard 7695557b1a2f67bae2ae0ab1989d878a281c1bab
#rm -r /path/to/NEMO4/src
#cp -r /path/from/NEMOCFG/cfgs/eORCA025.L121-OPM006/src /path/to/NEMO4Build NEMO
Log-into ARCHER2. Based on NEMO4 ARCHER2 help page. Set-up your environment with:
module -s restore /work/n01/shared/acc/n01_modules/ucx_envFrom:
/work/n01/shared/acc/arch-X86_ARCHER2-Cray.fcmput into your ARCH folder, i.e.,/lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/arch/arch-X86_ARCHER2-Cray.fcm
%NCDF_HOME $NETCDF_DIR
%HDF5_HOME $HDF5_DIR
%XIOS_HOME /work/n01/shared/acc/xios-2.5
%NCDF_INC -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_LIB -L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios
%CPP cpp
%FC ftn
%FCFLAGS -em -s integer32 -s real64 -O1 -hflex_mp=intolerant
%FFLAGS -em -s integer32 -s real64 -O1 -hflex_mp=intolerant
%LD CC -Wl,"--allow-multiple-definition"
%FPPFLAGS -P -traditional
%LDFLAGS -lmpifort_cray
%AR ar
%ARFLAGS -r
%MK gmake
%USER_INC %XIOS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %NCDF_LIB
%CC cc
%CFLAGS -O0Add -J flag to FC_MODSEARCH in /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/ext/FCM/lib/Fcm/Config.pm, should look like
FC_MODSEARCH => '-J', # FC flag, specify "module" pathAdd key_nosignedzero to /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY/cpp_ISOMIPY.fcm, should have:
chbull@uan01:/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY> cat cpp_ISOMIPY.fcm
bld::tool::fppkeys key_iomput key_mpp_mpi key_nosignedzeroTest compile works, i.e.,
cd /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4
./makenemo -n 'ISOMIPY' -a 'ISOMIP' -m X86_ARCHER2-Cray -j 16
ls -lah tests/ISOMIPY/BLD/bin/nemo.exe
# beer time ?Run NEMO
We use Andrew’s script to create a template for a run file.
chbull@uan01:/work/n01/shared/acc> ./mkslurm -S 0 -s 0 -C 24
Running: mkslurm -S 0 -s 0 -m 2 -C 24 -c 2 -t 00:10:00 -a n01 -j nemo_test
.
.With a few trivial changes (saving to something like /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY/EXP00/runnemo.sh; which will make sense in a moment)..
#!/bin/bash
#SBATCH --job-name=nemo_test
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=24
#SBATCH --account=n02
#SBATCH --partition=standard
#SBATCH --qos=short
# Created by: mkslurm -S 0 -s 0 -m 2 -C 24 -c 2 -t 00:10:00 -a n01 -j nemo_test
module -s restore /work/n01/shared/acc/n01_modules/ucx_env
export OMP_NUM_THREADS=1
#
cat > myscript_wrapper2.sh << EOFB
#!/bin/ksh
#
set -A map ./xios_server.exe ./nemo
exec_map=( 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 )
#
exec \${map[\${exec_map[\$SLURM_PROCID]}]}
##
EOFB
chmod u+x ./myscript_wrapper2.sh
#
srun --mem-bind=local --cpu-bind=v,map_cpu:00,0x2,0x4,0x6,0x8,0xa,0xc,0xe,0x10,0x12,0x14,0x16,0x18,0x1a,0x1c,0x1e,0x20,0x22,0x24,0x26,0x28,0x2a,0x2c,0x2e, ./myscript_wrapper2.shSubmit with
cd /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY/EXP00
ln -s /work/n01/shared/acc/xios-2.5 xios_server.exe
sbatch --reservation=shortqos /lus/cls01095/work/n02/n02/chbull/nemo/models/NEMO4/tests/ISOMIPY/EXP00/runnemo.sh
# second beer ?Build NEMO tools
Compile REBUILD_NEMO (used to recombine outputs at the end of the jobs – see NEMO4/tools/REBUILD_NEMO/README.rst) and DOMAINcfg.
cd NEMO4/tools/
./maketools -h
./maketools -n REBUILD_NEMO -m X86_ARCHER2-Cray
./maketools -n DOMAINcfg -m X86_ARCHER2-Cray
# to check that everything went fine
ls -lah REBUILD_NEMO/BLD/bin/rebuild_nemo.exe
ls -lah DOMAINcfg/BLD/bin/make_domain_cfg.exe