Adastra (CINES)
The Adastra cluster is located at CINES (France). Each node contains 4 AMD MI250X GPUs, each with 2 Graphics Compute Dies (GCDs) for a total of 8 GCDs per node. You can think of the 8 GCDs as 8 separate GPUs, each having 64 GB of high-bandwidth memory (HBM2E).
Introduction
If you are new to this system, please see the following resources:
Batch system: Slurm
-
$SHAREDSCRATCHDIR: meant for short-term data storage, shared with all members of a project, purged every 30 days (17.6 TB default quota)$SCRATCHDIR: meant for short-term data storage, single user, purged every 30 days$SHAREDWORKDIR: meant for mid-term data storage, shared with all members of a project, never purged (4.76 TB default quota)$WORKDIR: meant for mid-term data storage, single user, never purged$STORE: meant for long term storage, single user, never purged, backed up$SHAREDHOMEDIR: meant for scripts and tools, shared with all members of a project, never purged, backed up$HOME: meant for scripts and tools, single user, never purged, backed up
Preparation
The following instructions will install WarpX in the $SHAREDHOMEDIR directory,
which is shared among all the members of a given project. Due to the inode
quota enforced for this machine, a shared installation of WarpX is advised.
Use the following commands to download the WarpX source code:
# If you have multiple projects, activate the project that you want to use with:
#
# myproject -a YOUR_PROJECT_NAME
#
git clone https://github.com/BLAST-WarpX/warpx.git $SHAREDHOMEDIR/src/warpx
We use system software modules, add environment hints and further dependencies via the file $SHAREDHOMEDIR/adastra_warpx.profile.
Create it now:
cp $SHAREDHOMEDIR/src/warpx/Tools/machines/adastra-cines/adastra_warpx.profile.example $SHAREDHOMEDIR/adastra_warpx.profile
Edit the 2nd line of this script, which sets the export proj="" variable using a text editor
such as nano, emacs, or vim (all available by default on Adastra login nodes) and
uncomment the 3rd line (which sets $proj as the active project).
Important
Now, and as the first step on future logins to Adastra, activate these environment settings:
source $SHAREDHOMEDIR/adastra_warpx.profile
Finally, since Adastra does not yet provide software modules for some of our dependencies, install them once:
bash $SHAREDHOMEDIR/src/warpx/Tools/machines/adastra-cines/install_dependencies.sh
source $SHAREDHOMEDIR/sw/adastra/gpu/venvs/warpx-adastra/bin/activate
Compilation
Use the following cmake commands to compile the application executable:
cd $SHAREDHOMEDIR/src/warpx
rm -rf build_adastra
cmake -S . -B build_adastra -DWarpX_COMPUTE=HIP -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_adastra -j 16
The WarpX application executables are now in $SHAREDHOMEDIR/src/warpx/build_adastra/bin/.
Additionally, the following commands will install WarpX as a Python module:
rm -rf build_adastra_py
cmake -S . -B build_adastra_py -DWarpX_COMPUTE=HIP -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_adastra_py -j 16 --target pip_install
Now, you can submit Adstra compute jobs for WarpX Python (PICMI) scripts (example scripts). Or, you can use the WarpX executables to submit Adastra jobs (example inputs). For executables, you can reference their location in your job script .
Update WarpX & Dependencies
If you already installed WarpX in the past and want to update it, start by getting the latest source code:
cd $SHAREDHOMEDIR/src/warpx
# read the output of this command - does it look ok?
git status
# get the latest WarpX source code
git fetch
git pull
# read the output of these commands - do they look ok?
git status
git log # press q to exit
And, if needed,
log out and into the system, activate the now updated environment profile as usual,
As a last step, clean the build directory rm -rf $HOME/src/warpx/build_adastra and rebuild WarpX.
Running
MI250X GPUs (2x64 GB)
In non-interactive runs:
#!/bin/bash
#SBATCH --account=<account_to_charge>
#SBATCH --job-name=warpx
#SBATCH --constraint=MI250
#SBATCH --nodes=2
#SBATCH --exclusive
#SBATCH --output=%x-%j.out
#SBATCH --time=00:10:00
module purge
# A CrayPE environment version
module load cpe/24.07
# An architecture
module load craype-accel-amd-gfx90a craype-x86-trento
# A compiler to target the architecture
module load PrgEnv-cray
# Some architecture related libraries and tools
module load develop
module load CCE-GPU-4.0.0
# AMD related libraries
module load rocm/6.1.2
module load amd-mixed/6.1.2
# note
# cray-mpich versions 8.1.28 and 8.1.30 have known issues
# that cause node memory increase over time which leads
# to slowdown and out-of-memory crashes.
module load cray-mpich/8.1.26
date
module list
export MPICH_GPU_SUPPORT_ENABLED=1
# note
# this environment setting is currently needed to work-around a
# known issue with Libfabric
#export FI_MR_CACHE_MAX_COUNT=0 # libfabric disable caching
# or, less invasive:
export FI_MR_CACHE_MONITOR=memhooks # alternative cache monitor
# note
# On machines with similar architectures (Frontier, OLCF) these settings
# seem to prevent the following issue:
# OLCFDEV-1597: OFI Poll Failed UNDELIVERABLE Errors
# https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#olcfdev-1597-ofi-poll-failed-undeliverable-errors
export MPICH_SMP_SINGLE_COPY_MODE=NONE
export FI_CXI_RX_MATCH_MODE=software
# note
# this environment setting is needed to avoid that rocFFT writes a cache in
# the home directory, which does not scale.
export ROCFFT_RTC_CACHE_PATH=/dev/null
export OMP_NUM_THREADS=1
export WARPX_NMPI_PER_NODE=8
export TOTAL_NMPI=$(( ${SLURM_JOB_NUM_NODES} * ${WARPX_NMPI_PER_NODE} ))
srun -N${SLURM_JOB_NUM_NODES} -n${TOTAL_NMPI} --ntasks-per-node=${WARPX_NMPI_PER_NODE} \
--cpus-per-task=8 --threads-per-core=1 --gpu-bind=closest \
./warpx inputs > output.txt
Post-Processing
Note
TODO: Document any Jupyter or data services.
Known System Issues
Warning
May 16th, 2022: There is a caching bug in Libfabric that causes WarpX simulations to occasionally hang on on more than 1 node.
As a work-around, please export the following environment variable in your job scripts until the issue is fixed:
#export FI_MR_CACHE_MAX_COUNT=0 # libfabric disable caching
# or, less invasive:
export FI_MR_CACHE_MONITOR=memhooks # alternative cache monitor
Warning
Sep 2nd, 2022: rocFFT in ROCm 5.1-5.3 tries to write to a cache in the home area by default. This does not scale, disable it via:
export ROCFFT_RTC_CACHE_PATH=/dev/null
Warning
January, 2023: We discovered a regression in AMD ROCm, leading to 2x slower current deposition (and other slowdowns) in ROCm 5.3 and 5.4. Reported to AMD and fixed for the next release of ROCm.
Stay with the ROCm 5.2 module to avoid.
Warning
April 30th, 2025:
We observed several issues that can cause WarpX simulations to hang or crash on releases 25.02 and 25.03.
Releases <=25.01 and >=25.04 are currently working.
Warning
August 2025:
We observed a heavy node memory increase over time when using module cray-mpich versions 8.1.28 and 8.1.30, which
causes simulations to slow down and eventually crash.
While no cray-mpich version >8.1.30 is available on Adastra, stay with version 8.1.26 to avoid this issue.