AMReX basics (excessively basic)

WarpX is built on the Adaptive Mesh Refinement (AMR) library AMReX. This section provides a very sporadic description of the main AMReX classes and concepts relevant for WarpX, that can serve as a reminder. Please read the AMReX basics doc page, of which this section is largely inspired.

  • amrex::Box: Dimension-dependent lower and upper indices defining a rectangular volume in 3D (or surface in 2D) in the index space. Box is a lightweight meta-data class, with useful member functions.

  • amrex::BoxArray: Collection of Box on a single AMR level. The information of which MPI rank owns which Box in a BoxArray is in DistributionMapping.

  • amrex::FArrayBox: Fortran-ordered array of floating-point amrex::Real elements defined on a Box. A FArrayBox can represent scalar data or vector data, with ncomp components.

  • amrex::MultiFab: Collection of FAB (= FArrayBox) on a single AMR level, distributed over MPI ranks. The concept of ghost cells is defined at the MultiFab level.

  • amrex::ParticleContainer: A collection of particles, typically for particles of a physical species. Particles in a ParticleContainer are organized per Box. Particles in a Box are organized per tile (this feature is off when running on GPU). Particles within a tile are stored in several structures, each being contiguous in memory: (i) a Struct-of-Array (SoA) for amrex::ParticleReal data such as positions, weight, momentum, etc., (ii) a Struct-of-Array (SoA) for int data, such as ionization levels, and (iii) a Struct-of-Array (SoA) for a uint64_t unique identifier index per particle (containing a 40bit id and 24bit cpu sub-identifier as assigned at particle creation time). This id is also used to check if a particle is active/valid or marked for removal.

The simulation domain is decomposed in several Box, and each MPI rank owns (and performs operations on) the fields and particles defined on a few of these Box, but has the metadata of all of them. For convenience, AMReX provides iterators, to easily iterate over all FArrayBox (or even tile-by-tile, optionally) in a MultiFab own by the MPI rank (MFIter), or over all particles in a ParticleContainer on a per-box basis (ParIter, or its derived class WarpXParIter). These are respectively done in loops like:

// mf is a pointer to MultiFab
for ( amrex::MFIter mfi(mf, false); mfi.isValid(); ++mfi ) { ... }

and

// *this is a pointer to a ParticleContainer
for (WarpXParIter pti(*this, lev); pti.isValid(); ++pti) { ... }

When looping over FArrayBox in a MultiFab, the iterator provides functions to retrieve the metadata of the Box on which the FAB is defined (MFIter::box(), MFIter::tilebox() or variations) or the particles defined on this Box (ParIter::GetParticles()).