Skip to content

Instantly share code, notes, and snippets.

@sayerhs

sayerhs/AMR-Wind-Intro.md

Last active July 7, 2021 04:19
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sayerhs/f7db3b238051342bb869f334405e6437 to your computer and use it in GitHub Desktop.
Save sayerhs/f7db3b238051342bb869f334405e6437 to your computer and use it in GitHub Desktop.
Download ZIP
AMR-Wind example
Raw
AMR-Wind-Intro.md

Useful links

Recommended procedure for building on NREL Eagle

  • Purge all modules, remove conda environment
# Clone source code
git clone --recurse-submodules https://github.com/exawind/amr-wind.git

# Make build directory and switch to build directory
cd amr-wind
mkdir build_gcc
cd build_gcc

# Link build script that is pre-configured for Eagle
ln -s /nopt/nrel/ecom/exawind/exawind/scripts/amr-wind-gcc.sh

# CMake configure step
./amr-wind-gcc.sh cmake

# Compile code (add `-j 36` if you are on a compute node)
./amr-wind-gcc.sh make

Successful build will have amr_wind and amr_wind_unit_tests executable in your build directory. Switch gcc to intel for Intel builds.

General time integration scheme logic

  • Regrid (if necessary)
  • Perform pre-timestep actions (physics pre-processing)
  • Advance timestep (see implementation)
    • Compute turbulence updates (e.g. effective viscosities)
    • Update source term
    • Estimate velocity at n + 1/2 by extrapolation (uses Taylor series expansion)
    • Compute fluxes on faces using velocity at n + 1/2 state
    • MAC Projection to ensure face velocities are divergence free (Poisson solve)
    • Populate linear system - explicit, implicit, or Crank-Nicholson form for diffusion term; source terms at n + 1/2 if possible
    • Advance scalars
    • Advance momentum (with updated source terms, e.g., Boussinesq buoyancy)
    • Nodal Projection (approximately enforce divergence-free condition on cell-centered velocities)
  • Perform post-timestep actions
  • Perform I/O

Tasks

  • Evaluation of numerical dissipation for wind-energy use cases
  • ABL with wind turbines
    • Generalize inflow reading capability
    • Testing/hardening of inflow/outflow capability
    • Source terms and "mean Boussinesq term" in inflow/outflow mode (with turbines)
    • Clean up ABL physics class, especially ABLStats behavior
    • Mesoscale integration
    • Parallel NetCDF I/O issues with Spectrum MPI/Summit
  • Actuator line
    • Standalone actuator line/disk capability (i.e., no OpenFAST)
    • Actuator Disk w/ OpenFAST
    • Better handling of OpenFAST input files
    • Cleaner regression test setup when OpenFAST is enabled (e.g., controller)
  • Immersed boundary
    • Is it viable approach for modeling complex terrain, sub-structure, etc.?
    • Geometry definition module
      • Ability to read different types of geometry definition files
      • Parallel immersed boundary holecutting - leverage TIOGA?
      • Mesh motion module
      • OpenFAST interface, needs to be lifted out of actuator to a higher level
  • Linear solvers
    • MLMG performance on GPUs
    • MLMG coarsening strategies for cell-centered solvers with overset
    • Hypre bottom solver performance with AMReX
    • Reuse MacProjector and NodalProjector (especially with hypre bottom-solve)
  • GPU performance
    • Segregated momentum solve
    • Mac/Nodal Poisson solves (GPU performance)
    • ABLStats
  • Utilities
    • Solution transfer/mapping utilities
    • Solution blending utilities (e.g., for turbine simulations)
    • Post-processing, post-processing, post-processing...
  • Code design
    • Increase unit-testing coverage
      • Several pieces missing coverage
      • Some tests merely test execution not correctness
    • Numerical scheme refactor
      • Different time-integration schemes
      • Ability to add/select different spatial schemes in user-selectable fashion
      • Modular time-integration code (replace incflo_advance)
    • Inter-module communication/registration interface
    • Known code-duplication issues
      • Level mask logic (could be centralized and cleaned up)
      • Field interpolation using particles
    • Other issues
      • FieldRepo with CPU/GPU field views
      • Cleaner interface to register/output derived fields
      • Registration mechanism for outputting debugging fields
    • CMake
      • Clean-up acceleration backend behavior
      • More functions for common tasks
      • Ability to turn off the build of AMR-Wind shared library in standalone builds
    • Documentation
Raw
amr_wind_example.cpp
void amr_wind_example(CFDSim& sim)
{
// Get an instance of the mesh
const auto& mesh = sim.mesh();
// Database holding the field information
auto& repo = sim.repo();
// Generic field declaration interface
auto& vel = repo.declare_field( // Create a new field
"velocity", // Name of the field
3, // number of components
3, // number of ghost cells
2, // Number of states
FieldLoc::CELL // Location of the field
);
// Declare cell-centered field
auto& density = repo.declare_cc_field("density", 1, 3);
// Declare nodal field
auto& pressure = repo.declare_nd_field("pressure", 1, 3);
// Accessing existing fields
auto& vel_copy = repo.get_field("velocity");
// High-level operations
vel.setVal(30.0, 0, 1); // Ux = 30.0
vel.setVal(20.0, 1, 1); // Uy = 20.0
vel.setVal(10.0, 2, 1); // Uz = 10.0
density.setVal(1.0);
// Setting all components
// vel.setVal(30.0);
// Also supports some BLAS-type operations
// density(i, j, k) = 3.0 * vel(i, j, k, 0) + density(i, j, k)
field_ops::saxpy(density, // Destination field (dst)
3.0, // Factor
vel, // Source field (src)
0, // Starting component from src
0, // Starting component in dst
1, // Number of components
3 // Number of ghosts to update
);
// Create a temporary scratch field for work
auto rhoU = repo.create_scratch_field(3);
const int nlevels = repo.num_active_levels();
for (int lev = 0; lev < nlevels; ++lev) {
// Mesh related info
const auto& dx = mesh.Geom(lev).CellSizeArray();
const auto& problo = mesh.Geom(lev).ProbLoArray();
const auto& probhi = mesh.Geom(lev).ProbHiArray();
const amrex::Real cellvol = dx[0] * dx[1] * dx[2];
const amrex::Real dom_height = probhi[2] - problo[2];
auto& vfab = vel(lev);
const auto& rhofab = density(lev);
auto& mdotfab = (*rhoU)(lev);
auto& pfab = pressure(lev);
for (amrex::MFIter mfi(vfab); mfi.isValid(); ++mfi) {
// Processing cell-centered data
const auto bx = mfi.tilebox();
const auto& varr = vfab.array(mfi);
const auto& rhoarr = rhofab.const_array(mfi);
const auto& mdotarr = mdotfab.array(mfi);
amrex::ParallelFor(
bx, [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept {
amrex::Real xco = problo[0] + (i + 0.5) * dx[0];
amrex::Real yco = problo[1] + (j + 0.5) * dx[1];
amrex::Real zco = problo[2] + (k + 0.5) * dx[2];
varr(i, j, k, 0) = 10.0 + zco * 5.0 / dom_height;
for (int n = 0; n < AMREX_SPACEDIM; ++n) {
mdotarr(i, j, k, n) =
rhoarr(i, j, k) * varr(i, j, k, n);
// Example of ghost cell usage
const amrex::Real rho_avg_z =
0.5 * (rhoarr(i, j, k - 1) + rhoarr(i, j, k));
}
});
// Processing node-centered data
const auto nbx = mfi.nodaltilebox();
const auto& parr = pfab.array(mfi);
amrex::ParallelFor(
nbx, [=] AMREX_GPU_DEVICE(int i, int j, int k) noexcept {
amrex::Real xco = problo[0] + i * dx[0];
amrex::Real yco = problo[1] + j * dx[1];
amrex::Real zco = problo[2] + k * dx[2];
parr(i, j, k) = 10.0 * zco;
});
}
}
// Fillpatch ghost cells
vel.fillpatch(sim.time().current_time());
pressure.fillpatch(sim.time().current_time());
// Compute divergence/laplacian etc. (see turbulence models)
auto div_vel = fvm::divergence(vel);
auto lap_vel = fvm::laplacian(vel);
auto vorticity_mag = fvm::vorticity_mag(vel);
}
Raw
sample-input.txt
time.stop_time = 22000.0 # Max (simulated) time to evolve
time.max_step = 10 # Max number of time steps
time.fixed_dt = 0.5 # Use this constant dt if > 0
time.cfl = 0.95 # CFL factor
io.KE_int = 1
time.plot_interval = 10 # Steps between plot files
time.checkpoint_interval = -5 # Steps between checkpoint files
incflo.gravity = 0. 0. -9.81 # Gravitational force (3D)
incflo.density = 1.0 # Reference density
incflo.use_godunov = 1
transport.viscosity = 1.0e-5
transport.laminar_prandtl = 0.7
transport.turbulent_prandtl = 0.3333
turbulence.model = OneEqKsgsM84
incflo.physics = ABL
ICNS.source_terms = BoussinesqBuoyancy CoriolisForcing ABLForcing
TKE.source_terms = KsgsM84Src
BoussinesqBuoyancy.reference_temperature = 300.0
CoriolisForcing.latitude = 41.3
ABLForcing.abl_forcing_height = 90
incflo.velocity = 6.128355544951824 5.142300877492314 0.0
ABL.temperature_heights = 650.0 750.0 1000.0
ABL.temperature_values = 300.0 308.0 308.75
ABL.reference_temperature = 300.0
ABL.kappa = .41
ABL.surface_roughness_z0 = 0.15
amr.n_cell = 48 48 48 # Grid cells at coarsest AMRlevel
amr.max_level = 0 # Max AMR level in hierarchy
geometry.prob_lo = 0. 0. 0. # Lo corner coordinates
geometry.prob_hi = 1000. 1000. 1000. # Hi corner coordinates
geometry.is_periodic = 1 1 0 # Periodicity x y z (0/1)
# Boundary conditions
zlo.type = "wall_model"
zlo.tke_type = "zero_gradient"
zhi.type = "slip_wall"
zhi.temperature_type = "fixed_gradient"
zhi.temperature = 0.003 # tracer is used to specify potential temperature gradient
incflo.verbose = 0 # incflo_level
Raw
Sample-output.txt
==============================================================================
AMR-Wind (https://github.com/exawind/amr-wind)
AMR-Wind Git SHA :: 04f7f1262143-dirty
AMReX version :: 20.12.1-37-gc98be7422796 ( 20.12.1-37-gc98be7422796 )
Exec. date :: Wed Jan 6 15:39:45 2021
Build date :: Jan 6 2021 15:38:36
C++ compiler :: AppleClang 11.0.0.11000033
MPI :: ON (Num. ranks = 4)
GPU :: OFF
OpenMP :: OFF
This software is released under the BSD 3-clause license.
See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details.
------------------------------------------------------------------------------
MPI initialized with 4 MPI processes
MPI initialized with thread support level 0
AMReX (20.12.1-37-gc98be7422796) initialized
Initializing AMR-Wind ...
Initializing boundary conditions for velocity_mueff
Initializing boundary conditions for velocity
Initializing boundary conditions for density
Initializing boundary conditions for p
Initializing boundary conditions for velocity_src_term
Initializing boundary conditions for gp
Creating PDESystem instance: ICNS-Godunov
ABLWallFunction: log_law_height not specified for ABL physics. Assuming log_law_height = first cell height
ABLWallFunction: Neither surface_temp_flux nor surface_temp_rate specified for ABL physics. Assuming Neutral Stratification
Initializing boundary conditions for temperature_mueff
Initializing boundary conditions for temperature
Initializing boundary conditions for temperature_src_term
Creating PDESystem instance: Temperature-Godunov
Creating Physics instance: ABL
Initializing boundary conditions for mu_turb
Initializing boundary conditions for tke_mueff
Initializing boundary conditions for tke
Initializing boundary conditions for tke_src_term
Creating PDESystem instance: TKE-Godunov
Creating TurbulenceModel instance: OneEqKsgsM84-ConstTransport
Initializing I/O manager
Creating mesh... done
Grid summary:
Level 0 8 grids 110592 cells 100 % of domain
smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 32
For godunov_type select between plm, ppm, ppm_nolim and weno: it defaults to ppm
Creating MomentumSource instance: BoussinesqBuoyancy
Creating MomentumSource instance: CoriolisForcing
Creating MomentumSource instance: ABLForcing
For godunov_type select between plm, ppm, ppm_nolim and weno: it defaults to ppm
For godunov_type select between plm, ppm, ppm_nolim and weno: it defaults to ppm
Creating TKESource instance: KsgsM84Src
Begin initial projection
Nodal_projection 0 2.134403765e-17 2.134403765e-17
Completed initial projection
Begin initial pressure iterations. Num. iters = 3
dt: 0.5
CFL: 0.170223 (conv: 0.170223 diff: 0 src: 0 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 11 0.00340474802 9.606464929e-15
temperature_solve 2 0.0003219768467 4.848743629e-11
tke_solve 2 1.235805638e-06 2.813860256e-13
velocity_solve 3 0.01191515277 8.18900503e-13
Nodal_projection 7 0.01368931524 3.031017277e-14
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 8 0.0001226205269 2.77238627e-15
temperature_solve 2 0.0003219768467 4.848743629e-11
tke_solve 2 1.235805581e-06 2.813999034e-13
velocity_solve 3 0.01191571972 8.18900503e-13
Nodal_projection 5 4.624444549e-05 4.355321442e-15
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 8 0.0001156771922 2.952197812e-15
temperature_solve 2 0.0003219768467 4.848743629e-11
tke_solve 2 1.235805558e-06 2.814137812e-13
velocity_solve 3 0.01191566886 8.171241461e-13
Nodal_projection 5 5.610203535e-06 2.697309684e-16
Completed initial pressure iterations
Writing plot file plt00000 at time 0
Initialization successful. Time elapsed = 1.658684015
Time, Kinetic Energy: 0, 32.03011268
Begin simulation:
Run until 22000 sec. or 10 timesteps
Fixed timestepping with dt = 0.5; max. CFL from inputs = 0.95
==============================================================================
Step: 1 dt: 0.5 Time: 0 to 0.5
CFL: 0.171314 (conv: 0.170223 diff: 0 src: 0.013672 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 8 0.0001162588231 2.958364212e-15
temperature_solve 2 0.0003219768467 4.848743629e-11
tke_solve 2 1.235805553e-06 2.813999034e-13
velocity_solve 3 0.0119156701 8.171241461e-13
Nodal_projection 7 0.01371709208 3.031667799e-14
Time, Kinetic Energy: 0.5, 32.02766115
WallClockTime: 1 Pre: 0.000564 Solve: 0.5067 Post: 0.000912 Total: 0.5081
==============================================================================
Step: 2 dt: 0.5 Time: 0.5 to 1
CFL: 0.171343 (conv: 0.170252 diff: 0 src: 0.0136714 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.0002610719766 8.288590083e-16
temperature_solve 2 0.0003170770808 4.763478501e-11
tke_solve 2 2.352603464e-06 5.445227602e-13
velocity_solve 3 0.0118704071 8.153477893e-13
Nodal_projection 7 0.01371709337 3.030106548e-14
Time, Kinetic Energy: 1, 32.02524155
WallClockTime: 2 Pre: 0.000503 Solve: 0.5278 Post: 0.000911 Total: 0.5293
==============================================================================
Step: 3 dt: 0.5 Time: 1 to 1.5
CFL: 0.171276 (conv: 0.170259 diff: 0 src: 0.0131943 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.0003599906515 1.833575609e-15
temperature_solve 2 0.0003122708012 4.689582056e-11
tke_solve 2 3.46256154e-06 7.856354456e-13
velocity_solve 3 0.01182522785 8.117950756e-13
Nodal_projection 7 0.01371709337 3.030713701e-14
Time, Kinetic Energy: 1.5, 32.02284413
WallClockTime: 3 Pre: 0.000329 Solve: 0.5164 Post: 0.00168 Total: 0.5184
==============================================================================
Step: 4 dt: 0.5 Time: 1.5 to 2
CFL: 0.171281 (conv: 0.17025 diff: 0 src: 0.0132849 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.0004840768426 2.80601856e-15
temperature_solve 2 0.0003075557981 4.598632586e-11
tke_solve 3 4.585783163e-06 5.134781489e-16
velocity_solve 3 0.0117800575 8.117950756e-13
Nodal_projection 7 0.01371709337 3.030930541e-14
Time, Kinetic Energy: 2, 32.02045894
WallClockTime: 4 Pre: 0.000311 Solve: 0.5244 Post: 0.000802 Total: 0.5255
==============================================================================
Step: 5 dt: 0.5 Time: 2 to 2.5
CFL: 0.171252 (conv: 0.170223 diff: 0 src: 0.013275 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.0006192608167 3.853115608e-15
temperature_solve 2 0.0003029299247 4.513367458e-11
tke_solve 3 5.811882505e-06 6.245004514e-16
velocity_solve 3 0.01173422985 8.100187188e-13
Nodal_projection 7 0.01371709337 3.03049686e-14
Time, Kinetic Energy: 2.5, 32.01807787
WallClockTime: 5 Pre: 0.000379 Solve: 0.508 Post: 0.000877 Total: 0.5093
==============================================================================
Step: 6 dt: 0.5 Time: 2.5 to 3
CFL: 0.171195 (conv: 0.170176 diff: 0 src: 0.0132122 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.0007588534496 4.882951817e-15
temperature_solve 2 0.0002983910945 4.439471013e-11
tke_solve 3 7.247114643e-06 7.771561172e-16
velocity_solve 3 0.01168713916 8.082423619e-13
Nodal_projection 7 0.01371709337 3.030757069e-14
Time, Kinetic Energy: 3, 32.01569894
WallClockTime: 6 Pre: 0.000326 Solve: 0.504 Post: 0.00115 Total: 0.5055
==============================================================================
Step: 7 dt: 0.5 Time: 3 to 3.5
CFL: 0.171134 (conv: 0.170106 diff: 0 src: 0.0132646 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.0008920213384 5.924981914e-15
temperature_solve 2 0.0002939372798 4.359890227e-11
tke_solve 3 8.715370428e-06 9.436895709e-16
velocity_solve 3 0.01163732633 8.055778267e-13
Nodal_projection 7 0.01371709337 3.030713701e-14
Time, Kinetic Energy: 3.5, 32.01333043
WallClockTime: 7 Pre: 0.000408 Solve: 0.4902 Post: 0.00081 Total: 0.4914
==============================================================================
Step: 8 dt: 0.5 Time: 3.5 to 4
CFL: 0.171035 (conv: 0.170013 diff: 0 src: 0.0132246 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.001010780545 6.954194707e-15
temperature_solve 2 0.0002895665098 4.285993782e-11
tke_solve 3 1.008634875e-05 1.096345237e-15
velocity_solve 3 0.0115826695 8.055778267e-13
Nodal_projection 7 0.01371709336 3.029759603e-14
Time, Kinetic Energy: 4, 32.01097813
WallClockTime: 8 Pre: 0.000328 Solve: 0.5198 Post: 0.000819 Total: 0.5209
==============================================================================
Step: 9 dt: 0.5 Time: 4 to 4.5
CFL: 0.170924 (conv: 0.169889 diff: 0 src: 0.0132957 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.00111597 7.600338544e-15
temperature_solve 2 0.0002852768685 4.217781679e-11
tke_solve 3 1.119036343e-05 1.207367539e-15
velocity_solve 3 0.01152352334 8.038014698e-13
Nodal_projection 7 0.01371709336 3.030366756e-14
Time, Kinetic Energy: 4.5, 32.00864432
WallClockTime: 9 Pre: 0.000323 Solve: 0.5195 Post: 0.000785 Total: 0.5206
==============================================================================
Step: 10 dt: 0.5 Time: 4.5 to 5
CFL: 0.170759 (conv: 0.169716 diff: 0 src: 0.0133418 )
Godunov:
System Iters Initial residual Final residual
----------------------------------------------------------------------------
MAC_projection 9 0.001208684792 8.9619745e-15
temperature_solve 2 0.0002810664929 4.143885235e-11
tke_solve 3 1.313136215e-05 1.346145417e-15
velocity_solve 3 0.01146362575 7.984723993e-13
Nodal_projection 7 0.01371709335 3.030128232e-14
Writing plot file plt00010 at time 5
Time, Kinetic Energy: 5, 32.00632472
WallClockTime: 10 Pre: 0.000343 Solve: 0.5444 Post: 0.0508 Total: 0.5956
==============================================================================
Time spent in InitData(): 1.658684015
Time spent in Evolve(): 5.233803034
Unused ParmParse Variables:
[TOP]::amr.plot_file(nvals = 1) :: [plt]
AMReX (20.12.1-37-gc98be7422796) finalized
@sayerhs
Copy link
Author

sayerhs commented Apr 21, 2021

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment