Segfault on GPU in abl_meso_input_*pa

[marchdf]

Bug description

Seeing GPU segfaults in abl_meso_input_*pa

Steps to reproduce

Steps to reproduce the behavior:

1. Compiler used
   • GCC
   • LLVM
   • oneapi (Intel)
   • nvcc (NVIDIA)
   • rocm (AMD)
   • with MPI
   • other:
2. Operating system
   • Linux
   • OSX
   • Windows
   • other (do tell ;)):
3. Hardware:
   • CPU
   • GPU
4. Machine details ():

ellis

5. Input file attachments abl_meso_input_*pa regression tests
6. If this is a segfault, a stack trace from a debug build (paste or attach):

Host Name: kl5
=== If no file names and line numbers are shown below, one can run
            addr2line -Cpfie my_exefile my_line_address
    to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
    Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
    One can use
            readelf -wl my_exefile | grep my_line_address'
    to find out the offset for that line.

 0: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0xeae56a]
    amrex::BLBackTrace::print_backtrace_info(_IO_FILE*)
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/submods/amrex/Src/Base/AMReX_BLBackTrace.cpp:203:25

 1: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0xeadf79]
    amrex::BLBackTrace::handler(int)
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/submods/amrex/Src/Base/AMReX_BLBackTrace.cpp:101:7

 2: /lib64/libc.so.6(+0x4eb50) [0x7f59b0a75b50]

 3: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0xad6889]
    amr_wind::interp::Index amr_wind::interp::check_bounds<double const*, double>(double const*, double const*, double const&) inlined at /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/\
utilities/linear_interpolation.H:41:24 in amrex::Vector<double, std::allocator<double> >::value_type amr_wind::interp::linear<amrex::PODVector<double, amrex::ArenaAllocator<double> >, amrex::Vector<double, std::allocator<double> > >(am\
rex::PODVector<double, amrex::ArenaAllocator<double> > const&, amrex::Vector<double, std::allocator<double> > const&, amrex::PODVector<double, amrex::ArenaAllocator<double> >::value_type const&)
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/utilities/linear_interpolation.H:27:20
amr_wind::interp::Index amr_wind::interp::bisection_search<double const*, double>(double const*, double const*, double const&)
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/utilities/linear_interpolation.H:41:24
std::iterator_traits<double const*>::value_type amr_wind::interp::linear<double const*, double const*>(double const*, double const*, double const*, std::iterator_traits<double const*>::value_type const&)
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/utilities/linear_interpolation.H:92:34
amrex::Vector<double, std::allocator<double> >::value_type amr_wind::interp::linear<amrex::PODVector<double, amrex::ArenaAllocator<double> >, amrex::Vector<double, std::allocator<double> > >(amrex::PODVector<double, amrex::ArenaAllocat\
or<double> > const&, amrex::Vector<double, std::allocator<double> > const&, amrex::PODVector<double, amrex::ArenaAllocator<double> >::value_type const&)
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/utilities/linear_interpolation.H:109:14

 4: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0xad3ee4]
    amr_wind::pde::icns::ABLMesoForcingMom::mean_velocity_heights(amr_wind::VelPlaneAveragingFine const&, std::unique_ptr<amr_wind::ABLMesoscaleInput, std::default_delete<amr_wind::ABLMesoscaleInput> > const&)
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/equation_systems/icns/source_terms/ABLMesoForcingMom.cpp:197:67

 5: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0x7971e0]
    amr_wind::ABL::pre_advance_work()
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/wind_energy/ABL.cpp:217:6

 6: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0x425d41]
    incflo::pre_advance_stage2()
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/incflo_advance.cpp:26:33

 7: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0x436bb7]
    incflo::Evolve()
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/incflo.cpp:277:54

 8: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0x4242c7]
    main
/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-cuda/amr-wind/amr-wind/main.cpp:78:57

 9: /lib64/libc.so.6(__libc_start_main+0xe5) [0x7f59b0a61d85]

10: /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/spack/opt/spack/linux-rhel8-zen3/gcc-12.2.0/amr-wind-main-sfuysvfwd7jtqnzhexzy5epawjdoiy6y/bin/amr_wind() [0x423c2e]
    _start at ??:?

Expected behavior

No segfault