Skip to content

Convert Array4 to ArrayND#4868

Merged
WeiqunZhang merged 26 commits intoAMReX-Codes:developmentfrom
ankithadas:ArrayND
Jan 16, 2026
Merged

Convert Array4 to ArrayND#4868
WeiqunZhang merged 26 commits intoAMReX-Codes:developmentfrom
ankithadas:ArrayND

Conversation

@ankithadas
Copy link
Copy Markdown
Contributor

Summary

This PR introduces a preliminary implementation that generalizes Array4 to an N-dimensional case, extending AMReX support for arbitrary dimensionality as outlined in #3955.

The new ArrayND implementation has the same size as Array4 (64 bytes) and is intended to exhibit identical behaviour. However, this change modifies the types of begin and end and removes ncomp, which may break downstream code. In particular, pyamrex will likely need to be updated to accommodate these type changes. All internal AMReX source code that uses begin, end, or ncomp has been updated.

One open question concerns the behaviour of ArrayND<dim>::operator()(idx...) when the number of indices provided is dim-1. In Array4, this case sets the last dimension ncomp to zero. Should this behaviour be extended to all dim cases in ArrayND ?

Additional background

Checklist

The proposed changes:

  • fix a bug or incorrect behavior in AMReX
  • add new capabilities to AMReX
  • changes answers in the test suite to more than roundoff level
  • are likely to significantly affect the results of downstream AMReX users
  • include documentation in the code and/or rst files, if appropriate

@WeiqunZhang
Copy link
Copy Markdown
Member

/run-hpsf-gitlab-ci

@github-actions
Copy link
Copy Markdown

@zingale
Copy link
Copy Markdown
Member

zingale commented Dec 26, 2025

does std::array work with HIP on device?

@BenWibking
Copy link
Copy Markdown
Contributor

does std::array work with HIP on device?

We use it. I haven't seen any issues.

@amrex-gitlab-ci-reporter
Copy link
Copy Markdown

GitLab CI 1365185 finished with status: failed. See details at https://gitlab.spack.io/amrex/amrex/-/pipelines/1365185.

Copy link
Copy Markdown
Member

@AlexanderSinn AlexanderSinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently there are a lot of special cases for dim == 4 for both the accessors and constructors. I think in general it is useful to use an ArrayND with n-1 spatial dimensions and 1 component dimension, even if dim != 4. For example, I would use this for a 2+1D array. There should be these 5 accessor overloads, each for both operator() and ptr():

  • idx... i: This should cover both sizeof...(idx) == dim and dim-1 like currently, but also replace the int, int, int, int version. For this, we might have to allow all types that are implicitly convertible to int for idx.
  • IntVectND<Dim>:  for Dim == dim, dim-1 and DIM < dim-1. In the last case, the remaining indexes are set to 0, except the last one, which is set to lo[dim-1] but not present in the offset calculation.
  • IntVectND<Dim>, int: for Dim == dim-1 and DIM < dim-1. In the second case, the IntVect is zero-extended as above. This overload won't be usable if dim == 1
  • Dim3 kept for backward compatibility, only works if dim == 4.
  • Dim3, int kept for backward compatibility, only works if dim == 4.

In addition to that, I think there should be more constructors, for example, taking a BoxND as input or an ArrayND of different dimensions.

@ankithadas
Copy link
Copy Markdown
Contributor Author

@AlexanderSinn Could you provide more information regarding the types of constructors that would be beneficial for your application?

I am not convinced that BoxND as an input is a good idea. There is an implicit understanding that BoxND is related to AMREX_SPACEDIM of the application, whereas ArrayND is not necessarily tied to this. This mismatch can lead to unexpected behaviour.

@AlexanderSinn
Copy link
Copy Markdown
Member

Yes, currently lower dimensions also use Array4. I assume this is out of simplicity and to make it easy to write code that works in 1D, 2D and 3D by always passing i,j,k,n to the array. However, I think eventually it should also be possible to use the ArrayND of the dimension the code is compiled for (plus one), as this should be slightly more performant and cleaner. The constructors should allow for both ways of using the array and should essentially mirror the interface of operator() for what combination of inputs and dimensions are allowed. For constructors using BoxND I would suggest:

  • T*, BoxND<Dim>:  for Dim == dim, dim-1 and DIM < dim-1. In the last case, the remaining dimensions are set to begin=0, end=1.
  • T*, BoxND<Dim>, int ncomp: for Dim == dim-1 and DIM < dim-1. In the second case, the Box is extended with begin=0, end=1. This overload won't be usable if dim == 1

Additionally, the ArrayND, ncomp and ArrayND, start_comp, ncomp constructors could be made to work with N>=2 instead of only N==4.

For the application I'm working on, I currently use implicit conversions from Array4 to a 2D array and a 3D array, where the 3D version removes the second to last dimension but keeps the last one. Dimensions that are removed are asserted to have size one if the constructor is called on the CPU.  This might be best left to the application, not AMReX.

@ankithadas
Copy link
Copy Markdown
Contributor Author

ankithadas commented Jan 2, 2026

I believe the final issue to address is the default indices used by ArrayND::operator()(IntVect<Dim>) when Dim < N. In my opinion, the default value should be begin rather than zeros.

@ankithadas ankithadas marked this pull request as ready for review January 2, 2026 02:22
@AlexanderSinn
Copy link
Copy Markdown
Member

The forward declaration

template <typename T> struct Array4;

in AMReX_BaseFwd.H still needs to be updated.

Also in the ArrayND, ncomp constructors, replace 3 with N-1.

@WeiqunZhang
Copy link
Copy Markdown
Member

@yut23 Note that this PR will break the GDB script (https://github.com/AMReX-Codes/amrex/tree/development/Tools/GDB) contributed by you. It would be great if you can fix it when this is merged.

@zingale
Copy link
Copy Markdown
Member

zingale commented Jan 5, 2026

@WeiqunZhang
Copy link
Copy Markdown
Member

The Castro issue is that the indices passed to ArrayND are Real not integers.

    Real jfac = 1.0_rt;
    Real kfac = 1.0_rt;
    fine(i,j,k) = crse(clo.x, j / jfac, k / kfac) / rfac;

@ankithadas
Copy link
Copy Markdown
Contributor Author

one of our Castro problem setups fails to build with this PR: http://groot.astro.sunysb.edu/Castro/test-suite/gfortran/2026-01-05-003/rad-2Tshock-3d.make.out

The error is from this line

  327 |                 fine(i,j,k) = crse(clo.x, j / jfac, k / kfac) / rfac;

I believe jfac is of type amex::Real. Previously this code relied on implicit conversion from Real to int when indexing Array4. The changes in this PR now require all index arguments to be explicitly int.

@ankithadas
Copy link
Copy Markdown
Contributor Author

Sorry, my earlier test case was incorrect, since the IntVect overloads were only introduced in this branch.
The following code fails to compile in the ArrayND branch with C++17, but works on the dev branch. Compiling with C++20 fixes the issue.

Real* nullptr_real = nullptr;
Dim3 begin = IntVect(0).dim3();
Dim3 end = IntVect(ncells).dim3();
Array4 t (nullptr_real, begin, end, 1);

@ankithadas
Copy link
Copy Markdown
Contributor Author

For discussion. Should we allow this construction now that last_dim_component is a template parameter

ArrayND<Real, 5> a(nullptr, IntVectND<1>(0), IntVectND<1>(ncells));
ArrayND<Real, 5> b(nullptr, BoxND(IntVectND<1>(0), IntVectND<1>(ncells)));

I find this rather confusing tbh.

@WeiqunZhang
Copy link
Copy Markdown
Member

Agreed. It's very confusing. We should only allow M==N and M+1==N && last_dim_component.

@WeiqunZhang
Copy link
Copy Markdown
Member

/run-hpsf-gitlab-ci

@github-actions
Copy link
Copy Markdown

@amrex-gitlab-ci-reporter
Copy link
Copy Markdown

GitLab CI 1389041 finished with status: success. See details at https://gitlab.spack.io/amrex/amrex/-/pipelines/1389041.

@WeiqunZhang
Copy link
Copy Markdown
Member

This now compiles.

constexpr Array4<int> a{};
static_assert(a.size() == 0);

@WeiqunZhang WeiqunZhang merged commit 909028c into AMReX-Codes:development Jan 16, 2026
73 of 74 checks passed
@ankithadas
Copy link
Copy Markdown
Contributor Author

Thank you for patiently reviewing the PR so many times.

@WeiqunZhang
Copy link
Copy Markdown
Member

Thank you for the PR! You did the work! If there are still bugs/defects there, it's all my fault for not catching it. :)

@zingale
Copy link
Copy Markdown
Member

zingale commented Jan 16, 2026

This change causes Castro when built with DEBUG=TRUE to run 3x slower. Is there a way to get the performance back in debug mode? Things only seem slightly slower when optimized.

@WeiqunZhang
Copy link
Copy Markdown
Member

WeiqunZhang commented Jan 16, 2026 via email

ax3l pushed a commit to AMReX-Codes/pyamrex that referenced this pull request Jan 16, 2026
The innard of amrex::Array has changed in
AMReX-Codes/amrex#4868.
@WeiqunZhang
Copy link
Copy Markdown
Member

I have a version that is only 10% slower than the original version in debug mode.

@WeiqunZhang
Copy link
Copy Markdown
Member

See #4900

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants