excited to see the docs preview

@dkytezab is there a way we can compare to an estimate or exact solution to make sure the answer is correct?

https://github.com/NumericalEarth/Breeze.jl/actions/runs/23207750341/job/67447818390?pr=565#step:7:31

Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.
ERROR: LoadError: StackOverflowError:
Stacktrace:
 [1] promote_to(::Type{Reactant.TracedRArray{Float64, 3}}, rhs::Oceananigans.Fields.Field{Oceananigans.Grids.Center, Oceananigans.Grids.Center, O

CC @Pangoraw

Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.

we can reproduce with show(dδρᵢ).

Not sure how to reproduce this but on 1.12 we also hit the 'llvm.call' op incorrect number of operands (3) for callee (expecting: 4) error.

There are some pretty big discrepancies between the finite-difference derivatives and automatic derivatives. I think this is unrelated to timestepping: there is a difference even if we just call set! and don't time step.

edit: there is some problem differentiating through update_state! which gets called before timestepping. For zero timesteps, the automatic derivatives are nearly-uniformly 1 except at the corners/sides where they are 4/2. This is obviously surprising, because there is zero propagation time, so the sensitivity should only be nonzero at the receiver. One explanation is that we are appending the adjoint of global reads (i.e. writes) hence the 1's, and the adjoints of several halo fills. Hence the 2's and 4's.

derivative of just calling set!(model; ρ = ρᵗ, θ = θ₀, u = uᵇᵍ)

#575 will allow new Ocenanigans, which will get the fix for the docs crash in.

Unless we build docs on 1.11 it will also need CliMA/Oceananigans.jl#5415 (or upcoming call conv reactant fix) to get around the next crash in the pipeline.

Ok, now we're getting (unfortunately Literate swallows full error messages and stacktraces):

ERROR: LoadError: LoadError: "failed to run pass manager on module"
in expression starting at /__w/Breeze.jl/Breeze.jl/docs/src/literated/acoustic_wave.md:2
when executing the following code block from inputfile `/__w/Breeze.jl/Breeze.jl/examples/acoustic_wave.jl`
```julia
@info "Compiling differentiated model — this may take a minute..."
compiled_grad = Reactant.@compile raise=true raise_first=true sync=true grad_loss(
    model_ad, dmodel_ad, δρᵢ, dδρᵢ,
    ρᵗ, ρᵇᵍ, uᵇᵍ, θ₀, Δt, Nsteps, target_i, target_k)
@info "Running gradient..."
dδρ, J = compiled_grad(
    model_ad, dmodel_ad, δρᵢ, dδρᵢ,
    ρᵗ, ρᵇᵍ, uᵇᵍ, θ₀, Δt, Nsteps, target_i, target_k)
xs = xnodes(grid_ad, Center())
zs = znodes(grid_ad, Center())
x_target = xs[target_i]
z_target = zs[target_k]
@info @sprintf("Receiver density J = %.6e  at (x=%.1f m, z=%.1f m) after %d steps",
               Float64(only(J)), x_target, z_target, Nsteps)
```

Is that what CliMA/Oceananigans.jl#5415 addresses?

Still getting the same error after checking out CliMA/Oceananigans.jl@4da9e65

Yeah, there is some underlying issue with Reactant + Julia 1.12 codegen. See EnzymeAD/Reactant.jl#2701

Should we switch the docs to using julia v1.11 then? It's doable, but I'll need to change the docker image.

Should we switch the docs to using julia v1.11 then? It's doable, but I'll need to change the docker image.

I'm ok with this esp if its computationally feasible

Docs job is failing because stuff like (not the only case, there are multiple errors)

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Preview at https://numericalearth.github.io/BreezeDocumentation/previews/PR565/literated/acoustic_wave/#Differentiability:-sensitivity-to-the-initial-perturbation. But figures aren't displayed anywhere because the figure objects aren't ever at the end of a "cell", which is what's displayed below. Edit: should be fixed in the latest version

Should we wait for EnzymeAD/Reactant.jl#2704 to make it into a new release so that we can go back to Julia v1.12 here? Edit: Reactant tests in #577 with Julia v1.12 are passing, so we may wait for a new Reactant version and go back to building the docs with Julia v1.12 👀

Does log10(error) larger than 0 mean that the relative error is larger than 100%? 🤔 Edit: yes, that's visible in the generated docs:

[ Info: Rel. error: min=2.01e-04  median=4.08e-02  mean=1.29e-01  max=3.18e+00

After playing with some parameters locally, I'm convinced that the thing here is that the derivative in those two points is "very small", and with AD vs FD we get two different "very small" derivatives (things like -3.69401e-6 vs -1.42948e-5), but by its nature, finite differences isn't necessarily The Right Answer™️. Should we add a comment along the lines of "while it's useful to compare the patterns between AD and FD, FD is inherently unstable, and as such it doesn't necessarily provide a reliable reference", to justify the difference?

Maybe we should figure out how to compute a gradient that will be reasonably smooth and easy to access accuracy

I can work on a suitable example