Skip to content

Using field dependent boundary conditions with latitude-longitude grid gives InvalidIRErrror on GPU #4659

Open
Open
Using field dependent boundary conditions with latitude-longitude grid gives InvalidIRErrror on GPU#4659
@matt-graham

Description

@matt-graham
matt-graham
opened on Jul 21, 2025

I have been trying to get the double gyre example from #3087 running with Oceananigans v0.97.0. I have managed to get a version which runs on CPU but changing to run on GPUI consistently get a InvalidIRError during the initializing simulation stage

ERROR: LoadError: InvalidIRError: compiling MethodInstance for Oceananigans.BoundaryConditions.gpu__apply_z_bcs!(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(60, 60)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.StaticSize{(4, 4)}, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Nothing, Nothing}}, ::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, ::Tuple{Center, Face, Center}, ::LatitudeLongitudeGrid{Float64, Bounded, Bounded, Bounded, Oceananigans.Grids.StaticVerticalDiscretization{OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, CuDeviceVector{Float64, 1}}, OffsetArrays.OffsetVector{Float64, CuDeviceVector{Float64, 1}}, OffsetArrays.OffsetVector{Float64, CuDeviceVector{Float64, 1}}, OffsetArrays.OffsetVector{Float64, CuDeviceVector{Float64, 1}}, Float64, Float64, Nothing, Int64}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Face, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(v_drag), Nothing, Nothing, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity4)}}}, ::BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ::Tuple{@NamedTuple{time::Float64, last_Δt::Float64, last_stage_Δt::Float64, iteration::Int64, stage::Int64}, @NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, w::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, η::OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, U::Field{Face, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, V::Field{Center, Face, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}}, Nothing, Nothing}) resulted in invalid LLVM IR
Reason: unsupported call to an unknown function (call to ijl_get_nth_field_checked)

With a bit of trial and error, I've managed to come with the following more minimal reproducing example

using Oceananigans
using Oceananigans.Units
using CUDA

grid = LatitudeLongitudeGrid(
    GPU();
    size = (60, 60, 25),
    longitude = (-30, +30),
    latitude = (15, 75),
    z = (-1.8kilometers, 0),
    topology = (Bounded, Bounded, Bounded),
    halo = (4, 4, 4)
)

@inline v_drag(λ, φ, t, v) = - 0.001 * v

v_drag_bc = FluxBoundaryCondition(v_drag, field_dependencies = :v)
v_bcs = FieldBoundaryConditions(bottom = v_drag_bc)

model = HydrostaticFreeSurfaceModel(; grid, boundary_conditions = (; v = v_bcs))
simulation = Simulation(model, Δt = 10minutes, stop_time = 1hours)
run!(simulation)

It appears the combination of running on GPU plus field dependent boundary conditions and latitude-longitude grid is required to produce the error.

For example, changing to run on CPU

using Oceananigans
using Oceananigans.Units

grid = LatitudeLongitudeGrid(
    CPU();
    size = (60, 60, 25),
    longitude = (-30, +30),
    latitude = (15, 75),
    z = (-1.8kilometers, 0),
    topology = (Bounded, Bounded, Bounded),
    halo = (4, 4, 4)
)

@inline v_drag(λ, φ, t, v) = - 0.001 * v

v_drag_bc = FluxBoundaryCondition(v_drag, field_dependencies = :v)
v_bcs = FieldBoundaryConditions(bottom = v_drag_bc)

model = HydrostaticFreeSurfaceModel(; grid, boundary_conditions = (; v = v_bcs))
simulation = Simulation(model, Δt = 10minutes, stop_time = 1hours)
run!(simulation)

using a rectlinear grid

using Oceananigans
using Oceananigans.Units
using CUDA

grid = RectilinearGrid(
    GPU();
    size = (60, 60, 25),
    x = (-3333kilometers, +3333kilometers),
    y = (-3333kilometers, +3333kilometers),
    z = (-1.8kilometers, 0),
    topology = (Bounded, Bounded, Bounded),
    halo = (4, 4, 4)
)

@inline v_drag(x, y, t, v) = -0.001 * v

v_drag_bc = FluxBoundaryCondition(v_drag, field_dependencies = :v)
v_bcs = FieldBoundaryConditions(bottom = v_drag_bc)

model = HydrostaticFreeSurfaceModel(; grid, boundary_conditions = (; v = v_bcs))
simulation = Simulation(model, Δt = 10minutes, stop_time = 1hours)
run!(simulation)

or with a non field-dependent boundary condition

using Oceananigans
using Oceananigans.Units
using CUDA

grid = LatitudeLongitudeGrid(
    GPU();
    size = (60, 60, 25),
    longitude = (-30, +30),
    latitude = (15, 75),
    z = (-1.8kilometers, 0),
    topology = (Bounded, Bounded, Bounded),
    halo = (4, 4, 4)
)

@inline v_drag(λ, φ, t) = -0.001

v_drag_bc = FluxBoundaryCondition(v_drag)
v_bcs = FieldBoundaryConditions(bottom = v_drag_bc)

model = HydrostaticFreeSurfaceModel(; grid, boundary_conditions = (; v = v_bcs))
simulation = Simulation(model, Δt = 10minutes, stop_time = 1hours)
run!(simulation)

all run without issues.

This appears to be related to #4165 though in this case for boundary conditions rather than forcing functions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions