Skip to content

arm64-specific: wasm fails to instantiate: out of bounds memory access #2382

Closed

Description

@davidmdm

Describe the bug
Running a wasm binary returns the error:

error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
        .time.__dataIO_.big8(i32) i32
        .github.com_BurntSushi_toml_internal.init(i32) i32
        .runtime.doInit1(i32) i32
        .runtime.main(i32) i32
        .wasm_pc_f_loop()
        ._rt0_wasm_wasip1()

To Reproduce
Running on Darwin/arm64 M2 chip

set -x

if [ ! -d "yoke" ]; then
  git clone --depth=1 https://github.com/yokecd/yoke.git
fi

(
  cd yoke
  GOOS=wasip1 GOARCH=wasm go build -o ../redis-installer.wasm ./examples/redis
)

# will run successfully
go run github.com/tetratelabs/wazero/cmd/[email protected] run ./redis-installer.wasm

# will fail with aforementioned error
go run github.com/tetratelabs/wazero/cmd/[email protected] run ./redis-installer.wasm

Expected behavior
Expected same output as [email protected]

Environment (please complete the relevant information):

  • Go version: 1.24.0
  • wazero Version: 1.9.0
  • Host architecture: arm64
  • Runtime mode: compiler

Activity

  1. evacchi commented on Feb 24, 2025

    @evacchi
    Member

    can you run this with amd64 emulation to see if it's specific to arm64? this should do it:

    GOARCH=amd64 go run github.com/tetratelabs/wazero/cmd/[email protected] run ./redis-installer.wasm
    
  2. davidmdm commented on Feb 24, 2025

    @davidmdm
    ContributorAuthor

    @evacchi it works under amd64 emulation. So its a bug specific to arm64!

  3. changed the title [-]wasm fails to instantiate: out of bounds memory access[/-] [+]arm64-specific: wasm fails to instantiate: out of bounds memory access[/+] on Feb 24, 2025
  4. davidmdm commented on Feb 25, 2025

    @davidmdm
    ContributorAuthor

    Perhaps still a trampoline issue? I remember when certain programs would hang in version 1.7.x and that a fix was made to the arm64 backend then. Perhaps an off-by-one of sorts is causing a memory out of bounds issue?

  5. evacchi commented on Feb 25, 2025

    @evacchi
    Member

    @davidmdm out of curiosity, have you tried against Go 1.23? let's rule out something related to #2375

  6. davidmdm commented on Feb 25, 2025

    @davidmdm
    ContributorAuthor

    @evacchi Just ran against a version with the wasm produced by go1.23.6 and it produces an identical error.
    I checked the new wasm did not match the one produced by go1.24.0 as well to be sure.

  7. ncruces commented on Feb 25, 2025

    @ncruces
    Contributor

    I don't think that's what @evacchi asked for. Don't build the Wasm with 1.23, rather use 1.23 to go run wazero.

  8. davidmdm commented on Feb 25, 2025

    @davidmdm
    ContributorAuthor

    @ncruces Good catch. I got that confused.

    @evacchi Same result unfortunately!

    GOTOOLCHAIN=go1.23.6 go run github.com/tetratelabs/wazero/cmd/[email protected] run ./redis-installer.wasm
    
    go: downloading go1.23.6 (darwin/arm64)
    error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
    wasm stack trace:
            .time.__dataIO_.big8(i32) i32
            .github.com_BurntSushi_toml_internal.init(i32) i32
            .runtime.doInit1(i32) i32
            .runtime.main(i32) i32
            .wasm_pc_f_loop()
            ._rt0_wasm_wasip1()
    exit status 1
  9. evacchi commented on Feb 26, 2025

    @evacchi
    Member

    interesting, so unrelated issue

  10. evacchi commented on Feb 27, 2025

    @evacchi
    Member

    FWIW 😅

    ❯ go run github.com/tetratelabs/wazero/cmd/[email protected] run -env GODEBUG='inittrace=1' ./redis-installer.wasm
    init runtime @0.006 ms, 0.024 ms clock, 0 bytes, 0 allocs
    init crypto/internal/fips140deps/cpu @0.16 ms, 0.004 ms clock, 0 bytes, 0 allocs
    init errors @0.18 ms, 0.005 ms clock, 0 bytes, 0 allocs
    init iter @0.20 ms, 0.003 ms clock, 16 bytes, 1 allocs
    init sync @0.21 ms, 0.007 ms clock, 0 bytes, 0 allocs
    init internal/godebug @0.23 ms, 0.15 ms clock, 312 bytes, 7 allocs
    init encoding/base32 @0.40 ms, 0.007 ms clock, 640 bytes, 2 allocs
    init crypto @0.42 ms, 0.004 ms clock, 160 bytes, 1 allocs
    init hash/crc32 @0.43 ms, 0.026 ms clock, 1264 bytes, 13 allocs
    init syscall @0.47 ms, 0.044 ms clock, 272 bytes, 3 allocs
    init time @0.52 ms, 0.008 ms clock, 256 bytes, 2 allocs
    init context @0.54 ms, 0.010 ms clock, 112 bytes, 1 allocs
    init github.com/BurntSushi/toml/internal @0.56 ms, 0.034 ms clock, 0 bytes, 0 allocs
    error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
    wasm stack trace:
            .time.Time.absSec(i32) i32
            .time.Time.Month(i32) i32
            .time.__Time_.Month(i32) i32
            .github.com_google_go_cmp_cmp_internal_diff.init(i32) i32
            .runtime.doInit1(i32) i32
            .runtime.main(i32) i32
            .wasm_pc_f_loop()
            ._rt0_wasm_wasip1()
    exit status 1
    
  11. evacchi commented on Feb 28, 2025

    @evacchi
    Member
  12. evacchi commented on Mar 1, 2025

    @evacchi
    Member
    ❯ wasm-opt ../redis-installer.wasm -o ../redis-installer-opt.wasm -Os -all
    
    ❯ go run ./cmd/wazero run ../repro/redis-installer-opt.wasm
    t.Kind == 0
    panic: unknown type kind
    
    goroutine 1 [running]:
    reflect.(*abiSeq).regAssign(0x0, 0xa, 0x6259a0)
            /opt/homebrew/Cellar/go/1.24.0/libexec/src/reflect/abi.go:247 +0x59
    strconv.Itoa(...)
            /opt/homebrew/Cellar/go/1.24.0/libexec/src/strconv/itoa.go:35
    github.com/davidmdm/ansi.Esc({0x246ce08, 0x1, 0x1})
            /Users/evacchi/.local/go/pkg/mod/github.com/davidmdm/[email protected]/style.go:15 +0x5
    github.com/davidmdm/ansi.init()
            /Users/evacchi/.local/go/pkg/mod/github.com/davidmdm/[email protected]/style.go:20 +0x8
    exit status 2
    

    🤪

  13. evacchi commented on Mar 1, 2025

    @evacchi
    Member

    probably a red herring, I think most of what wasm-opt did in this is case is removing DWARF symbols, and with -env GODEBUG=inittrace=1 it still crashes with OOB

    error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
    wasm stack trace:
            .$2211(i32) i32
            .$2183(i32) i32
            .$2184(i32) i32
            .$2254(i32) i32
            .$2321(i32) i32
            .$47976(i32) i32
            .$1098(i32) i32
            .$951(i32) i32
            .$1667()
            .$1671()
    
  14. evacchi commented on Mar 2, 2025

    @evacchi
    Member

    ok here's what I have learned so far. In function #2320 (big8) register (x11) gets assigned a value that's higher than the given bound:

         x15 = 0x0000000028d90004
         x14 = 0x0000000002800000
    

    and it "correctly" fails.
    this is not even a scaling issue because it would be still 0x28d9000 > 0x2800000

    the value is loaded waaay earlier in x10 from [x10] and then eventually copied to x11 (it goes through a bunch of spills and reloads too):

        0x1285db1f4: b      0x1285db1f4
        0x1285db1f8: add    x10, x12, x10
        0x1285db1fc: ldur   x10, [x10]
    

    before the load, x10 = 0x000001402776adf8

    in post-regalloc, this appears to be block L148:

    L148:
            add x10, x12, x10
            ldr x10, [x10]
    

    if I elide the bound check it is hard to track what truly is different because many branches obviously disappear (the bound checks themselves), some loads fold UXTW into the addressing mode, and all registers are thus different

  15. 4 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions