Skip to content

invalid table access only with Go 1.24 and amd64 #2375

Closed

Description

@anuraaga

Describe the bug

go-re2 usage crashes only with Go 1.24 on amd64. I guess this must be some interaction with different codegen in 1.24 on that platform. I believe it is happening when invoking a wasi host function and may be trampoline related. It is crashing right away with no concurrency or anything special.

❯ GOARCH=amd64 ~/tmp/go/bin/go test .
--- FAIL: TestBadCompile (1.34s)
panic: wasm error: invalid table access
        wasm stack trace:
                libcre2-noopt.so.fwrite(i32,i32,i32,i32) i32
                        0xc6b: wasisdk:/v25.0/build/sysroot/wasi-libc-wasm32-wasip1-threads/libc-top-half/musl/src/stdio/fwrite.c:10:36 (inlined)
                               wasisdk:/v25.0/build/sysroot/wasi-libc-wasm32-wasip1-threads/libc-top-half/musl/src/stdio/fwrite.c:33:6
                libcre2-noopt.so.absl::lts_20240722::log_internal::WriteToStderr(std::__2::basic_string_view<char, std::__2::char_traits<char>>, absl::lts_20240722::LogSeverity)(i32,i32)
                        0x38259: /re2/abseil-cpp/absl/log/internal/globals.cc:79:3
                libcre2-noopt.so.void absl::lts_20240722::base_internal::CallOnceImpl<absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)::'lambda'()>(std::__2::atomic<unsigned int>*, absl::lts_20240722::base_internal::SchedulingMode, absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)::'lambda'()&&)(i32)
                        0x3d0e7: /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:104:7 (inlined)
                                 /opt/wasi-sdk/share/wasi-sysroot/include/wasm32-wasip1-threads/c++/v1/__type_traits/invoke.h:149:25 (inlined)
                                 /opt/wasi-sdk/share/wasi-sysroot/include/wasm32-wasip1-threads/c++/v1/__functional/invoke.h:28:10 (inlined)
                                 /re2/abseil-cpp/absl/base/call_once.h:182:5
                libcre2-noopt.so.absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)(i32,i32)
                        0x3cfd2: /re2/abseil-cpp/absl/base/call_once.h:216:5 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:99:5
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogToSinks(absl::lts_20240722::LogEntry const&, absl::lts_20240722::Span<absl::lts_20240722::LogSink*>, bool)(i32,i32,i32)
                        0x3caec: /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:260:13 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:201:9 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:281:31
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::SendToLog()(i32)
                        0x472cd: /re2/abseil-cpp/absl/log/internal/log_message.cc:551:3
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::Flush()(i32)
                        0x471c5: /re2/abseil-cpp/absl/log/internal/log_message.cc:465:3
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::~LogMessage()(i32) i32
                        0x46f9f: /re2/abseil-cpp/absl/log/internal/log_message.cc:299:3
                libcre2-noopt.so.re2::RE2::Init(std::__2::basic_string_view<char, std::__2::char_traits<char>>, re2::RE2::Options const&)(i32,i32,i32)
                        0x6fd7a: /re2/re2/re2.cc:237:7
                libcre2-noopt.so.re2::RE2::RE2(std::__2::basic_string_view<char, std::__2::char_traits<char>>, re2::RE2::Options const&)(i32,i32,i32) i32
                        0x705b6: /re2/re2/re2.cc:158:3
                libcre2-noopt.so.cre2_new(i32,i32,i32) i32
                        0x756c0: /cre2/cre2.cpp:135:27 [recovered]
        panic: wasm error: invalid table access
        wasm stack trace:
                libcre2-noopt.so.fwrite(i32,i32,i32,i32) i32
                        0xc6b: wasisdk:/v25.0/build/sysroot/wasi-libc-wasm32-wasip1-threads/libc-top-half/musl/src/stdio/fwrite.c:10:36 (inlined)
                               wasisdk:/v25.0/build/sysroot/wasi-libc-wasm32-wasip1-threads/libc-top-half/musl/src/stdio/fwrite.c:33:6
                libcre2-noopt.so.absl::lts_20240722::log_internal::WriteToStderr(std::__2::basic_string_view<char, std::__2::char_traits<char>>, absl::lts_20240722::LogSeverity)(i32,i32)
                        0x38259: /re2/abseil-cpp/absl/log/internal/globals.cc:79:3
                libcre2-noopt.so.void absl::lts_20240722::base_internal::CallOnceImpl<absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)::'lambda'()>(std::__2::atomic<unsigned int>*, absl::lts_20240722::base_internal::SchedulingMode, absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)::'lambda'()&&)(i32)
                        0x3d0e7: /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:104:7 (inlined)
                                 /opt/wasi-sdk/share/wasi-sysroot/include/wasm32-wasip1-threads/c++/v1/__type_traits/invoke.h:149:25 (inlined)
                                 /opt/wasi-sdk/share/wasi-sysroot/include/wasm32-wasip1-threads/c++/v1/__functional/invoke.h:28:10 (inlined)
                                 /re2/abseil-cpp/absl/base/call_once.h:182:5
                libcre2-noopt.so.absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)(i32,i32)
                        0x3cfd2: /re2/abseil-cpp/absl/base/call_once.h:216:5 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:99:5
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogToSinks(absl::lts_20240722::LogEntry const&, absl::lts_20240722::Span<absl::lts_20240722::LogSink*>, bool)(i32,i32,i32)
                        0x3caec: /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:260:13 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:201:9 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:281:31
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::SendToLog()(i32)
                        0x472cd: /re2/abseil-cpp/absl/log/internal/log_message.cc:551:3
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::Flush()(i32)
                        0x471c5: /re2/abseil-cpp/absl/log/internal/log_message.cc:465:3
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::~LogMessage()(i32) i32
                        0x46f9f: /re2/abseil-cpp/absl/log/internal/log_message.cc:299:3
                libcre2-noopt.so.re2::RE2::Init(std::__2::basic_string_view<char, std::__2::char_traits<char>>, re2::RE2::Options const&)(i32,i32,i32)
                        0x6fd7a: /re2/re2/re2.cc:237:7
                libcre2-noopt.so.re2::RE2::RE2(std::__2::basic_string_view<char, std::__2::char_traits<char>>, re2::RE2::Options const&)(i32,i32,i32) i32
                        0x705b6: /re2/re2/re2.cc:158:3
                libcre2-noopt.so.cre2_new(i32,i32,i32) i32
                        0x756c0: /cre2/cre2.cpp:135:27

goroutine 10 [running]:
testing.tRunner.func1.2({0x3b8d7e0, 0xc001a76ec0})
        /Users/anuraag/tmp/go/src/testing/testing.go:1734 +0x21c
testing.tRunner.func1()
        /Users/anuraag/tmp/go/src/testing/testing.go:1737 +0x35e
panic({0x3b8d7e0?, 0xc001a76ec0?})
        /Users/anuraag/tmp/go/src/runtime/panic.go:787 +0x132
github.com/wasilibs/go-re2/internal.newRE(0xc0009e7e00, {0x3b487f8?, 0x0?}, {0x0?, 0x0?, 0x0?, 0x0?})
        /Users/anuraag/git/go-re2/internal/re2_wazero.go:301 +0x269
github.com/wasilibs/go-re2/internal.Compile({0x3b487f8, 0x1}, {0x28?, 0x93?, 0xd7?, 0x3?})
        /Users/anuraag/git/go-re2/internal/re2.go:67 +0x165
github.com/wasilibs/go-re2.Compile(...)
        /Users/anuraag/git/go-re2/re2.go:46
github.com/wasilibs/go-re2.compileTest(0xc000083dc0, {0x3b487f8, 0x1}, {0x3ae0295, 0x2c})
        /Users/anuraag/git/go-re2/all_test.go:51 +0x5a
github.com/wasilibs/go-re2.TestBadCompile(0xc000083dc0)
        /Users/anuraag/git/go-re2/all_test.go:71 +0x45
testing.tRunner(0xc000083dc0, 0x3bdf4b0)
        /Users/anuraag/tmp/go/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
        /Users/anuraag/tmp/go/src/testing/testing.go:1851 +0x413
FAIL    github.com/wasilibs/go-re2      2.604s
FAIL

To Reproduce
Description of the host and wasm code that reproduces the behavior.
Smoothest debugging will be if you can share a repository with the
actual code.

With go 1.24, run go-re2 tests. debug-build branch has unattested debug binary, main will crash without symbols

git clone https://github.com/anuraaga/go-re2
cd go-re2
git checkout debug-build
GOARCH=amd64 go test .

Expected behavior

Doesn't crash

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the relevant information):

  • Go version: [1.24.0]
  • wazero Version: [1.8.2]
  • Host architecture: [amd64]
  • Runtime mode: [compiler]

Additional context

Originally reported in wasilibs/go-re2#190

Dump with host function logging - while I thought it may be only the first host function call that is affected, it appears there is one clock_time_get call that succeeds before the failing one within fwrite.

https://gist.github.com/anuraaga/83ef5402670e51f492bcd4b6f25014e8

Activity

  1. anuraaga commented on Feb 13, 2025

    @anuraaga
    ContributorAuthor

    For reference, I do get a crash on wazero tests using rosetta with go 1.24 that doesn't happen on 1.23

    ok      github.com/tetratelabs/wazero/imports/assemblyscript    (cached)
    ok      github.com/tetratelabs/wazero/imports/assemblyscript/example    (cached)
    ok      github.com/tetratelabs/wazero/imports/emscripten        (cached)
    --- FAIL: Test_fdReaddir_ls (1.98s)
        --- FAIL: Test_fdReaddir_ls/tinygo (0.06s)
            --- FAIL: Test_fdReaddir_ls/tinygo/directory_with_tons_of_entries (0.01s)
                require.go:331: expected no error, but was wasm error: out of bounds memory access
                    wasm stack trace:
                            main.runtime.sliceAppend(i32,i32,i32,i32,i32,i32,i32)
                            main.main.printFileNames(i32,i32)
                            main._start():
                    /Users/anuraag/git/wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:268
                    /Users/anuraag/git/wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:238
                    /Users/anuraag/git/wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:148
    FAIL
    FAIL    github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1    14.352s
    ok      github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1/example    (cached)
    

    But it doesn't seem to happen if I run on a github runner so maybe just rosetta-specific (above error happens on native too)

    https://github.com/anuraaga/wazero/actions/runs/13302580071/job/37146554276?pr=2

  2. corani commented on Feb 13, 2025

    @corani

    But it doesn't seem to happen if I run on a github runner so maybe just rosetta-specific (above error happens on native too)

    Interesting, I tried running gitleaks (where I first reported the issue: gitleaks/gitleaks#1751) in a GitHub Action, and that indeed works. However, locally I'm on WSL2 so this isn't Rosetta-specific (though apparently not reproducible for everyone).

  3. corani commented on Feb 14, 2025

    @corani

    FYI, on WSL2 with both go-1.24.0 and gotip-1.25-5ff7a634e this fails (but works with go-1.23.6):

    $ go test ./imports/wasi_snapshot_preview1/
    --- FAIL: Test_fdReaddir_ls (0.59s)
        --- FAIL: Test_fdReaddir_ls/tinygo (0.03s)
            --- FAIL: Test_fdReaddir_ls/tinygo/empty_directory (0.03s)
                require.go:331: expected no error, but was wasm error: unreachable
                    wasm stack trace:
                            main.runtime._panic(i32,i32)
                            main._start(): panic: fdopendir ./a-: errno 21
    
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:268
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:238
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:104
            --- FAIL: Test_fdReaddir_ls/tinygo/directory_with_entries (0.00s)
                require.go:331: expected no error, but was wasm error: unreachable
                    wasm stack trace:
                            main.runtime._panic(i32,i32)
                            main._start(): panic: fdopendir .: errno 21
    
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:268
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:238
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:118
            --- FAIL: Test_fdReaddir_ls/tinygo/directory_with_tons_of_entries (0.00s)
                require.go:331: expected no error, but was wasm error: unreachable
                    wasm stack trace:
                            main.runtime._panic(i32,i32)
                            main._start(): panic: fdopendir .: errno 21
    
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:268
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:238
                    /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:148
    FAIL
    FAIL    github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1    9.651s
    FAIL
    
  4. anuraaga commented on Feb 14, 2025

    @anuraaga
    ContributorAuthor

    We have found this commit from Go to be the cause of the issues

    golang/go@601ea46#diff-ab6c974d2f954725a8b1b04f85fb0f1d6aa453692188e727b47ffc8ff202603bL46

    Currently we call the runtime's memmove directly but this can cause problems with certain changes to the implementation. The solution to this and forwards compatibility will be to vendor in Go 1.23's memmove implementation and call it instead of the runtime's.

  5. corani commented on Feb 14, 2025

    @corani

    We have found this commit from Go to be the cause of the issues

    Does that mean it's a compiler bug? Or is memmove being used in a sketchy way here? 😉

  6. ncruces commented on Feb 14, 2025

    @ncruces
    Contributor

    There's a lot of unsafe here, some of which breaks unsafe rules (particularly around the non-moving collector assumption), some go:linkname, and more. Especially in the compiler.

    The point isn't that the compiler is impervious to Go runtime changes. The point is that you can self contain those breakages to wazero, and have a reasonably safe and performant way to run external code.

  7. corani commented on Feb 19, 2025

    @corani

    Thanks for the quick turnaround on this one! Much appreciated 🙇‍♂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions