Skip to content

Windows MinGW native runtimes (vulkan, cpu) fail to load — ggml.dll imports 5 codecvt symbols absent from the bundled libstdc++-6.dll #1215

Closed as duplicate

Description

@jds1-ai

Summary

On Windows, the vulkan and cpu native runtime artifacts cannot be loaded. LoadLibraryExW fails
with ERROR_PROC_NOT_FOUND (127) on ggml.dll and, consequently, on libllama.dll.

The cause is inside the published artifact: ggml.dll imports 54 symbols from libstdc++-6.dll, and
5 of them are not exported by the libstdc++-6.dll shipped in the same archive.

This reproduces on v0.74.0 and v0.75.0, on freshly downloaded archives that were never installed,
and does not involve any host application — a bare LoadLibraryExW is enough.

The MSVC-built cuda12 artifact is unaffected and loads correctly.

Affected artifacts

artifact toolchain result
meshllm-native-runtime-windows-x86_64-vulkan v0.74.0 MinGW 127
meshllm-native-runtime-windows-x86_64-vulkan v0.75.0 MinGW 127
meshllm-native-runtime-windows-x86_64-cpu v0.74.0 MinGW 127
meshllm-native-runtime-windows-x86_64-cuda12 v0.74.0 MSVC loads OK

The MinGW artifacts are identifiable by the lib prefix (libllama.dll, libllama-common.dll,
libmtmd.dll) and by shipping libstdc++-6.dll / libgomp-1.dll / libgcc_s_seh-1.dll /
libwinpthread-1.dll. The cuda12 artifact ships llama.dll and no MinGW runtime.

Reproduction

Extract any affected archive and, from a clean process:

Add-Type @"
using System; using System.Runtime.InteropServices;
public class L { [DllImport("kernel32", SetLastError=true, CharSet=CharSet.Unicode)]
  public static extern IntPtr LoadLibraryExW(string p, IntPtr h, uint f); }
"@
$d = '<extracted>\meshllm-native-runtime-windows-x86_64-vulkan\lib'
foreach ($dll in 'ggml-base.dll','ggml-cpu.dll','ggml-vulkan.dll','ggml.dll','libllama.dll') {
  $h = [L]::LoadLibraryExW((Join-Path $d $dll), [IntPtr]::Zero, 8)  # LOAD_WITH_ALTERED_SEARCH_PATH
  $e = [Runtime.InteropServices.Marshal]::GetLastWin32Error()
  if ($h -ne [IntPtr]::Zero) { "OK    $dll" } else { "FAIL  $dll err=$e" }
}

Observed:

OK    ggml-base.dll
OK    ggml-cpu.dll
OK    ggml-vulkan.dll
FAIL  ggml.dll     err=127
FAIL  libllama.dll err=127

LOAD_WITH_ALTERED_SEARCH_PATH (0x8) is required so the archive's own lib directory resolves
siblings; without it every DLL fails with 126 instead, which is expected and not the issue here.

Root cause

Resolving ggml.dll's import table against the exports of each DLL shipped beside it:

ggml.dll imports:
  libgcc_s_seh-1.dll   needs 1,  exports 128,  missing 0
  ggml-base.dll        needs 9,  exports 807,  missing 0
  ggml-cpu.dll         needs 1,  exports 539,  missing 0
  ggml-vulkan.dll      needs 1,  exports 4488, missing 0
  libstdc++-6.dll      needs 54, exports 7058, MISSING 5

The five unresolved symbols:

_ZNKSt25__codecvt_utf8_utf16_baseIwE10do_unshiftER9_MbstatetPcS3_RS3_
_ZNKSt25__codecvt_utf8_utf16_baseIwE5do_inER9_MbstatetPKcS4_RS4_PwS6_RS6_
_ZNKSt25__codecvt_utf8_utf16_baseIwE6do_outER9_MbstatetPKwS4_RS4_PcS6_RS6_
_ZNKSt25__codecvt_utf8_utf16_baseIwE9do_lengthER9_MbstatetPKcS4_y
_ZNSt7codecvtIwc9_MbstatetEC2Ey

These are std::codecvt_utf8_utf16<wchar_t> and std::codecvt<wchar_t, char, mbstate_t> members —
the wide-path conversion machinery typically pulled in via std::filesystem on Windows.
std::codecvt_utf8_utf16 is deprecated since C++17, so this looks like ggml.dll having been
compiled against a libstdc++ that still exported these while the DLL packaged alongside it no longer
does — i.e. a toolchain skew between the compile step and the runtime DLL that gets bundled,
rather than a missing file.

Reproduced with:

import pefile
pe = pefile.PE('ggml.dll')
for e in pe.DIRECTORY_ENTRY_IMPORT:
    print(e.dll.decode(), [i.name.decode() for i in e.imports if i.name])

We also checked two other libstdc++-6.dll builds available on the test machine (Git for Windows'
MinGW, and the copy Buzz bundles). All three are missing all five symbols, so this is not
fixable by substituting a libstdc++ that happens to be on the host — the fix has to be in how the
artifact is built or packaged.

Environment

  • Windows 11 Pro 26200, x64
  • NVIDIA driver 591.86 / CUDA 13.1
  • Reproduced against archives downloaded directly from the v0.74.0 and v0.75.0 releases

Secondary observation — no Blackwell-capable Windows CUDA runtime

Not the subject of this issue, but relevant to anyone hitting the above and reaching for CUDA
instead. From the published native-runtimes.json:

artifact toolkit_major gpu_arches
windows-x86_64-cuda12 (v0.74.0 and v0.75.0) 12 75, 80, 86, 87, 89, 90
linux-x86_64-cuda13 (v0.74.0 and v0.75.0) 13 75, 80, 86, 87, 89, 90, 100, 103, 120, 121

sm_120 (Blackwell / RTX 50-series) is published for Linux CUDA 13 only. On a Blackwell card under
Windows the resolver correctly rejects cuda12 on SM architecture and falls through to vulkan,
which then hits the load failure above — so there is currently no working backend on that hardware.
A windows-x86_64-cuda13 artifact would close it.

One further note: the cuda12 artifact fails with 126 on a clean machine because
cudart64_12.dll, cublas64_12.dll and cublasLt64_12.dll are not present — current NVIDIA
drivers ship CUDA 13. Supplying those three (we used the nvidia-cuda-runtime-cu12 and
nvidia-cublas-cu12 wheels) makes all five libraries load. Documenting that dependency, or
bundling those redistributables, would likely resolve a class of "CUDA runtime not found" reports on
sm 75–90 hardware.

Happy to help

We can test a candidate build on Windows 11 / RTX 5060 Ti and report back, including the
import-resolution dump above for any new artifact.

Activity

  1. jason-bohan commented on Aug 10, 2026

    @jason-bohan

    Reproduced the vulkan failure exactly as described (error 127 on ggml.dll after all its own deps load clean).

    One divergence worth flagging: your table shows cpu failing the same way (127 on ggml.dll), tested on v0.74.0. Testing the v0.75.0 cpu runtime specifically, ggml.dll actually loads fine for me — the failure is one step later, on llama-common.dll:

    vcomp140.dll      -> ok
    ggml-base.dll     -> ok
    ggml-cpu.dll      -> ok
    ggml.dll          -> ok
    llama-common.dll  -> FAILS, Win32 error 126 (ERROR_MOD_NOT_FOUND)
    llama.dll         -> ok (loads independently right after, despite llama-common failing)
    mtmd.dll          -> ok
    

    126 means one of llama-common.dll's own dependencies is missing from the bundle — not the same codecvt/libstdc++-6.dll export problem you found for ggml.dll. Confirmed the usual suspects (vcruntime140.dll, vcruntime140_1.dll, msvcp140.dll, concrt140.dll) are all present in System32, so it's not a missing system redistributable either. So it looks like v0.75.0's cpu runtime may have picked up a second, distinct missing-dependency bug on top of (or instead of) the codecvt one — worth checking whether the libstdc++-6.dll fix here also needs to touch llama-common.dll's link step.

    Practical impact worth noting for anyone hitting this: falling back to v0.74.0 avoids the load failure but isn't a real workaround for split/mesh-pooling use cases — the coordinator explicitly excludes v0.74.0 peers from Skippy stage-splits (stage_protocol_generation mismatch, "Upgrade this peer so it advertises current stage protocol support"). So right now there's no working path to a Windows participant in a stage-split on any release.

  2. jds1-ai commented on Aug 10, 2026

    @jds1-ai
    Author

    Thanks for running this — and your divergence has caught a real over-claim in my issue text, which
    I want to correct explicitly rather than bury in a thread.

    The cpu @ v0.75.0 cell was never measured

    The "Affected artifacts" table lists three MinGW rows: vulkan v0.74.0, vulkan v0.75.0, and
    cpu v0.74.0. The summary sentence above it says the failure "reproduces on vulkan and cpu,
    at v0.74.0 and v0.75.0" — which reads as a 2x2. It isn't. The fourth cell was generalised from
    the other three, not run. Your result is the first actual measurement of it, and it does not match
    what I implied. Corrected:

    artifact toolchain result measured by
    vulkan v0.74.0 MinGW 127 on ggml.dll me
    vulkan v0.75.0 MinGW 127 on ggml.dll me; independently reproduced by @jason-bohan
    cpu v0.74.0 MinGW 127 on ggml.dll me
    cpu v0.75.0 MinGW 126 on llama-common.dll @jason-bohan

    I have not independently confirmed the 126 — I don't have that archive on this box, only the
    0.74.0 vulkan and cuda12 trees. I'm happy to download it and run the same bare LoadLibraryExW
    probe if a second data point is useful; say the word.

    Agreed that it's a distinct bug

    126 ERROR_MOD_NOT_FOUND on llama-common.dll is a different failure from 127 ERROR_PROC_NOT_FOUND on ggml.dll: a missing module, not a missing export. The codecvt /
    libstdc++-6.dll export skew this issue describes does not explain it, so the fix here should not
    be assumed to cover it.

    The diagnostic that settled the 127 should settle this one too — parse llama-common.dll's
    import table and resolve each named DLL against (a) the archive's own lib\ directory and
    (b) System32, rather than checking the usual redistributables by hand. That is how the five
    codecvt symbols surfaced; an ASCII scan or a "looks present" check won't distinguish an import
    from an export, and I got that wrong myself earlier in this investigation. Given you've already
    ruled out the MSVC redistributables, the likely candidate is a sibling in the archive or another
    MinGW runtime DLL that llama-common.dll picked up as a link-time dependency at v0.75.0.

    Given the different mechanism, this may deserve its own issue rather than living under this title,
    which names the ggml.dll / codecvt cause specifically. Happy either way — I'd rather it be
    tracked accurately than tidily.

    Sharper detail on the 127 cause, since it may help whoever fixes it

    Re-parsing the export table today with a positive control, the gap is narrower and more specific
    than the issue currently states. libstdc++-6.dll (the copy shipped in the vulkan archive, 7058
    exports) does export the __codecvt_utf8_utf16_base family — 267 symbols containing codecvt,
    44 containing utf8_utf16. Both the char32_t (IDi) and char16_t (IDs) instantiations are
    present:

    _ZNKSt25__codecvt_utf8_utf16_baseIDiE5do_inERiPKcS3_RS3_PDiS5_RS5_    present
    _ZNKSt25__codecvt_utf8_utf16_baseIDsE10do_unshiftERiPcS2_RS2_         present
    _ZNKSt25__codecvt_utf8_utf16_baseIwE5do_inER9_MbstatetPKcS4_RS4_PwS6_RS6_   ABSENT
    

    The only instantiation missing is wchar_t (Iw) — same base class, same methods, differing
    only in the template parameter. So this isn't a wholesale absence of codecvt support in the
    bundled runtime; it's specifically the wchar_t specialisations, which is consistent with
    ggml.dll having been compiled against a libstdc++ that still instantiated them and packaged
    against one that no longer does.

    Mentioning the positive control explicitly because "all five missing" reads identically to "I typed
    the mangled names wrong" — the 44 sibling utf8_utf16 exports are what rule that out.

    The stage-split point is the important part

    the coordinator explicitly excludes v0.74.0 peers from Skippy stage-splits
    (stage_protocol_generation mismatch)

    This is new to me and it closes off the last apparent workaround. I had been treating "pin to
    v0.74.0" as a degraded-but-usable option; if v0.74.0 peers can't participate in a stage-split at
    all, then it isn't one. Combined with the other half of this — sm_120 (Blackwell) is published
    only for linux-x86_64-cuda13, so on an RTX 50-series Windows box the resolver correctly rejects
    windows-x86_64-cuda12 (gpu_arches = 75,80,86,87,89,90) and falls through to these MinGW
    runtimes — the practical position is that there is currently no working Windows participant in a
    mesh stage-split on any published release
    , by either route.

    Worth stating plainly in the issue, I think, because it changes the priority: this isn't only a
    packaging bug on two artifacts, it's the reason a whole platform can't join.

  3. jds1-ai commented on Aug 11, 2026

    @jds1-ai
    Author

    Ran the v0.75.0 cpu probe I offered above. Two corrections, both to things I wrote in my previous
    comment.

    1. The 126 is a probe artifact, not a packaging bug. llama-common.dll imports llama.dll, a
    sibling inside the archive. Loaded by full path without LOAD_WITH_ALTERED_SEARCH_PATH, the loader
    never searches the DLL's own directory, so the sibling can't resolve:

    flags llama-common.dll
    8 (the repro in the issue body) ok
    0 (plain LoadLibraryExW) FAILS 126

    That also accounts for llama.dll loading fine immediately after — it simply isn't in memory yet
    when llama-common needs it. Generally: 127 (ERROR_PROC_NOT_FOUND) is a missing export and a
    real defect; 126 (ERROR_MOD_NOT_FOUND) is a missing module, which a search-path choice also
    produces.

    2. I mislabelled the toolchain, and that's the actual news. I put cpu v0.75.0 in the table as
    MinGW. It isn't — upstream rebuilt the Windows cpu runtime with MSVC between v0.74.0 and
    v0.75.0
    , and it loads clean. By import table:

    archive ggml.dll imports toolchain
    v0.74.0 vulkan libgcc_s_seh-1.dll, libstdc++-6.dll MinGW
    v0.75.0 vulkan libgcc_s_seh-1.dll, libstdc++-6.dll MinGW (unchanged)
    v0.75.0 cpu msvcp140, vcruntime140, vcruntime140_1 MSVC

    The v0.75.0 cpu archive also drops the lib prefix and ships no MinGW runtime DLLs — which is the
    heuristic already stated in this issue's body, so it was checkable without downloading anything. The
    switch is cpu-only; I re-downloaded v0.75.0 vulkan to confirm it's still MinGW and still 127.

    Corrected matrix:

    artifact toolchain result
    vulkan v0.74.0 MinGW 127
    vulkan v0.75.0 MinGW 127
    cpu v0.74.0 MinGW 127
    cpu v0.75.0 MSVC loads clean

    Retracting my closing line. I wrote "there is currently no working Windows participant in a mesh
    stage-split on any published release." That's wrong — a loadable Windows runtime exists as of
    v0.75.0. It should have said there's no working GPU path: vulkan is still MinGW and fails at both
    versions, and sm_120 is still published only for linux-x86_64-cuda13.

    Caveats so this isn't over-read: loading the DLLs is not an end-to-end share-compute test, and
    consumers pinned to v0.74.0 don't get this — skippy_abi differs across versions (0.1.32 / 0.1.35)
    and is matched exactly, so the pin has to move as a pair.

    Suggest narrowing this issue to the MinGW-built runtimes — vulkan at both versions, cpu at
    v0.74.0. If the cpu toolchain switch was deliberate, doing the same for vulkan would close it out.

  4. ndizazzo commented on Sep 7, 2026

    @ndizazzo
    Collaborator

    Duplicate of #1160

    This reports the same MinGW/libstdc++ ABI mismatch and five missing codecvt exports. I preserved the later version matrix and retractions on #1160.

  5. added
    duplicateThis issue or pull request already exists
    on Sep 7, 2026
  6. added theissue type on Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions