Repository navigation
Description
Summary
On Windows, the vulkan and cpu native runtime artifacts cannot be loaded. LoadLibraryExW fails
with ERROR_PROC_NOT_FOUND (127) on ggml.dll and, consequently, on libllama.dll.
The cause is inside the published artifact: ggml.dll imports 54 symbols from libstdc++-6.dll, and
5 of them are not exported by the libstdc++-6.dll shipped in the same archive.
This reproduces on v0.74.0 and v0.75.0, on freshly downloaded archives that were never installed,
and does not involve any host application — a bare LoadLibraryExW is enough.
The MSVC-built cuda12 artifact is unaffected and loads correctly.
Affected artifacts
| artifact | toolchain | result |
|---|---|---|
meshllm-native-runtime-windows-x86_64-vulkan v0.74.0 |
MinGW | 127 |
meshllm-native-runtime-windows-x86_64-vulkan v0.75.0 |
MinGW | 127 |
meshllm-native-runtime-windows-x86_64-cpu v0.74.0 |
MinGW | 127 |
meshllm-native-runtime-windows-x86_64-cuda12 v0.74.0 |
MSVC | loads OK |
The MinGW artifacts are identifiable by the lib prefix (libllama.dll, libllama-common.dll,
libmtmd.dll) and by shipping libstdc++-6.dll / libgomp-1.dll / libgcc_s_seh-1.dll /
libwinpthread-1.dll. The cuda12 artifact ships llama.dll and no MinGW runtime.
Reproduction
Extract any affected archive and, from a clean process:
Add-Type @"
using System; using System.Runtime.InteropServices;
public class L { [DllImport("kernel32", SetLastError=true, CharSet=CharSet.Unicode)]
public static extern IntPtr LoadLibraryExW(string p, IntPtr h, uint f); }
"@
$d = '<extracted>\meshllm-native-runtime-windows-x86_64-vulkan\lib'
foreach ($dll in 'ggml-base.dll','ggml-cpu.dll','ggml-vulkan.dll','ggml.dll','libllama.dll') {
$h = [L]::LoadLibraryExW((Join-Path $d $dll), [IntPtr]::Zero, 8) # LOAD_WITH_ALTERED_SEARCH_PATH
$e = [Runtime.InteropServices.Marshal]::GetLastWin32Error()
if ($h -ne [IntPtr]::Zero) { "OK $dll" } else { "FAIL $dll err=$e" }
}Observed:
OK ggml-base.dll
OK ggml-cpu.dll
OK ggml-vulkan.dll
FAIL ggml.dll err=127
FAIL libllama.dll err=127
LOAD_WITH_ALTERED_SEARCH_PATH (0x8) is required so the archive's own lib directory resolves
siblings; without it every DLL fails with 126 instead, which is expected and not the issue here.
Root cause
Resolving ggml.dll's import table against the exports of each DLL shipped beside it:
ggml.dll imports:
libgcc_s_seh-1.dll needs 1, exports 128, missing 0
ggml-base.dll needs 9, exports 807, missing 0
ggml-cpu.dll needs 1, exports 539, missing 0
ggml-vulkan.dll needs 1, exports 4488, missing 0
libstdc++-6.dll needs 54, exports 7058, MISSING 5
The five unresolved symbols:
_ZNKSt25__codecvt_utf8_utf16_baseIwE10do_unshiftER9_MbstatetPcS3_RS3_
_ZNKSt25__codecvt_utf8_utf16_baseIwE5do_inER9_MbstatetPKcS4_RS4_PwS6_RS6_
_ZNKSt25__codecvt_utf8_utf16_baseIwE6do_outER9_MbstatetPKwS4_RS4_PcS6_RS6_
_ZNKSt25__codecvt_utf8_utf16_baseIwE9do_lengthER9_MbstatetPKcS4_y
_ZNSt7codecvtIwc9_MbstatetEC2Ey
These are std::codecvt_utf8_utf16<wchar_t> and std::codecvt<wchar_t, char, mbstate_t> members —
the wide-path conversion machinery typically pulled in via std::filesystem on Windows.
std::codecvt_utf8_utf16 is deprecated since C++17, so this looks like ggml.dll having been
compiled against a libstdc++ that still exported these while the DLL packaged alongside it no longer
does — i.e. a toolchain skew between the compile step and the runtime DLL that gets bundled,
rather than a missing file.
Reproduced with:
import pefile
pe = pefile.PE('ggml.dll')
for e in pe.DIRECTORY_ENTRY_IMPORT:
print(e.dll.decode(), [i.name.decode() for i in e.imports if i.name])We also checked two other libstdc++-6.dll builds available on the test machine (Git for Windows'
MinGW, and the copy Buzz bundles). All three are missing all five symbols, so this is not
fixable by substituting a libstdc++ that happens to be on the host — the fix has to be in how the
artifact is built or packaged.
Environment
- Windows 11 Pro 26200, x64
- NVIDIA driver 591.86 / CUDA 13.1
- Reproduced against archives downloaded directly from the v0.74.0 and v0.75.0 releases
Secondary observation — no Blackwell-capable Windows CUDA runtime
Not the subject of this issue, but relevant to anyone hitting the above and reaching for CUDA
instead. From the published native-runtimes.json:
| artifact | toolkit_major |
gpu_arches |
|---|---|---|
windows-x86_64-cuda12 (v0.74.0 and v0.75.0) |
12 | 75, 80, 86, 87, 89, 90 |
linux-x86_64-cuda13 (v0.74.0 and v0.75.0) |
13 | 75, 80, 86, 87, 89, 90, 100, 103, 120, 121 |
sm_120 (Blackwell / RTX 50-series) is published for Linux CUDA 13 only. On a Blackwell card under
Windows the resolver correctly rejects cuda12 on SM architecture and falls through to vulkan,
which then hits the load failure above — so there is currently no working backend on that hardware.
A windows-x86_64-cuda13 artifact would close it.
One further note: the cuda12 artifact fails with 126 on a clean machine because
cudart64_12.dll, cublas64_12.dll and cublasLt64_12.dll are not present — current NVIDIA
drivers ship CUDA 13. Supplying those three (we used the nvidia-cuda-runtime-cu12 and
nvidia-cublas-cu12 wheels) makes all five libraries load. Documenting that dependency, or
bundling those redistributables, would likely resolve a class of "CUDA runtime not found" reports on
sm 75–90 hardware.
Happy to help
We can test a candidate build on Windows 11 / RTX 5060 Ti and report back, including the
import-resolution dump above for any new artifact.
Activity
jason-bohan commentedon Aug 10, 2026
Reproduced the
vulkanfailure exactly as described (error 127 onggml.dllafter all its own deps load clean).One divergence worth flagging: your table shows
cpufailing the same way (127 onggml.dll), tested on v0.74.0. Testing the v0.75.0cpuruntime specifically,ggml.dllactually loads fine for me — the failure is one step later, onllama-common.dll:126 means one of
llama-common.dll's own dependencies is missing from the bundle — not the samecodecvt/libstdc++-6.dllexport problem you found forggml.dll. Confirmed the usual suspects (vcruntime140.dll,vcruntime140_1.dll,msvcp140.dll,concrt140.dll) are all present inSystem32, so it's not a missing system redistributable either. So it looks like v0.75.0'scpuruntime may have picked up a second, distinct missing-dependency bug on top of (or instead of) thecodecvtone — worth checking whether thelibstdc++-6.dllfix here also needs to touchllama-common.dll's link step.Practical impact worth noting for anyone hitting this: falling back to v0.74.0 avoids the load failure but isn't a real workaround for split/mesh-pooling use cases — the coordinator explicitly excludes v0.74.0 peers from Skippy stage-splits (
stage_protocol_generationmismatch, "Upgrade this peer so it advertises current stage protocol support"). So right now there's no working path to a Windows participant in a stage-split on any release.jds1-ai commentedon Aug 10, 2026
Thanks for running this — and your divergence has caught a real over-claim in my issue text, which
I want to correct explicitly rather than bury in a thread.
The
cpu@ v0.75.0 cell was never measuredThe "Affected artifacts" table lists three MinGW rows:
vulkanv0.74.0,vulkanv0.75.0, andcpuv0.74.0. The summary sentence above it says the failure "reproduces onvulkanandcpu,at v0.74.0 and v0.75.0" — which reads as a 2x2. It isn't. The fourth cell was generalised from
the other three, not run. Your result is the first actual measurement of it, and it does not match
what I implied. Corrected:
vulkanv0.74.0127onggml.dllvulkanv0.75.0127onggml.dllcpuv0.74.0127onggml.dllcpuv0.75.0126onllama-common.dllI have not independently confirmed the
126— I don't have that archive on this box, only the0.74.0
vulkanandcuda12trees. I'm happy to download it and run the same bareLoadLibraryExWprobe if a second data point is useful; say the word.
Agreed that it's a distinct bug
126 ERROR_MOD_NOT_FOUNDonllama-common.dllis a different failure from127 ERROR_PROC_NOT_FOUNDonggml.dll: a missing module, not a missing export. Thecodecvt/libstdc++-6.dllexport skew this issue describes does not explain it, so the fix here should notbe assumed to cover it.
The diagnostic that settled the
127should settle this one too — parsellama-common.dll'simport table and resolve each named DLL against (a) the archive's own
lib\directory and(b)
System32, rather than checking the usual redistributables by hand. That is how the fivecodecvtsymbols surfaced; an ASCII scan or a "looks present" check won't distinguish an importfrom an export, and I got that wrong myself earlier in this investigation. Given you've already
ruled out the MSVC redistributables, the likely candidate is a sibling in the archive or another
MinGW runtime DLL that
llama-common.dllpicked up as a link-time dependency at v0.75.0.Given the different mechanism, this may deserve its own issue rather than living under this title,
which names the
ggml.dll/codecvtcause specifically. Happy either way — I'd rather it betracked accurately than tidily.
Sharper detail on the
127cause, since it may help whoever fixes itRe-parsing the export table today with a positive control, the gap is narrower and more specific
than the issue currently states.
libstdc++-6.dll(the copy shipped in thevulkanarchive, 7058exports) does export the
__codecvt_utf8_utf16_basefamily — 267 symbols containingcodecvt,44 containing
utf8_utf16. Both thechar32_t(IDi) andchar16_t(IDs) instantiations arepresent:
The only instantiation missing is
wchar_t(Iw) — same base class, same methods, differingonly in the template parameter. So this isn't a wholesale absence of
codecvtsupport in thebundled runtime; it's specifically the
wchar_tspecialisations, which is consistent withggml.dllhaving been compiled against a libstdc++ that still instantiated them and packagedagainst one that no longer does.
Mentioning the positive control explicitly because "all five missing" reads identically to "I typed
the mangled names wrong" — the 44 sibling
utf8_utf16exports are what rule that out.The stage-split point is the important part
This is new to me and it closes off the last apparent workaround. I had been treating "pin to
v0.74.0" as a degraded-but-usable option; if v0.74.0 peers can't participate in a stage-split at
all, then it isn't one. Combined with the other half of this —
sm_120(Blackwell) is publishedonly for
linux-x86_64-cuda13, so on an RTX 50-series Windows box the resolver correctly rejectswindows-x86_64-cuda12(gpu_arches = 75,80,86,87,89,90) and falls through to these MinGWruntimes — the practical position is that there is currently no working Windows participant in a
mesh stage-split on any published release, by either route.
Worth stating plainly in the issue, I think, because it changes the priority: this isn't only a
packaging bug on two artifacts, it's the reason a whole platform can't join.
jds1-ai commentedon Aug 11, 2026
Ran the v0.75.0
cpuprobe I offered above. Two corrections, both to things I wrote in my previouscomment.
1. The
126is a probe artifact, not a packaging bug.llama-common.dllimportsllama.dll, asibling inside the archive. Loaded by full path without
LOAD_WITH_ALTERED_SEARCH_PATH, the loadernever searches the DLL's own directory, so the sibling can't resolve:
llama-common.dll8(the repro in the issue body)0(plainLoadLibraryExW)That also accounts for
llama.dllloading fine immediately after — it simply isn't in memory yetwhen
llama-commonneeds it. Generally:127(ERROR_PROC_NOT_FOUND) is a missing export and areal defect;
126(ERROR_MOD_NOT_FOUND) is a missing module, which a search-path choice alsoproduces.
2. I mislabelled the toolchain, and that's the actual news. I put
cpuv0.75.0 in the table asMinGW. It isn't — upstream rebuilt the Windows
cpuruntime with MSVC between v0.74.0 andv0.75.0, and it loads clean. By import table:
ggml.dllimportsvulkanlibgcc_s_seh-1.dll,libstdc++-6.dllvulkanlibgcc_s_seh-1.dll,libstdc++-6.dllcpumsvcp140,vcruntime140,vcruntime140_1The v0.75.0
cpuarchive also drops thelibprefix and ships no MinGW runtime DLLs — which is theheuristic already stated in this issue's body, so it was checkable without downloading anything. The
switch is
cpu-only; I re-downloaded v0.75.0vulkanto confirm it's still MinGW and still127.Corrected matrix:
vulkanv0.74.0127vulkanv0.75.0127cpuv0.74.0127cpuv0.75.0Retracting my closing line. I wrote "there is currently no working Windows participant in a mesh
stage-split on any published release." That's wrong — a loadable Windows runtime exists as of
v0.75.0. It should have said there's no working GPU path:
vulkanis still MinGW and fails at bothversions, and
sm_120is still published only forlinux-x86_64-cuda13.Caveats so this isn't over-read: loading the DLLs is not an end-to-end share-compute test, and
consumers pinned to v0.74.0 don't get this —
skippy_abidiffers across versions (0.1.32 / 0.1.35)and is matched exactly, so the pin has to move as a pair.
Suggest narrowing this issue to the MinGW-built runtimes —
vulkanat both versions,cpuatv0.74.0. If the
cputoolchain switch was deliberate, doing the same forvulkanwould close it out.ndizazzo commentedon Sep 7, 2026
Duplicate of #1160
This reports the same MinGW/libstdc++ ABI mismatch and five missing codecvt exports. I preserved the later version matrix and retractions on #1160.