GH-46797: [C++] Support Meson on Windows #47282

WillAyd · 2025-08-07T20:19:02Z

Rationale for this change

This allows users on the Windows platform to use the Meson build system

What changes are included in this PR?

Updates to the Meson configurations and source files to accommodate Windows

Are these changes tested?

Yes

Are there any user-facing changes?

No

GitHub Issue: [C++][CI] Add Windows CI Job for Meson #46797

github-actions · 2025-08-07T20:19:29Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

WillAyd · 2025-08-07T20:20:36Z

cpp/src/arrow/compute/kernels/ree_util_internal.h

@@ -347,7 +347,7 @@ Result<std::shared_ptr<ArrayData>> PreallocateRunEndsArray(
 /// \param has_validity_buffer a validity buffer must be allocated
 /// \param length the length of the values array
 /// \param data_buffer_size the size of the data buffer for string and binary types
-ARROW_COMPUTE_EXPORT Result<std::shared_ptr<ArrayData>> PreallocateValuesArray(
+ARROW_EXPORT Result<std::shared_ptr<ArrayData>> PreallocateValuesArray(


Although this file is in the compute/kernels tree, the associated module is built as part of the arrow lib irrespective of whether compute is enabled or not. As such, I figured it made more sense to use the EXPORT semantics from the arrow lib rather than arrow_compute

Really?
It seems that it's used by libarrow_compute not libarrow:

arrow/cpp/src/arrow/CMakeLists.txt

Line 767 in 97d7cdb

compute/kernels/ree_util_internal.cc

Ah you are right - I can fix this

cpp/src/arrow/compute/CMakeLists.txt

WillAyd · 2025-08-07T20:23:16Z

cpp/src/arrow/util/meson.build

@@ -224,18 +222,25 @@ utility_test_srcs = [
 ]

 if host_machine.system() == 'windows'
+    # meson does not natively support manifest files like CMake does


I don't think the Meson support for manifest files is as nice as CMake's, but I'm also not super familiar with them. Need to research this a bit more...

mesonbuild/meson#13041 may be related.

/MANIFEST:EMBED /MANIFESTINPUT:io_util_test.manifest linker options may work.
See also: https://learn.microsoft.com/en-us/cpp/build/reference/manifestinput-specify-manifest-input?view=msvc-170

kou

Could you open an issue for this?

kou · 2025-08-08T05:36:25Z

.github/workflows/cpp_extra.yml

+          key: cpp-ccache-windows-${{ env.CACHE_VERSION }}-${{ hashFiles('cpp/**') }}
+          restore-keys: cpp-ccache-windows-${{ env.CACHE_VERSION }}-


Suggested change

key: cpp-ccache-windows-${{ env.CACHE_VERSION }}-${{ hashFiles('cpp/**') }}

restore-keys: cpp-ccache-windows-${{ env.CACHE_VERSION }}-

key: cpp-ccache-windows-meson-${{ env.CACHE_VERSION }}-${{ hashFiles('cpp/**') }}

restore-keys: cpp-ccache-windows-meson-${{ env.CACHE_VERSION }}-

kou · 2025-08-08T05:37:10Z

.github/workflows/cpp_extra.yml

+        shell: cmd
+        run: |
+          call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
+          bash -c "ARROW_USE_MESON=ON; ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}"


Suggested change

bash -c "ARROW_USE_MESON=ON; ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}"

bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build"

kou · 2025-08-08T05:37:20Z

.github/workflows/cpp_extra.yml

+        run: |
+          # For ORC
+          export TZDIR=/c/msys64/usr/share/zoneinfo
+          ARROW_USE_MESON=ON ci/scripts/cpp_test.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}


Suggested change

ARROW_USE_MESON=ON ci/scripts/cpp_test.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}

ci/scripts/cpp_test.sh $(pwd) $(pwd)/build

kou · 2025-08-08T05:37:43Z

.github/workflows/cpp_extra.yml

+          # We can invalidate the current cache by updating this.
+          CACHE_VERSION: "2022-09-13"
+      - name: Build
+        shell: cmd


Suggested change

shell: cmd

shell: cmd

env:

ARROW_USE_MESON: ON

.github/workflows/cpp_extra.yml

kou · 2025-08-08T05:42:23Z

cpp/src/arrow/compute/kernels/ree_util_internal.h

@@ -347,7 +347,7 @@ Result<std::shared_ptr<ArrayData>> PreallocateRunEndsArray(
 /// \param has_validity_buffer a validity buffer must be allocated
 /// \param length the length of the values array
 /// \param data_buffer_size the size of the data buffer for string and binary types
-ARROW_COMPUTE_EXPORT Result<std::shared_ptr<ArrayData>> PreallocateValuesArray(
+ARROW_EXPORT Result<std::shared_ptr<ArrayData>> PreallocateValuesArray(


Really?
It seems that it's used by libarrow_compute not libarrow:

arrow/cpp/src/arrow/CMakeLists.txt

Line 767 in 97d7cdb

compute/kernels/ree_util_internal.cc

kou · 2025-08-08T05:44:08Z

cpp/src/arrow/meson.build

+if host_machine.system() != 'windows'
+  dl_dep = dependency('dl')
+else
+  dl_dep = declare_dependency()
+endif


Could you swap these branches for easy to read? (In general, no "not" is easier to read.)

Suggested change

if host_machine.system() != 'windows'

dl_dep = dependency('dl')

else

dl_dep = declare_dependency()

endif

if host_machine.system() == 'windows'

dl_dep = declare_dependency()

else

dl_dep = dependency('dl')

endif

kou · 2025-08-08T05:46:21Z

cpp/src/arrow/meson.build

 )

+arrow_compile_args = []
+if host_machine.system() == 'windows' and get_option('default_library') != 'shared'


We can always define ARROW_STATIC:

Suggested change

if host_machine.system() == 'windows' and get_option('default_library') != 'shared'

if get_option('default_library') != 'shared'

BTW, does this work with default_library=both?

I haven't gotten default_library=both to work yet, but haven't put a ton of effort into debugging it either. Happy to do so after getting shared/static support stabilized

kou · 2025-08-08T05:50:42Z

cpp/src/arrow/meson.build

@@ -484,14 +505,22 @@ if needs_compute
    arrow_compute_lib = library(
        'arrow-compute',
        sources: arrow_compute_lib_sources,
-        dependencies: arrow_dep,
+        dependencies: [arrow_dep] + int128_deps,


Hmm. Why do we need this?

If libarrow is a static library, it must have int128_deps dependency. libarrow consumers should not care about int128_deps.

This is to build the library itself, not to expose it to consumers. I believe that MSVC/Windows do not offer a native 128 bit integer, so the int128_deps grabs it from the boost multiprecision header. On other platforms, this dependency is empty

Ah, compute also uses arrow::internal::int128_t. Sorry. I thought that decimal in libarrow only uses arrow::internal::int128_t.

kou · 2025-08-08T06:07:06Z

cpp/src/arrow/util/meson.build

@@ -224,18 +222,25 @@ utility_test_srcs = [
 ]

 if host_machine.system() == 'windows'
+    # meson does not natively support manifest files like CMake does


mesonbuild/meson#13041 may be related.

/MANIFEST:EMBED /MANIFESTINPUT:io_util_test.manifest linker options may work.
See also: https://learn.microsoft.com/en-us/cpp/build/reference/manifestinput-specify-manifest-input?view=msvc-170

github-actions · 2025-08-08T13:31:27Z

⚠️ GitHub issue #46797 has been automatically assigned in GitHub to PR creator.

WillAyd · 2025-08-08T13:31:43Z

Could you open an issue for this?

Sorry about that - there is an open issue I just messed up the commit name. I've updated the title now and can squash on next push

WillAyd · 2025-08-08T15:14:15Z

.github/workflows/cpp_extra.yml

+          call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64
+          echo %PATH%
+          dir "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64"
+          bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options || '' }}"


I believe the current error has to do with the fact that calling `bash -c link ..." invokes the linker that comes bundled with bash, and not the linker that the vcvarsall.bat script is supposed to activate. You can reproduce this locally:

> link /help Microsoft (R) Incremental Linker Version 14.44.35214.0 Copyright (C) Microsoft Corporation. All rights reserved. For help on Linker, type `link /link' or `link' For help on Library Manager, type `link /lib' or `lib' For help on Dumper, type `link /dump' or `dumpbin' For help on Editor, type `link /edit' or `editbin' For help on CvtCIL, type `link /cvtcil' For help on PushThunkObj Generator, type `link /pushthunkobj' > bash -c "link /help" link: missing operand after ‘/help’ Try 'link --help' for more information. > bash -c "link --help" Usage: link FILE1 FILE2 or: link OPTION Call the link function to create a link named FILE2 to an existing FILE1. --help display this help and exit --version output version information and exit GNU coreutils online help: <https://www.gnu.org/software/coreutils/> Report any translation bugs to <https://translationproject.org/team/> Full documentation <https://www.gnu.org/software/coreutils/link> or available locally via: info '(coreutils) link invocation'

I guess the current CMake configuration doesn't care about this, although it makes sense for Meson to be suspect of this environment.

Maybe its worth creating a separate cpp_build.bat script for Windows platforms to use? What do you think @kou ?

How about trying meson setup --vsenv instead of calling vcvarsall.bat manually?

https://mesonbuild.com/Using-with-Visual-Studio.html

If no Visual Studio Command Prompt was detected, and no mingw compilers are detected either, meson will attempt to find "a" Visual Studio installation for you automatically, by asking Microsoft's "vswhere" program. If you want to ignore mingw compilers, pass the --vsenv option on the meson command line. If you need to guarantee a specific Visual Studio version, set it up manually.

WillAyd · 2025-09-18T16:32:54Z

cpp/src/parquet/meson.build

+thrift_dep = dependency(
+    'thrift',
+    allow_fallback: false,
+    method: host_machine.system() == 'windows' ? 'cmake' : 'auto',


This appears to be an upstream issue where the library names used in the pkg-config files do not match the library names distributed on windows. See https://issues.apache.org/jira/browse/THRIFT-5894

WillAyd · 2025-09-24T19:39:08Z

Still looking at this - unfortunately it looks like Thrift must be located on the system (hence the conda usage), but the grpc/libprotobuf used in conda don't seem to play nicely with the symbol visibility expectations on Windows.

WillAyd · 2025-10-23T13:16:27Z

I am still looking at this, but I think it will take a bit longer unfortunately. gRPC has some custom library naming conventions on Windows that don't appear reflected in the upstream Meson wrap entry. Its not possible to use the conda packaged gRPC libraries because they have symbol visibility issues on Windows, so the former will likely need to be fixed before this can continue

WillAyd requested review from assignUser, jonkeane, kou and raulcd as code owners August 7, 2025 20:19

github-actions bot added Component: C++ awaiting review Awaiting review labels Aug 7, 2025

WillAyd commented Aug 7, 2025

View reviewed changes

WillAyd added the CI: Extra Run extra CI label Aug 7, 2025

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 7, 2025

WillAyd mentioned this pull request Aug 7, 2025

[C++] 32 bit MSVC doesn't build #32111

Open

kou reviewed Aug 8, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Aug 8, 2025

WillAyd changed the title ~~Meson win support~~ GH-46797: [C++] Support Meson on Windows Aug 8, 2025

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 8, 2025

WillAyd force-pushed the meson-win-support branch 2 times, most recently from c3b74be to d7c42c4 Compare August 8, 2025 14:55

WillAyd commented Aug 8, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Aug 9, 2025

WillAyd force-pushed the meson-win-support branch 3 times, most recently from aca8728 to 39c6cf0 Compare August 11, 2025 19:27

WillAyd force-pushed the meson-win-support branch 2 times, most recently from 5a1752d to e74984e Compare September 18, 2025 14:38

github-actions bot added the Component: Parquet label Sep 18, 2025

WillAyd commented Sep 18, 2025

View reviewed changes

WillAyd force-pushed the meson-win-support branch 3 times, most recently from 1b564ca to 48805dd Compare September 24, 2025 18:53

WillAyd force-pushed the meson-win-support branch from 48805dd to 4322abe Compare October 6, 2025 19:31

WillAyd added 17 commits October 7, 2025 17:04

apacheGH-46797: [C++] Support Meson on Windows

734476c

kou feedback

60e998e

Force grpc subproject usage

95af2dd

Remove Boost crc

4b54c81

Add compat for grpc system versus subproject detection

4410e87

Try crc compat on Windows

c94bc22

Simplified condition

bfa77a8

Simplify export macros

2f4cdfd

Refactor gRPC options to subproject

e1c7889

Always use URI_STATIC_BUILD

3fae2a7

Try installing conda in CI

ff31bdc

Change != 'shared' to == 'static'

2439882

Fix parquet symbol visibility on Windows

d69b4d3

try cmake detection method on windows

206430b

Skip parquet encryption example compilation on MSVC

ee1badd

Fix symbol visibility issue with flight

76902fe

Use grpc as subproject on Windows

fc1c08b

WillAyd force-pushed the meson-win-support branch from 4322abe to fc1c08b Compare October 8, 2025 11:43

WillAyd mentioned this pull request Oct 20, 2025

GH-47836: [C++] Fix Meson configuration after bpacking changes #47837

Merged

WillAyd mentioned this pull request Nov 13, 2025

GH-36411: [C++][Python] Use meson-python for PyArrow build system #45854

Open

		key: cpp-ccache-windows-${{ env.CACHE_VERSION }}-${{ hashFiles('cpp/**') }}
		restore-keys: cpp-ccache-windows-${{ env.CACHE_VERSION }}-

	bash -c "ARROW_USE_MESON=ON; ci/scripts/cpp_build.sh $(pwd) $(pwd)/build ${{ matrix.run-options \|\| '' }}"
	bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build"

	ARROW_USE_MESON=ON ci/scripts/cpp_test.sh $(pwd) $(pwd)/build ${{ matrix.run-options \|\| '' }}
	ci/scripts/cpp_test.sh $(pwd) $(pwd)/build

	if host_machine.system() == 'windows' and get_option('default_library') != 'shared'
	if get_option('default_library') != 'shared'

GH-46797: [C++] Support Meson on Windows #47282

Are you sure you want to change the base?

GH-46797: [C++] Support Meson on Windows #47282

Uh oh!

Conversation

WillAyd commented Aug 7, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Aug 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 8, 2025

Uh oh!

WillAyd commented Aug 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillAyd commented Sep 24, 2025

Uh oh!

WillAyd commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

WillAyd commented Aug 7, 2025 •

edited by github-actions bot

Loading