Skip to content

Commit b668595

Browse files
joemarshallkouwestonpacepitrou
authored
GH-35176: [C++] Add support for disabling threading for emscripten (#35672)
As previously discussed in #35176 this is a patch that adds an option `ARROW_ENABLE_THREADING`. When it is turned off, arrow threadpool and serial executors don't spawn threads, and instead run tasks in the main thread when futures are waited for. It doesn't mess with threading in projects included as dependencies, e.g. multithreaded malloc implementations because if you're building for a non threaded environment, you can't use those anyway. Basically where this is at is that it runs the test suite okay, and I think should work well enough to be a backend for pandas on emscripten/pyodide. What this means is: 1) It is possible to use arrow in non-threaded emscripten/webassembly environments (with some build patches specific to emscripten which I'll put in once this is in) 2) Most of arrow just works, albeit slower in parts. Things that don't work and probably won't: 1) Server stuff that relies on threads. Not a massive problem I think because environments with threading restrictions are currently typically also restricted from making servers anyway (i.e. they are web browsers) 2) Anything that relies on actually doing two things at once (for obvious reasons) Things that don't work yet and could be fixed in future: 1) use of asynchronous file/network APIs in emscripten which would mean I/O could work efficiently in one thread. 2) asofjoin - right now the implementation relies on std::thread - it needs refactoring to work with threadpool like everything else in arrow, but I'm not sure I am expert enough in the codebase to do it well. * Closes: #35176 Lead-authored-by: Joe Marshall <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Weston Pace <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
1 parent f2b1c14 commit b668595

25 files changed

Lines changed: 629 additions & 20 deletions

‎ci/scripts/cpp_build.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ cmake \
106106
-DARROW_C_FLAGS_RELWITHDEBINFO="${ARROW_C_FLAGS_RELWITHDEBINFO:-}" \
107107
-DARROW_DATASET=${ARROW_DATASET:-ON} \
108108
-DARROW_DEPENDENCY_SOURCE=${ARROW_DEPENDENCY_SOURCE:-AUTO} \
109+
-DARROW_ENABLE_THREADING=${ARROW_ENABLE_THREADING:-ON} \
109110
-DARROW_ENABLE_TIMING_TESTS=${ARROW_ENABLE_TIMING_TESTS:-ON} \
110111
-DARROW_EXTRA_ERROR_CONTEXT=${ARROW_EXTRA_ERROR_CONTEXT:-OFF} \
111112
-DARROW_FILESYSTEM=${ARROW_FILESYSTEM:-ON} \

‎cpp/cmake_modules/DefineOptions.cmake‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,8 @@ takes precedence over ccache if a storage backend is configured" ON)
196196

197197
define_option(ARROW_WITH_MUSL "Whether the system libc is musl or not" OFF)
198198

199+
define_option(ARROW_ENABLE_THREADING "Enable threading in Arrow core" ON)
200+
199201
#----------------------------------------------------------------------
200202
set_option_category("Test and benchmark")
201203

‎cpp/src/arrow/acero/CMakeLists.txt‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,14 @@ add_arrow_acero_test(hash_join_node_test SOURCES hash_join_node_test.cc
173173
bloom_filter_test.cc)
174174
add_arrow_acero_test(pivot_longer_node_test SOURCES pivot_longer_node_test.cc
175175
test_nodes.cc)
176-
add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc test_nodes.cc)
176+
177+
# asof_join_node uses std::thread internally
178+
# and doesn't use ThreadPool so it will
179+
# be broken if threading is turned off
180+
if(ARROW_ENABLE_THREADING)
181+
add_arrow_acero_test(asof_join_node_test SOURCES asof_join_node_test.cc test_nodes.cc)
182+
endif()
183+
177184
add_arrow_acero_test(tpch_node_test SOURCES tpch_node_test.cc)
178185
add_arrow_acero_test(union_node_test SOURCES union_node_test.cc)
179186
add_arrow_acero_test(aggregate_node_test SOURCES aggregate_node_test.cc)
@@ -221,7 +228,9 @@ if(ARROW_BUILD_BENCHMARKS)
221228
add_arrow_acero_benchmark(project_benchmark SOURCES benchmark_util.cc
222229
project_benchmark.cc)
223230

224-
add_arrow_acero_benchmark(asof_join_benchmark SOURCES asof_join_benchmark.cc)
231+
if(ARROW_ENABLE_THREADING)
232+
add_arrow_acero_benchmark(asof_join_benchmark SOURCES asof_join_benchmark.cc)
233+
endif()
225234

226235
add_arrow_acero_benchmark(tpch_benchmark SOURCES tpch_benchmark.cc)
227236

@@ -244,7 +253,9 @@ if(ARROW_BUILD_BENCHMARKS)
244253
target_link_libraries(arrow-acero-expression-benchmark PUBLIC arrow_acero_static)
245254
target_link_libraries(arrow-acero-filter-benchmark PUBLIC arrow_acero_static)
246255
target_link_libraries(arrow-acero-project-benchmark PUBLIC arrow_acero_static)
247-
target_link_libraries(arrow-acero-asof-join-benchmark PUBLIC arrow_acero_static)
256+
if(ARROW_ENABLE_THREADING)
257+
target_link_libraries(arrow-acero-asof-join-benchmark PUBLIC arrow_acero_static)
258+
endif()
248259
target_link_libraries(arrow-acero-tpch-benchmark PUBLIC arrow_acero_static)
249260
if(ARROW_BUILD_OPENMP_BENCHMARKS)
250261
target_link_libraries(arrow-acero-hash-join-benchmark PUBLIC arrow_acero_static)
@@ -253,7 +264,9 @@ if(ARROW_BUILD_BENCHMARKS)
253264
target_link_libraries(arrow-acero-expression-benchmark PUBLIC arrow_acero_shared)
254265
target_link_libraries(arrow-acero-filter-benchmark PUBLIC arrow_acero_shared)
255266
target_link_libraries(arrow-acero-project-benchmark PUBLIC arrow_acero_shared)
256-
target_link_libraries(arrow-acero-asof-join-benchmark PUBLIC arrow_acero_shared)
267+
if(ARROW_ENABLE_THREADING)
268+
target_link_libraries(arrow-acero-asof-join-benchmark PUBLIC arrow_acero_shared)
269+
endif()
257270
target_link_libraries(arrow-acero-tpch-benchmark PUBLIC arrow_acero_shared)
258271
if(ARROW_BUILD_OPENMP_BENCHMARKS)
259272
target_link_libraries(arrow-acero-hash-join-benchmark PUBLIC arrow_acero_shared)

‎cpp/src/arrow/acero/asof_join_node.cc‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
#include "arrow/type_traits.h"
5050
#include "arrow/util/bit_util.h"
5151
#include "arrow/util/checked_cast.h"
52+
#include "arrow/util/config.h"
5253
#include "arrow/util/future.h"
5354
#include "arrow/util/string.h"
5455

@@ -1707,6 +1708,10 @@ class AsofJoinNode : public ExecNode {
17071708
}
17081709

17091710
Status StartProducing() override {
1711+
#ifndef ARROW_ENABLE_THREADING
1712+
return Status::NotImplemented("ASOF join requires threading enabled");
1713+
#endif
1714+
17101715
ARROW_ASSIGN_OR_RAISE(process_task_, plan_->query_context()->BeginExternalTask(
17111716
"AsofJoinNode::ProcessThread"));
17121717
if (!process_task_.is_valid()) {

‎cpp/src/arrow/acero/bloom_filter.cc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include "arrow/acero/util.h" // PREFETCH
2121
#include "arrow/util/bit_util.h" // Log2
2222
#include "arrow/util/bitmap_ops.h" // CountSetBits
23+
#include "arrow/util/config.h"
2324

2425
namespace arrow {
2526
namespace acero {
@@ -426,6 +427,9 @@ void BloomFilterBuilder_Parallel::CleanUp() {
426427

427428
std::unique_ptr<BloomFilterBuilder> BloomFilterBuilder::Make(
428429
BloomFilterBuildStrategy strategy) {
430+
#ifndef ARROW_ENABLE_THREADING
431+
strategy = BloomFilterBuildStrategy::SINGLE_THREADED;
432+
#endif
429433
switch (strategy) {
430434
case BloomFilterBuildStrategy::SINGLE_THREADED: {
431435
std::unique_ptr<BloomFilterBuilder> impl{new BloomFilterBuilder_SingleThreaded()};

‎cpp/src/arrow/acero/bloom_filter_test.cc‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,8 @@
2929
#include "arrow/acero/util.h"
3030
#include "arrow/compute/key_hash.h"
3131
#include "arrow/util/bitmap_ops.h"
32+
#include "arrow/util/config.h"
33+
#include "arrow/util/cpu_info.h"
3234

3335
namespace arrow {
3436

@@ -468,7 +470,7 @@ TEST(BloomFilter, Basic) {
468470

469471
std::vector<BloomFilterBuildStrategy> strategies;
470472
strategies.push_back(BloomFilterBuildStrategy::SINGLE_THREADED);
471-
#ifndef ARROW_VALGRIND
473+
#if defined(ARROW_ENABLE_THREADING) && !defined(ARROW_VALGRIND)
472474
strategies.push_back(BloomFilterBuildStrategy::PARALLEL);
473475
#endif
474476

@@ -501,7 +503,10 @@ TEST(BloomFilter, Scaling) {
501503
num_build.push_back(4000000);
502504

503505
std::vector<BloomFilterBuildStrategy> strategies;
506+
#ifdef ARROW_ENABLE_THREADING
504507
strategies.push_back(BloomFilterBuildStrategy::PARALLEL);
508+
#endif
509+
strategies.push_back(BloomFilterBuildStrategy::SINGLE_THREADED);
505510

506511
for (const auto hardware_flags : HardwareFlagsForTesting()) {
507512
for (const auto& strategy : strategies) {

‎cpp/src/arrow/acero/plan_test.cc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
#include "arrow/testing/matchers.h"
3737
#include "arrow/testing/random.h"
3838
#include "arrow/util/async_generator.h"
39+
#include "arrow/util/config.h"
3940
#include "arrow/util/logging.h"
4041
#include "arrow/util/macros.h"
4142
#include "arrow/util/thread_pool.h"
@@ -1619,6 +1620,9 @@ TEST(ExecPlan, SourceEnforcesBatchLimit) {
16191620
}
16201621

16211622
TEST(ExecPlanExecution, SegmentedAggregationWithMultiThreading) {
1623+
#ifndef ARROW_ENABLE_THREADING
1624+
GTEST_SKIP() << "Test requires threading enabled";
1625+
#endif
16221626
BatchesWithSchema data;
16231627
data.batches = {ExecBatchFromJSON({int32()}, "[[1]]")};
16241628
data.schema = schema({field("i32", int32())});

‎cpp/src/arrow/acero/task_util.cc‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include <algorithm>
2121
#include <mutex>
2222

23+
#include "arrow/util/config.h"
2324
#include "arrow/util/logging.h"
2425

2526
namespace arrow {
@@ -316,7 +317,11 @@ Status TaskSchedulerImpl::StartScheduling(size_t thread_id, ScheduleImpl schedul
316317
int num_concurrent_tasks,
317318
bool use_sync_execution) {
318319
schedule_impl_ = std::move(schedule_impl);
320+
#ifdef ARROW_ENABLE_THREADING
319321
use_sync_execution_ = use_sync_execution;
322+
#else
323+
use_sync_execution_ = true;
324+
#endif
320325
num_concurrent_tasks_ = num_concurrent_tasks;
321326
num_tasks_to_schedule_.value += num_concurrent_tasks;
322327
return ScheduleMore(thread_id);

‎cpp/src/arrow/acero/task_util.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424

2525
#include "arrow/acero/visibility.h"
2626
#include "arrow/status.h"
27+
#include "arrow/util/config.h"
2728
#include "arrow/util/logging.h"
2829

2930
namespace arrow {

‎cpp/src/arrow/acero/task_util_test.cc‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727

2828
#include "arrow/acero/util.h"
2929
#include "arrow/testing/gtest_util.h"
30+
#include "arrow/util/config.h"
3031
#include "arrow/util/thread_pool.h"
3132

3233
namespace arrow {
@@ -101,6 +102,9 @@ TaskScheduler::TaskGroupContinuationImpl MakeFinalContinuation(
101102
// concurrently. When all groups in that stage finish the next
102103
// stage is started.
103104
TEST(TaskScheduler, Stress) {
105+
#ifndef ARROW_ENABLE_THREADING
106+
GTEST_SKIP() << "Test requires threading support";
107+
#endif
104108
constexpr int kNumThreads = 8;
105109
constexpr int kNumGroups = 8;
106110
constexpr int kGroupsPerStage = 3;
@@ -176,6 +180,9 @@ TEST(TaskScheduler, Stress) {
176180
// thread starts a task group while another thread is finishing
177181
// the last of its tasks.
178182
TEST(TaskScheduler, StressTwo) {
183+
#ifndef ARROW_ENABLE_THREADING
184+
GTEST_SKIP() << "Test requires threading support";
185+
#endif
179186
constexpr int kNumThreads = 16;
180187
constexpr int kNumGroups = 8;
181188
constexpr int kTasksPerGroup = 1;

0 commit comments

Comments
 (0)