Skip to content

Commit 7f7c703

Browse files
committed
Copy a tensor back from the device before ETDump writes it
ETDump records an intermediate tensor by handing the tensor's data pointer to a data sink, and every sink reads those bytes with a plain host read: memcpy(cur_data_begin, ptr, length); When the tensor lives on an accelerator that pointer is not host memory, so the read segfaults. A program placed on CUDA crashes as soon as tracing is turned on, which is exactly when someone is trying to debug it. Returning an error instead would not help. All four callers wrap the result in ET_CHECK_MSG, so an error aborts the process rather than skipping the tensor. This change brings the data back to host memory first. When the tensor is not on CPU, ETDump looks up the allocator registered for that device type, stages the bytes into a temporary host buffer with copy_device_to_host, writes that buffer to the sink and frees it. A tensor on CPU keeps the old path and copies nothing extra. If no allocator is registered for the device, ETDump now reports NotFound and logs the device type instead of reading the pointer anyway. Test plan: Two new test files, each with a CMake target and a Buck target. devtools/etdump/tests/etdump_device_test.cpp registers the existing MockCudaAllocator, which backs its device memory with host memory, and has two tests. One logs a tensor tagged as CUDA and checks both that ETDump went through the allocator and that the bytes reached the debug buffer. The other logs a tensor on CPU and checks that the allocator was not used at all, so the CPU path is unchanged. devtools/etdump/tests/etdump_device_no_allocator_test.cpp covers the case where nothing is registered for the device. The registry is a process wide static with no way to remove an entry, so that case needs a binary that never registers anything, which is why it is a second file. devtools/etdump/tests/CMakeLists.txt was not referenced by any parent CMakeLists, so nothing in that directory was built by CMake. This adds add_subdirectory(tests) to devtools/etdump/CMakeLists.txt under BUILD_TESTING, so both new tests are picked up by ctest, which is how the C++ tests run. The pre-existing sdk_etdump_tests target stays out of the CMake build. It compiles etdump_test.cpp, which includes etdump_filter.h, which needs re2, and the devtools build does not pull re2 in. It is now guarded on re2 being available rather than being silently unreachable. With this change both new binaries pass under ctest: 1/2 Test #1: etdump_device_test ................ Passed 2/2 Test #2: etdump_device_no_allocator_test ... Passed With etdump_flatcc.cpp reverted to the old code and everything rebuilt, both fail: Expected equality of these values: g_mock_cuda.d2h_count_ Which is: 0 1 Death test: etdump_gen.log_evalue(EValue(tensor)) Result: failed to die. Also reproduced the real crash on one NVIDIA H100, with a small program that allocates through cudaMalloc, tags a tensor as CUDA and logs it: before: Segmentation fault (core dumped) after: debug buffer holds 1.5 2.5 3.5 4.5 Checked that etdump_flatcc.cpp still compiles with -DUSE_ATEN_LIB. clang-format reports no changes needed on the four touched C++ files. Not covered: The ATen mode branch of the device type conversion only compiles. There is no ATen mode CMake build to run it in, and an ATen mode tensor in ExecuTorch carries no device metadata today, so the branch has no caller that can reach it.
1 parent 368a849 commit 7f7c703

8 files changed

Lines changed: 353 additions & 10 deletions

File tree

‎devtools/etdump/CMakeLists.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,3 +92,7 @@ install(
9292
INCLUDES
9393
DESTINATION ${_common_include_directories}
9494
)
95+
96+
if(BUILD_TESTING)
97+
add_subdirectory(tests)
98+
endif()

‎devtools/etdump/etdump_flatcc.cpp‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
#include <executorch/devtools/etdump/etdump_schema_flatcc_builder.h>
1616
#include <executorch/devtools/etdump/etdump_schema_flatcc_reader.h>
1717
#include <executorch/devtools/etdump/utils.h>
18+
#include <executorch/runtime/core/device_allocator.h>
1819
#include <executorch/runtime/core/error.h>
1920
#include <executorch/runtime/core/exec_aten/exec_aten.h>
2021
#include <executorch/runtime/core/exec_aten/util/scalar_type_util.h>
@@ -105,6 +106,25 @@ Result<etdump_Tensor_ref_t> add_tensor_entry(
105106
return etdump_Tensor_end(builder_);
106107
}
107108

109+
// The allocator registry is keyed on the runtime's own device type. In ATen
110+
// mode a tensor carries the ATen enum instead, which names many more devices
111+
// than ExecuTorch can register an allocator for.
112+
Result<::executorch::runtime::etensor::DeviceType> to_runtime_device_type(
113+
::executorch::aten::DeviceType type) {
114+
#ifdef USE_ATEN_LIB
115+
switch (type) {
116+
case c10::DeviceType::CPU:
117+
return ::executorch::runtime::etensor::DeviceType::CPU;
118+
case c10::DeviceType::CUDA:
119+
return ::executorch::runtime::etensor::DeviceType::CUDA;
120+
default:
121+
return Error::NotSupported;
122+
}
123+
#else
124+
return type;
125+
#endif
126+
}
127+
108128
} // namespace
109129

110130
// Constructor implementation
@@ -725,6 +745,11 @@ Result<long> ETDumpGen::write_tensor_or_return_error(Tensor tensor) {
725745
if (!data_sink_) {
726746
return Error::InvalidArgument;
727747
}
748+
749+
if (!tensor.device().is_cpu()) {
750+
return write_device_tensor_or_return_error(tensor);
751+
}
752+
728753
Result<size_t> ret =
729754
data_sink_->write(tensor.const_data_ptr(), tensor.nbytes());
730755
if (!ret.ok()) {
@@ -733,5 +758,57 @@ Result<long> ETDumpGen::write_tensor_or_return_error(Tensor tensor) {
733758
return static_cast<long>(ret.get());
734759
}
735760

761+
Result<long> ETDumpGen::write_device_tensor_or_return_error(Tensor tensor) {
762+
// A data sink stores the bytes with a plain host read, so the data of a
763+
// tensor that lives on an accelerator has to be brought back to host memory
764+
// first. Handing the accelerator pointer straight to the sink crashes the
765+
// process.
766+
Result<::executorch::runtime::etensor::DeviceType> device_type =
767+
to_runtime_device_type(tensor.device().type());
768+
if (!device_type.ok()) {
769+
ET_LOG(
770+
Error,
771+
"ETDump cannot read a tensor on device type %d",
772+
static_cast<int>(tensor.device().type()));
773+
return device_type.error();
774+
}
775+
776+
::executorch::runtime::DeviceAllocator* allocator =
777+
::executorch::runtime::get_device_allocator(device_type.get());
778+
if (allocator == nullptr) {
779+
ET_LOG(
780+
Error,
781+
"No device allocator registered for device type %d, so a tensor on that device cannot be copied back to host memory",
782+
static_cast<int>(device_type.get()));
783+
return Error::NotFound;
784+
}
785+
786+
const size_t nbytes = tensor.nbytes();
787+
void* staging = malloc(nbytes);
788+
if (staging == nullptr) {
789+
ET_LOG(
790+
Error, "Failed to allocate %zu bytes to stage a device tensor", nbytes);
791+
return Error::MemoryAllocationFailed;
792+
}
793+
794+
Error copy_error = allocator->copy_device_to_host(
795+
staging,
796+
tensor.const_data_ptr(),
797+
nbytes,
798+
static_cast<::executorch::runtime::etensor::DeviceIndex>(
799+
tensor.device().index()));
800+
if (copy_error != Error::Ok) {
801+
free(staging);
802+
return copy_error;
803+
}
804+
805+
Result<size_t> ret = data_sink_->write(staging, nbytes);
806+
free(staging);
807+
if (!ret.ok()) {
808+
return ret.error();
809+
}
810+
return static_cast<long>(ret.get());
811+
}
812+
736813
} // namespace etdump
737814
} // namespace executorch

‎devtools/etdump/etdump_flatcc.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,11 @@ class ETDumpGen : public ::executorch::runtime::EventTracer {
186186

187187
Result<long> write_tensor_or_return_error(executorch::aten::Tensor tensor);
188188

189+
/// Stages a tensor that lives on an accelerator through host memory before
190+
/// handing it to the data sink, which can only read host pointers.
191+
Result<long> write_device_tensor_or_return_error(
192+
executorch::aten::Tensor tensor);
193+
189194
struct flatcc_builder* builder_;
190195
size_t num_blocks_ = 0;
191196
DataSinkBase* data_sink_;

‎devtools/etdump/targets.bzl‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,7 @@ def define_common_targets():
132132
"etdump_flatcc.h",
133133
],
134134
deps = [
135+
"//executorch/runtime/core:device_allocator",
135136
"//executorch/runtime/platform:platform",
136137
],
137138
exported_deps = [

‎devtools/etdump/tests/CMakeLists.txt‎

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -19,16 +19,29 @@ include(${EXECUTORCH_ROOT}/tools/cmake/Test.cmake)
1919

2020
set(_test_srcs etdump_test.cpp)
2121

22+
# etdump_test.cpp includes etdump_filter.h, which needs re2, and the devtools
23+
# build does not pull re2 in.
24+
if(TARGET re2::re2)
25+
et_cxx_test(
26+
sdk_etdump_tests
27+
SOURCES
28+
${_test_srcs}
29+
EXTRA_LIBS
30+
bundled_program
31+
etdump
32+
flatccrt
33+
)
34+
target_include_directories(
35+
sdk_etdump_tests PRIVATE ${CMAKE_INSTALL_PREFIX}/sdk/include
36+
${EXECUTORCH_ROOT}/third-party/flatcc/include
37+
)
38+
endif()
39+
2240
et_cxx_test(
23-
sdk_etdump_tests
24-
SOURCES
25-
${_test_srcs}
26-
EXTRA_LIBS
27-
bundled_program
28-
etdump
29-
flatccrt
41+
etdump_device_test SOURCES etdump_device_test.cpp EXTRA_LIBS etdump flatccrt
3042
)
31-
target_include_directories(
32-
sdk_etdump_tests PRIVATE ${CMAKE_INSTALL_PREFIX}/sdk/include
33-
${EXECUTORCH_ROOT}/third-party/flatcc/include
43+
44+
et_cxx_test(
45+
etdump_device_no_allocator_test SOURCES etdump_device_no_allocator_test.cpp
46+
EXTRA_LIBS etdump flatccrt
3447
)
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
// The allocator registry is a process wide static with no way to remove an
10+
// entry, so the "nothing registered for this device" path needs a binary that
11+
// never registers anything.
12+
13+
#include <gtest/gtest.h>
14+
#include <cstdint>
15+
#include <cstdlib>
16+
17+
#include <executorch/devtools/etdump/data_sinks/buffer_data_sink.h>
18+
#include <executorch/devtools/etdump/etdump_flatcc.h>
19+
#include <executorch/runtime/core/device_allocator.h>
20+
#include <executorch/runtime/core/portable_type/tensor_impl.h>
21+
#include <executorch/runtime/platform/runtime.h>
22+
#include <executorch/test/utils/DeathTest.h>
23+
24+
using namespace ::executorch::etdump;
25+
using namespace ::executorch::runtime;
26+
using namespace ::executorch::runtime::etensor;
27+
28+
namespace {
29+
30+
TEST(ETDumpNoDeviceAllocatorTest, LogTensorOnUnregisteredDeviceAborts) {
31+
runtime_init();
32+
ASSERT_EQ(get_device_allocator(DeviceType::CUDA), nullptr);
33+
34+
// Host memory, so the test fails by not dying rather than by crashing if
35+
// ETDump ever reads the pointer instead of reporting the missing allocator.
36+
float data[] = {1.5f, 2.5f, 3.5f, 4.5f};
37+
int32_t sizes[] = {4};
38+
uint8_t dim_order[] = {0};
39+
int32_t strides[] = {1};
40+
TensorImpl impl(
41+
ScalarType::Float,
42+
1,
43+
sizes,
44+
data,
45+
dim_order,
46+
strides,
47+
TensorShapeDynamism::STATIC,
48+
DeviceType::CUDA,
49+
0);
50+
Tensor tensor(&impl);
51+
52+
const size_t debug_buf_size = 2048;
53+
void* debug_buf = malloc(debug_buf_size);
54+
auto buffer_data_sink = BufferDataSink::create(debug_buf, debug_buf_size);
55+
ASSERT_TRUE(buffer_data_sink.ok());
56+
57+
ETDumpGen etdump_gen;
58+
etdump_gen.create_event_block("test_block");
59+
etdump_gen.set_data_sink(&buffer_data_sink.get());
60+
61+
ET_EXPECT_DEATH(
62+
etdump_gen.log_evalue(EValue(tensor)), "No device allocator registered");
63+
64+
free(debug_buf);
65+
}
66+
67+
} // namespace
Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
#include <gtest/gtest.h>
10+
#include <cstdint>
11+
#include <cstdlib>
12+
#include <cstring>
13+
14+
#include <executorch/devtools/etdump/data_sinks/buffer_data_sink.h>
15+
#include <executorch/devtools/etdump/etdump_flatcc.h>
16+
#include <executorch/devtools/etdump/etdump_schema_flatcc_builder.h>
17+
#include <executorch/devtools/etdump/etdump_schema_flatcc_reader.h>
18+
#include <executorch/runtime/core/device_allocator.h>
19+
#include <executorch/runtime/core/portable_type/tensor_impl.h>
20+
#include <executorch/runtime/core/test/mock_cuda_allocator.h>
21+
#include <executorch/runtime/platform/runtime.h>
22+
23+
using namespace ::executorch::etdump;
24+
using namespace ::executorch::runtime;
25+
using namespace ::executorch::runtime::etensor;
26+
using namespace ::executorch::runtime::testing;
27+
28+
namespace {
29+
30+
// Backs its device memory with host memory, so the copy path runs without a
31+
// GPU.
32+
MockCudaAllocator g_mock_cuda;
33+
34+
class ETDumpDeviceTest : public ::testing::Test {
35+
protected:
36+
// The registry only accepts allocators with static lifetime and aborts on a
37+
// second registration for the same device type, so register once per binary.
38+
static void SetUpTestSuite() {
39+
register_device_allocator(&g_mock_cuda);
40+
}
41+
42+
void SetUp() override {
43+
runtime_init();
44+
g_mock_cuda.reset();
45+
etdump_gen_ = new ETDumpGen();
46+
debug_buf_ = malloc(kDebugBufSize);
47+
}
48+
49+
void TearDown() override {
50+
delete etdump_gen_;
51+
free(debug_buf_);
52+
}
53+
54+
static constexpr size_t kDebugBufSize = 2048;
55+
56+
ETDumpGen* etdump_gen_ = nullptr;
57+
void* debug_buf_ = nullptr;
58+
int32_t sizes_[1] = {4};
59+
uint8_t dim_order_[1] = {0};
60+
int32_t strides_[1] = {1};
61+
};
62+
63+
TEST_F(ETDumpDeviceTest, LogTensorOnDeviceCopiesItBackToHost) {
64+
ASSERT_EQ(get_device_allocator(DeviceType::CUDA), &g_mock_cuda);
65+
66+
float device_data[] = {1.5f, 2.5f, 3.5f, 4.5f};
67+
TensorImpl impl(
68+
ScalarType::Float,
69+
1,
70+
sizes_,
71+
device_data,
72+
dim_order_,
73+
strides_,
74+
TensorShapeDynamism::STATIC,
75+
DeviceType::CUDA,
76+
0);
77+
Tensor tensor(&impl);
78+
79+
auto buffer_data_sink = BufferDataSink::create(debug_buf_, kDebugBufSize);
80+
ASSERT_TRUE(buffer_data_sink.ok());
81+
82+
etdump_gen_->create_event_block("test_block");
83+
etdump_gen_->set_data_sink(&buffer_data_sink.get());
84+
etdump_gen_->log_evalue(EValue(tensor));
85+
86+
EXPECT_EQ(g_mock_cuda.d2h_count_, 1);
87+
EXPECT_EQ(g_mock_cuda.last_d2h_size_, sizeof(device_data));
88+
EXPECT_EQ(g_mock_cuda.last_d2h_src_, device_data);
89+
EXPECT_EQ(g_mock_cuda.last_d2h_index_, 0);
90+
91+
ETDumpResult result = etdump_gen_->get_etdump_data();
92+
ASSERT_TRUE(result.buf != nullptr);
93+
ASSERT_TRUE(result.size != 0);
94+
95+
size_t size = 0;
96+
void* buf = flatbuffers_read_size_prefix(result.buf, &size);
97+
etdump_ETDump_table_t etdump =
98+
etdump_ETDump_as_root_with_identifier(buf, etdump_ETDump_file_identifier);
99+
etdump_RunData_vec_t run_data_vec = etdump_ETDump_run_data(etdump);
100+
ASSERT_EQ(etdump_RunData_vec_len(run_data_vec), 1);
101+
etdump_Event_vec_t events =
102+
etdump_RunData_events(etdump_RunData_vec_at(run_data_vec, 0));
103+
ASSERT_EQ(etdump_Event_vec_len(events), 1);
104+
etdump_Tensor_table_t logged =
105+
etdump_Value_tensor(etdump_DebugEvent_debug_entry(
106+
etdump_Event_debug_event(etdump_Event_vec_at(events, 0))));
107+
108+
// The sink aligns each blob, so the bytes start at the recorded offset rather
109+
// than at the front of the buffer.
110+
const long offset = etdump_Tensor_offset(logged);
111+
ASSERT_GE(offset, 0);
112+
EXPECT_EQ(
113+
memcmp((uint8_t*)debug_buf_ + offset, device_data, sizeof(device_data)),
114+
0);
115+
116+
free(result.buf);
117+
}
118+
119+
TEST_F(ETDumpDeviceTest, LogTensorOnCpuDoesNotStageThroughTheAllocator) {
120+
float host_data[] = {1.5f, 2.5f, 3.5f, 4.5f};
121+
TensorImpl impl(
122+
ScalarType::Float,
123+
1,
124+
sizes_,
125+
host_data,
126+
dim_order_,
127+
strides_,
128+
TensorShapeDynamism::STATIC,
129+
DeviceType::CPU,
130+
0);
131+
Tensor tensor(&impl);
132+
133+
auto buffer_data_sink = BufferDataSink::create(debug_buf_, kDebugBufSize);
134+
ASSERT_TRUE(buffer_data_sink.ok());
135+
136+
etdump_gen_->create_event_block("test_block");
137+
etdump_gen_->set_data_sink(&buffer_data_sink.get());
138+
etdump_gen_->log_evalue(EValue(tensor));
139+
140+
EXPECT_EQ(g_mock_cuda.d2h_count_, 0);
141+
142+
ETDumpResult result = etdump_gen_->get_etdump_data();
143+
ASSERT_TRUE(result.buf != nullptr);
144+
ASSERT_TRUE(result.size != 0);
145+
free(result.buf);
146+
}
147+
148+
} // namespace

0 commit comments

Comments
 (0)