Skip to content

Commit bdf98fa

Browse files
shawnwang18tensorflower-gardener
authored andcommitted
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
Imported from GitHub PR openxla/xla#45597 ## 📝 Summary of Changes This PR adds the `SKIP_PROFILED` command-buffer update mode. The mode automatically includes command-buffer-referenced preallocated temp allocations in the final VA-remap set while profiling other eligible non-constant allocations during the first three successful executions. Stable profile candidates are combined with the temp allocations so subsequent command-buffer executions can treat the final set as persistent and avoid unnecessary address updates. - Adds `DebugOptions::SKIP_PROFILED` and routes it through the GPU executable buffer-allocation policy. - Adds a per-executor profile state machine to `GpuExecutableVaRemapAllocator`: `kProfiling` transitions exactly once to `kActive` when an automatic temp or safe profile candidate exists, or to `kDisabled` when the final remap set is empty. - Automatically selects preallocated temp buffers and profiles eligible parameter and live-out candidates while filtering unstable addresses, non-VMM-backed parameters, and incompatible shared parameter buffers. - Counts only successful executions toward the profiling window and passes `std::nullopt` for persistent allocation indices while profiling, followed by one absent-to-present transition to the final persistent set. - Uses the parameter-remapping support from #45595 and output-buffer remapping support from #45596. Parameters and outputs remain externally owned in the const owning allocation table; finalized external addresses are mapped into an execution-only allocation-table copy immediately before execution. - Preserves the base allocator fallback when VMM remapping is unavailable or there are neither automatic temp buffers nor stable profile candidates. This PR is stacked on #45596, **[XLA:GPU] Support VA remapping of output buffer allocations**. It is part 3 of 3 and, together with #45595 and #45596, supersedes the closed pre-split PR #45568. ## 🎯 Justification `SKIP_TEMP` can stabilize preallocated temporary buffers, but parameters and live-out buffers may also keep stable physical addresses across repeated executions. Updating command-buffer node addresses for those stable allocations adds avoidable runtime overhead. `SKIP_PROFILED` discovers address stability for each executable/executor pair instead of assuming every dynamic allocation is stable. Workloads with repeatable allocation behavior can therefore stabilize more command-buffer-visible addresses, while unstable or unsupported workloads retain the existing fallback behavior. Mapping finalized owning addresses immediately before execution also keeps donation and copy-protection changes compatible with the const owning-allocation API introduced by the lower PRs. ## 🚀 Kind of Contribution ✨ New Feature ## 📊 Benchmark (for Performance Improvements) No benchmark result is claimed in this draft. The PR adds the policy and correctness coverage; a public-HLO performance benchmark can be added separately when reporting workload-specific speedups. ## 🧪 Unit Tests: - Extended `GpuExecutableTest.CommandBufferAllocationPolicy` expectations for automatic temp-buffer selection in `SKIP_PROFILED`. - Added host-only fake-VMM coverage for profiling, transition, active remapping, and disabled fallback states. - Verifies a temp-only executable transitions to an active persistent temp set, and an unstable parameter cannot disqualify an automatically selected temp. - Verifies stable parameter and output selection, including non-granularity-sized externally owned outputs. - Verifies failed executions do not advance the three-successful-execution profiling window. - Verifies fallback for unstable and non-VMM-backed parameters and incompatible shared parameter buffers. - Verifies copy protection both during profiling and after transition: the owning table retains the finalized external address while execution uses the stable reservation address. ## 🧪 Execution Tests: - Added one-device `VmmTest` coverage using a two-GEMM command-buffer workload across profiling, transition, and remapped executions, checking numerical correctness on every run. - Verifies one-time profile selection, automatic temp inclusion, VA-reservation creation and reuse, and successful deallocation of externally returned output buffers. - Added a non-VMM allocator execution test verifying the workload remains correct and creates no VA reservation when remapping is unavailable. Copybara import of the project: -- f53c11fbb9238a010cdfb8effe8df509a0fb6f31 by Shawn Wang <[email protected]>: [XLA:GPU] Add SKIP_PROFILED command buffer update mode Add SKIP_PROFILED to profile command-buffer-referenced non-constant allocation addresses for three successful executions, then VA-remap the stable candidates so recorded command buffers can skip updating them. The per-executor profile transitions once from profiling to active or disabled, preserving the required absent-to-present persistent-index transition. Active executions allocate selected temps at reservation addresses and map finalized selected input/output addresses into an execution-only BufferAllocations copy, leaving the owning table unchanged. Filter unstable candidates and parameter addresses that are not VMM-backed or are shared by multiple parameter allocations. Add flag parsing, allocation-policy coverage, GPU end-to-end coverage, and fallback behavior without a VMM allocator. -- d76dd3608493868568470916e2f90c67f8683bb2 by Shawn Wang <[email protected]>: Add host-only unit tests for SKIP_PROFILED Cover the SKIP_PROFILED profile state machine with the fake-VMM harness through a test-only profile-candidate seam. Verify stable parameters and outputs transition to execution-only reservation addresses while owning buffers remain external, failed executions do not advance profiling, and unstable, non-VMM-backed, or shared parameter buffers disable remapping. Verify copy-protected addresses are observed after output finalization and can be remapped both during a stable profile and when copy protection first appears after the transition. -- 89d3dd241d75df2e4de0198a50542777e6dc66b2 by Shawn Wang <[email protected]>: Remove profile candidate count accessor -- 51adad438dfea34c10f0c20b2bda9de2e1a3bb7b by Shawn Wang <[email protected]>: [XLA:GPU] Automatically remap temp buffers in SKIP_PROFILED -- e32b96bbcd29211ca08b682050cbd24eeb6401ba by Shawn Wang <[email protected]>: [XLA:GPU] Add direct VA remap allocator includes Merging this change closes #45597 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45597 from shawnwang18:shawnw/cmd_buffer_skip_profiled_split e32b96bbcd29211ca08b682050cbd24eeb6401ba PiperOrigin-RevId: 950773148
1 parent 517854b commit bdf98fa

11 files changed

Lines changed: 1224 additions & 62 deletions

‎third_party/xla/xla/debug_options_flags_test.cc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,8 @@ TEST(DebugOptions, CommandBufferUpdateModesParseFromFlags) {
8989
for (const auto& [name, expected] : std::vector<
9090
std::pair<const char*, DebugOptions::CommandBufferUpdateMode>>{
9191
{"ALWAYS_UPDATE", DebugOptions::ALWAYS_UPDATE},
92-
{"SKIP_TEMP", DebugOptions::SKIP_TEMP}}) {
92+
{"SKIP_TEMP", DebugOptions::SKIP_TEMP},
93+
{"SKIP_PROFILED", DebugOptions::SKIP_PROFILED}}) {
9394
int* pargc;
9495
std::vector<char*>* pargv;
9596
ResetFlagsFromEnvForTesting("XLA_FLAGS", &pargc, &pargv);

‎third_party/xla/xla/pjrt/gpu/se_gpu_pjrt_client_test.cc‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2541,6 +2541,13 @@ CompileOptions SkipTempCommandBufferOptions() {
25412541
return options;
25422542
}
25432543

2544+
CompileOptions SkipProfiledCommandBufferOptions() {
2545+
CompileOptions options = SkipTempCommandBufferOptions();
2546+
options.executable_build_options.mutable_debug_options()
2547+
->set_xla_gpu_command_buffer_update_mode(DebugOptions::SKIP_PROFILED);
2548+
return options;
2549+
}
2550+
25442551
Literal DiagonalMatrix(float diagonal) {
25452552
return LiteralUtil::CreateR2<float>({{diagonal, 0, 0, 0},
25462553
{0, diagonal, 0, 0},
@@ -2588,6 +2595,51 @@ void RunTwoGemmCommandBuffer(PjRtClient& client) {
25882595
}
25892596
}
25902597

2598+
// Runs a two-GEMM command buffer in SKIP_PROFILED mode across enough
2599+
// executions to cover the profiling phase (3 executions), the
2600+
// profile transition, and several remapped executions. Input buffers are
2601+
// created once and reused so parameter addresses stay stable and get selected
2602+
// for VA remapping; the output buffer is freed after each execution so its
2603+
// physical allocation is reused across executions.
2604+
void RunTwoGemmCommandBufferProfiled(PjRtClient& client) {
2605+
static constexpr char kHlo[] = R"(
2606+
HloModule skip_profiled_command_buffer_test
2607+
ENTRY main {
2608+
lhs = f32[4,4] parameter(0)
2609+
rhs0 = f32[4,4] parameter(1)
2610+
rhs1 = f32[4,4] parameter(2)
2611+
temp = f32[4,4] dot(lhs, rhs0),
2612+
lhs_contracting_dims={1}, rhs_contracting_dims={0}
2613+
ROOT output = f32[4,4] dot(temp, rhs1),
2614+
lhs_contracting_dims={1}, rhs_contracting_dims={0}
2615+
})";
2616+
2617+
ASSERT_OK_AND_ASSIGN(
2618+
auto executable,
2619+
CompileExecutable(kHlo, client, SkipProfiledCommandBufferOptions()));
2620+
ASSERT_OK_AND_ASSIGN(auto* memory_space,
2621+
client.addressable_devices()[0]->default_memory_space());
2622+
2623+
const Literal lhs = DiagonalMatrix(1.0f);
2624+
const Literal rhs0 = DiagonalMatrix(2.0f);
2625+
const Literal rhs1 = DiagonalMatrix(3.0f);
2626+
ASSERT_OK_AND_ASSIGN(auto lhs_buffer,
2627+
client.BufferFromHostLiteral(lhs, memory_space));
2628+
ASSERT_OK_AND_ASSIGN(auto rhs0_buffer,
2629+
client.BufferFromHostLiteral(rhs0, memory_space));
2630+
ASSERT_OK_AND_ASSIGN(auto rhs1_buffer,
2631+
client.BufferFromHostLiteral(rhs1, memory_space));
2632+
2633+
for (int run = 0; run < 6; ++run) {
2634+
auto result = executable->Execute(
2635+
{{lhs_buffer.get(), rhs0_buffer.get(), rhs1_buffer.get()}}, {});
2636+
ASSERT_OK_AND_ASSIGN(auto result_literal, ExtractSingleResult(result));
2637+
EXPECT_TRUE(LiteralTestUtil::Near(DiagonalMatrix(6.0f), *result_literal,
2638+
ErrorSpec{1e-5}))
2639+
<< "Mismatch on run " << run;
2640+
}
2641+
}
2642+
25912643
class ScopedBufferAllocatorVLog {
25922644
public:
25932645
ScopedBufferAllocatorVLog()
@@ -2647,6 +2699,53 @@ TEST_F(VmmTest, CommandBufferSkipTempFallsBackWithoutVmmAllocator) {
26472699
mock_log.StopCapturingLogs();
26482700
}
26492701

2702+
TEST_F(VmmTest, CommandBufferSkipProfiledTwoGemmChain) {
2703+
ASSERT_OK_AND_ASSIGN(auto client,
2704+
GetStreamExecutorGpuClient(VmmClientOptions()));
2705+
2706+
ScopedBufferAllocatorVLog vlog;
2707+
absl::ScopedMockLog mock_log(absl::MockLogDefault::kIgnoreUnexpected);
2708+
// The profile transition happens exactly once, combining the automatically
2709+
// selected temp buffer with stable parameter buffers, after which the VA
2710+
// reservation is created.
2711+
EXPECT_CALL(mock_log, Log(absl::LogSeverity::kInfo, ::testing::_,
2712+
::testing::HasSubstr("profile selected")))
2713+
.Times(1);
2714+
EXPECT_CALL(
2715+
mock_log,
2716+
Log(absl::LogSeverity::kInfo, ::testing::_,
2717+
::testing::HasSubstr(
2718+
"reserved range for module skip_profiled_command_buffer_test")))
2719+
.Times(1);
2720+
// Output buffers handed to the caller must be allocator-owned addresses,
2721+
// not per-execution reservation addresses, so releasing them never fails.
2722+
EXPECT_CALL(mock_log, Log(absl::LogSeverity::kError, ::testing::_,
2723+
::testing::HasSubstr("Buffer deallocation failed")))
2724+
.Times(0);
2725+
mock_log.StartCapturingLogs();
2726+
RunTwoGemmCommandBufferProfiled(*client);
2727+
mock_log.StopCapturingLogs();
2728+
}
2729+
2730+
TEST_F(VmmTest, CommandBufferSkipProfiledFallsBackWithoutVmmAllocator) {
2731+
GpuClientOptions options;
2732+
options.allowed_devices = {0};
2733+
options.allocator_config.kind = GpuAllocatorConfig::Kind::kPlatform;
2734+
ASSERT_OK_AND_ASSIGN(auto client, GetStreamExecutorGpuClient(options));
2735+
2736+
ScopedBufferAllocatorVLog vlog;
2737+
absl::ScopedMockLog mock_log(absl::MockLogDefault::kIgnoreUnexpected);
2738+
EXPECT_CALL(
2739+
mock_log,
2740+
Log(absl::LogSeverity::kInfo, ::testing::_,
2741+
::testing::HasSubstr(
2742+
"reserved range for module skip_profiled_command_buffer_test")))
2743+
.Times(0);
2744+
mock_log.StartCapturingLogs();
2745+
RunTwoGemmCommandBufferProfiled(*client);
2746+
mock_log.StopCapturingLogs();
2747+
}
2748+
26502749
#endif // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
26512750

26522751
TEST(StreamExecutorGpuClientTest, MemoryRegistrationDisabledByDefault) {

‎third_party/xla/xla/service/gpu/BUILD‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -719,13 +719,15 @@ cc_library(
719719
"//xla/stream_executor:device_address",
720720
"//xla/stream_executor:device_address_allocator",
721721
"//xla/stream_executor:device_address_vmm_allocator",
722+
"//xla/stream_executor:memory_allocation",
722723
"//xla/stream_executor:memory_reservation",
723724
"//xla/stream_executor:stream",
724725
"//xla/stream_executor:stream_executor_h",
725726
"//xla/tsl/platform:status_macros",
726727
"@com_google_absl//absl/base:core_headers",
727728
"@com_google_absl//absl/container:btree",
728729
"@com_google_absl//absl/container:flat_hash_map",
730+
"@com_google_absl//absl/container:flat_hash_set",
729731
"@com_google_absl//absl/container:node_hash_map",
730732
"@com_google_absl//absl/functional:function_ref",
731733
"@com_google_absl//absl/log",
@@ -763,6 +765,7 @@ xla_cc_test(
763765
"//xla/stream_executor:mock_stream",
764766
"//xla/stream_executor:mock_stream_executor",
765767
"//xla/stream_executor:stream_executor_h",
768+
"//xla/tsl/platform:status_macros",
766769
"@com_google_absl//absl/status",
767770
"@com_google_absl//absl/status:statusor",
768771
"@com_google_absl//absl/types:span",

‎third_party/xla/xla/service/gpu/gpu_executable_buffer_allocator.cc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,7 @@ GpuExecutableBufferAllocator::Create(
9191
module_name, allocations, result_shape, debug_options,
9292
thunk_executor);
9393
case DebugOptions::SKIP_TEMP:
94+
case DebugOptions::SKIP_PROFILED:
9495
return std::make_unique<GpuExecutableVaRemapAllocator>(
9596
module_name, allocations, result_shape, debug_options,
9697
thunk_executor);

‎third_party/xla/xla/service/gpu/gpu_executable_buffer_allocator.h‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -48,11 +48,11 @@ class ThunkExecutor;
4848
//
4949
// This base class implements the ALWAYS_UPDATE command buffer update mode,
5050
// which needs no allocation-address policy beyond global constants. The
51-
// SKIP_TEMP update mode is implemented by GpuExecutableVaRemapAllocator (see
52-
// gpu_executable_va_remap_allocator.h), which assigns stable addresses to
53-
// selected command-buffer allocations via VMM VA remapping. The base-class
54-
// behavior also serves as the runtime fallback for that mode when VA
55-
// remapping is unavailable for an execution.
51+
// SKIP_TEMP and SKIP_PROFILED update modes are implemented by
52+
// GpuExecutableVaRemapAllocator (see gpu_executable_va_remap_allocator.h),
53+
// which assigns stable addresses to selected command-buffer allocations via
54+
// VMM VA remapping. The base-class behavior also serves as the runtime
55+
// fallback for those modes when VA remapping is unavailable for an execution.
5656
class GpuExecutableBufferAllocator {
5757
public:
5858
struct ParameterBuffer {
@@ -157,8 +157,8 @@ class GpuExecutableBufferAllocator {
157157

158158
// Creates the buffer allocator implementing
159159
// `debug_options->xla_gpu_command_buffer_update_mode()`: this class for
160-
// ALWAYS_UPDATE, GpuExecutableVaRemapAllocator for SKIP_TEMP. Check-fails
161-
// on any other mode.
160+
// ALWAYS_UPDATE, GpuExecutableVaRemapAllocator for SKIP_TEMP and
161+
// SKIP_PROFILED. Check-fails on any other mode.
162162
static std::unique_ptr<GpuExecutableBufferAllocator> Create(
163163
absl::string_view module_name,
164164
absl::Span<const BufferAllocation* const> allocations,

‎third_party/xla/xla/service/gpu/gpu_executable_test.cc‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,6 +337,15 @@ TEST_F(GpuExecutableTest, CommandBufferAllocationPolicy) {
337337
EXPECT_EQ(skip_temp_policy.command_buffer_allocation_count, 2);
338338
EXPECT_THAT(skip_temp_policy.persistent_alloc_indices,
339339
Optional(ElementsAre(1)));
340+
341+
// Without a VMM allocator, SKIP_PROFILED falls back to the base scope and
342+
// passes only the constant (1) as persistent.
343+
ASSERT_OK_AND_ASSIGN(
344+
auto skip_profiled_policy,
345+
get_allocation_policy_without_vmm(DebugOptions::SKIP_PROFILED));
346+
EXPECT_EQ(skip_profiled_policy.command_buffer_allocation_count, 2);
347+
EXPECT_THAT(skip_profiled_policy.persistent_alloc_indices,
348+
Optional(ElementsAre(1)));
340349
}
341350

342351
TEST_F(GpuExecutableTest, ComputeComputationLayout) {

0 commit comments

Comments
 (0)