Skip to content

Commit 4757a15

Browse files
shawnwang18tensorflower-gardener
authored andcommitted
PR #45595: [XLA:GPU] Support VA remapping of parameter buffer allocations
Imported from GitHub PR openxla/xla#45595 ## 📝 Summary of Changes This PR extends command-buffer virtual-address remapping to entry-computation parameter allocations. Parameter buffers remain caller-owned, so each execution aliases their physical memory into stable VA-reservation addresses instead of transferring ownership to the VMM allocator. - Adds `ExecutionScope::ResolveParameterBuffer`; the base implementation returns the caller buffer, while the VA-remapping implementation installs and returns the stable reservation alias. - Renames the `PrepareReservation` hook to `Prepare` and makes `ExecuteWithBufferAllocations` accept mutable `BufferAllocations`, allowing addresses to be restored after execution. - Tracks per-execution parameter aliases with `RecordStepAlias`. - Releases those aliases after every execution and rewrites remapped entries back to their caller-owned addresses before teardown and result handling. The execution-scope destructor provides cleanup for error paths. - Adds `AddVaRemappedAllocationForTesting` and a host-only fake VMM allocator to exercise the parameter-remapping path without requiring a GPU. This PR is stacked on #45567, **[XLA:GPU] Split command buffer VA remapping into its own allocator class**. It is part 1 of 3 splitting the former #45568 implementation into reviewable changes; #45596 adds output-buffer remapping and #45597 adds the `SKIP_PROFILED` mode. ## 🎯 Justification `SKIP_TEMP` stabilizes only preallocated temporary buffers, but profiled command-buffer update policies also need to stabilize caller-owned parameters whose physical addresses remain constant across executions. Parameter ownership prevents allocating them directly through the command-buffer reservation. Creating a per-execution alias gives recorded command buffers a stable VA while preserving the caller's original address and ownership. Restoring the original address after execution ensures teardown and result handling never observe a reservation address. ## 🚀 Kind of Contribution ✨ New Feature ## 📊 Benchmark (for Performance Improvements) Not applicable as a standalone change. This PR adds parameter-remapping support but does not select it in a production update mode. Performance evaluation belongs to the final `SKIP_PROFILED` policy in #45597. ## 🧪 Unit Tests: - Added host-only `gpu_executable_va_remap_allocator_test.cc` coverage using a fake VMM allocator. - Verifies that a caller-owned parameter receives the same stable reservation address across executions. - Verifies persistent allocation-index handling, post-execution restoration to the caller address, and release of each per-execution mapping. - Covers cleanup behavior without requiring a GPU. ## 🧪 Execution Tests: No standalone execution test is added because no production update mode selects parameter allocations in this PR. End-to-end selection and execution coverage is provided by the stacked `SKIP_PROFILED` change in #45597. Copybara import of the project: -- a0de77a1e85642e79d8b9f5d4d988402280ecfbb by Shawn Wang <[email protected]>: [XLA:GPU] Support VA remapping of parameter buffer allocations Allow command buffer VA remapping to give stable reservation addresses to entry-computation-parameter allocations. Parameter buffers are owned by the caller, so they are aliased into the VA reservation with Map() each execution instead of being allocated through it: - GpuExecutableBufferAllocator::ExecutionScope gains a ResolveParameterBuffer hook (base returns the caller buffer), and ExecuteWithBufferAllocations now takes a mutable BufferAllocations so subclasses can rewrite entries after execution. The Prepare hook is renamed from PrepareReservation. - GpuExecutableVaRemapAllocator records each installed alias (RecordStepAlias) and, after every execution, UnMaps the aliases and rewrites the remapped BufferAllocations entries back to the caller-owned addresses so TearDown and result handling never see reservation addresses (ReleaseStepAliases; the scope destructor is a safety net for error paths). - No production update mode selects parameter allocations yet (SKIP_TEMP remaps only preallocated temp buffers); the AddVaRemappedAllocationForTesting seam plus a host-only fake VMM allocator (gpu_executable_va_remap_allocator_test.cc) exercise the new path: stable reservation address across executions, persistent allocation indices, post-execution rewrite, and mapping release. -- 9e7c7fb688b5e3e13538d5e6f1588faaea428932 by Shawn Wang <[email protected]>: Fix VA remap allocator CI failures -- 7ec80637270003249062139a8fec29202ce79422 by Shawn Wang <[email protected]>: [XLA:GPU] Preserve owning buffers during VA remapping -- b4db679bf775ee4bc2046303d3815488d711c084 by Shawn Wang <[email protected]>: [XLA:GPU] Return remapped buffer allocations by value -- 4d5aabb280d3aab7bbe57754e30cbf47386fd35b by Shawn Wang <[email protected]>: [XLA:GPU] Rename remapped allocation helper Merging this change closes #45595 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45595 from shawnwang18:shawnw/cmd_buffer_va_remap_params 4d5aabb280d3aab7bbe57754e30cbf47386fd35b PiperOrigin-RevId: 948758452
1 parent 07d9840 commit 4757a15

6 files changed

Lines changed: 703 additions & 15 deletions

‎third_party/xla/xla/service/gpu/BUILD‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -740,6 +740,36 @@ cc_library(
740740
],
741741
)
742742

743+
xla_cc_test(
744+
name = "gpu_executable_va_remap_allocator_test",
745+
srcs = ["gpu_executable_va_remap_allocator_test.cc"],
746+
deps = [
747+
":buffer_allocations",
748+
":gpu_executable_buffer_allocator",
749+
"//xla:executable_run_options",
750+
"//xla:shape_util",
751+
"//xla:xla_data_proto_cc",
752+
"//xla:xla_proto_cc",
753+
"//xla/backends/gpu/runtime:thunk",
754+
"//xla/backends/gpu/runtime:thunk_executor",
755+
"//xla/service:buffer_assignment",
756+
"//xla/service:service_executable_run_options",
757+
"//xla/stream_executor:device_address",
758+
"//xla/stream_executor:device_address_allocator",
759+
"//xla/stream_executor:device_address_vmm_allocator",
760+
"//xla/stream_executor:memory_allocation",
761+
"//xla/stream_executor:memory_reservation",
762+
"//xla/stream_executor:mock_platform",
763+
"//xla/stream_executor:mock_stream",
764+
"//xla/stream_executor:mock_stream_executor",
765+
"//xla/stream_executor:stream_executor_h",
766+
"@com_google_absl//absl/status",
767+
"@com_google_absl//absl/status:statusor",
768+
"@com_google_absl//absl/types:span",
769+
"@com_google_googletest//:gtest_main",
770+
],
771+
)
772+
743773
cc_library(
744774
name = "gpu_executable",
745775
srcs = [

‎third_party/xla/xla/service/gpu/gpu_executable_buffer_allocator.cc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,7 @@ GpuExecutableBufferAllocator::ExecutionScope::GenerateBufferAllocations(
227227
run_options->run_options().device_assignment());
228228

229229
const int64_t num_buffers = owner_->allocations_.size();
230-
RETURN_IF_ERROR(PrepareReservation(run_options, device_ordinal));
230+
RETURN_IF_ERROR(Prepare(run_options, device_ordinal));
231231

232232
std::vector<se::DeviceAddressBase> buffers;
233233
buffers.reserve(num_buffers);

‎third_party/xla/xla/service/gpu/gpu_executable_buffer_allocator.h‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -130,8 +130,8 @@ class GpuExecutableBufferAllocator {
130130

131131
// Hook called once per GenerateBufferAllocations before any allocation is
132132
// resolved. The base implementation does nothing.
133-
virtual absl::Status PrepareReservation(
134-
const ServiceExecutableRunOptions* run_options, int device_ordinal) {
133+
virtual absl::Status Prepare(const ServiceExecutableRunOptions* run_options,
134+
int device_ordinal) {
135135
return absl::OkStatus();
136136
}
137137

‎third_party/xla/xla/service/gpu/gpu_executable_va_remap_allocator.cc‎

Lines changed: 183 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,16 @@ GpuExecutableVaRemapAllocator::Remapping::GetReservationOffset(
6767
return it->second;
6868
}
6969

70+
absl::StatusOr<uint64_t>
71+
GpuExecutableVaRemapAllocator::Remapping::GetMappingSize(
72+
BufferAllocation::Index idx) const {
73+
auto it = allocation_to_mapping_size.find(idx);
74+
if (it == allocation_to_mapping_size.end()) {
75+
return Internal("No VA mapping size for allocation %d", idx);
76+
}
77+
return it->second;
78+
}
79+
7080
// Per-run execution scope with command buffer VA remapping active. The
7181
// command-buffer-referenced temp buffer allocations are backed by physical
7282
// VMM allocations while execution sees stable reserved VA addresses. Holds
@@ -85,10 +95,18 @@ class GpuExecutableVaRemapAllocator::VaRemapExecutionScope
8595
vmm_allocator_(vmm_allocator),
8696
remap_lock_(std::move(remap_lock)) {}
8797

98+
// Releases any reservation-address aliases still active for this
99+
// execution. This is a safety net for error paths; the normal release
100+
// happens inside ExecuteWithBufferAllocations.
101+
~VaRemapExecutionScope() override;
102+
88103
bool va_remap_enabled() const override { return true; }
89104

90-
// Passes the persistent allocation indices (constants and VA-remapped temp
91-
// allocations) to `execute`.
105+
// Builds an execution-only address table, maps caller- or allocator-owned
106+
// parameter buffers into their stable reservation addresses, and passes the
107+
// persistent allocation indices (constants and VA-remapped allocations) to
108+
// `execute`. The owning address table remains unchanged for result handling
109+
// and TearDown.
92110
absl::Status ExecuteWithBufferAllocations(
93111
const BufferAllocations& owning_buffer_allocations, int device_ordinal,
94112
absl::FunctionRef<
@@ -98,27 +116,62 @@ class GpuExecutableVaRemapAllocator::VaRemapExecutionScope
98116
execute) override;
99117

100118
private:
101-
absl::Status PrepareReservation(
102-
const ServiceExecutableRunOptions* run_options,
103-
int device_ordinal) override;
119+
// One reservation-address alias installed for this execution that must be
120+
// released with UnMap() when the execution finishes.
121+
struct StepAlias {
122+
uint64_t reservation_offset = 0;
123+
uint64_t mapping_size = 0;
124+
};
125+
126+
absl::Status Prepare(const ServiceExecutableRunOptions* run_options,
127+
int device_ordinal) override;
104128
absl::StatusOr<se::DeviceAddressBase> AllocateTransientBuffer(
105129
int device_ordinal, const BufferAllocation& allocation,
106130
int64_t buffer_size,
107131
se::DeviceAddressAllocator* memory_allocator) override;
108132

109133
bool ShouldRemapAllocation(BufferAllocation::Index index) const;
134+
// Returns the reservation slice [offset, offset + size) as an address.
135+
se::DeviceAddressBase ReservationSlice(uint64_t offset, uint64_t size) const;
136+
void RecordStepAlias(int device_ordinal, uint64_t reservation_offset,
137+
uint64_t mapping_size);
138+
// UnMaps all aliases recorded by RecordStepAlias. Aliases that fail to
139+
// unmap remain recorded so the execution-scope destructor can retry them.
140+
absl::Status ReleaseStepAliases();
110141
absl::StatusOr<se::ScopedDeviceAddress<uint8_t>> AllocateBuffer(
111142
int device_ordinal, const BufferAllocation& allocation,
112143
int64_t buffer_size);
144+
// Installs reservation aliases for selected parameter buffers and returns an
145+
// execution-only address table that refers to them. The owning address table
146+
// remains unchanged.
147+
absl::StatusOr<BufferAllocations> GetRemappedBufferAllocations(
148+
const BufferAllocations& owning_buffer_allocations, int device_ordinal);
113149

114150
const GpuExecutableVaRemapAllocator* owner_ = nullptr;
115151
Remapping* remapping_ = nullptr;
116152
se::DeviceAddressVmmAllocator* vmm_allocator_ = nullptr;
117153
std::unique_ptr<absl::MutexLock> remap_lock_;
154+
155+
// Per-execution alias bookkeeping filled by RecordStepAlias.
156+
int step_device_ordinal_ = -1;
157+
std::vector<StepAlias> step_aliases_;
118158
};
119159

120-
absl::Status
121-
GpuExecutableVaRemapAllocator::VaRemapExecutionScope::PrepareReservation(
160+
GpuExecutableVaRemapAllocator::VaRemapExecutionScope::~VaRemapExecutionScope() {
161+
if (!step_aliases_.empty()) {
162+
// Normally ReleaseStepAliases runs inside ExecuteWithBufferAllocations;
163+
// reaching this point means mapping, execution, or an earlier release
164+
// failed after some aliases were installed.
165+
absl::Status status = ReleaseStepAliases();
166+
if (!status.ok()) {
167+
LOG(ERROR) << "Failed to release command buffer VA remapping aliases "
168+
"for module "
169+
<< owner_->module_name() << ": " << status;
170+
}
171+
}
172+
}
173+
174+
absl::Status GpuExecutableVaRemapAllocator::VaRemapExecutionScope::Prepare(
122175
const ServiceExecutableRunOptions* run_options, int device_ordinal) {
123176
uint64_t granularity =
124177
vmm_allocator_->GetAllocationGranularity(run_options->stream()->parent());
@@ -138,13 +191,15 @@ GpuExecutableVaRemapAllocator::VaRemapExecutionScope::PrepareReservation(
138191
remapping_->granularity = granularity;
139192
remapping_->total_size = 0;
140193
remapping_->allocation_to_reservation_offset.clear();
194+
remapping_->allocation_to_mapping_size.clear();
141195
for (BufferAllocation::Index idx : owner_->va_remapped_alloc_indices_) {
142196
const BufferAllocation& allocation = *owner_->allocations()[idx];
143197
uint64_t buffer_size = allocation.size();
144-
remapping_->allocation_to_reservation_offset[idx] = remapping_->total_size;
145-
remapping_->total_size =
146-
remapping_->total_size +
198+
uint64_t mapping_size =
147199
RoundUpToGranularity(buffer_size, remapping_->granularity);
200+
remapping_->allocation_to_reservation_offset[idx] = remapping_->total_size;
201+
remapping_->allocation_to_mapping_size[idx] = mapping_size;
202+
remapping_->total_size = remapping_->total_size + mapping_size;
148203
}
149204
ASSIGN_OR_RETURN(
150205
remapping_->va_reservation,
@@ -163,6 +218,86 @@ bool GpuExecutableVaRemapAllocator::VaRemapExecutionScope::
163218
return owner_->va_remapped_alloc_indices_.contains(index);
164219
}
165220

221+
se::DeviceAddressBase
222+
GpuExecutableVaRemapAllocator::VaRemapExecutionScope::ReservationSlice(
223+
uint64_t offset, uint64_t size) const {
224+
return se::DeviceAddressBase(
225+
static_cast<char*>(remapping_->va_reservation->address().opaque()) +
226+
offset,
227+
size);
228+
}
229+
230+
void GpuExecutableVaRemapAllocator::VaRemapExecutionScope::RecordStepAlias(
231+
int device_ordinal, uint64_t reservation_offset, uint64_t mapping_size) {
232+
DCHECK(step_aliases_.empty() || step_device_ordinal_ == device_ordinal);
233+
step_device_ordinal_ = device_ordinal;
234+
step_aliases_.push_back({reservation_offset, mapping_size});
235+
}
236+
237+
absl::Status
238+
GpuExecutableVaRemapAllocator::VaRemapExecutionScope::ReleaseStepAliases() {
239+
absl::Status status;
240+
std::vector<StepAlias> failed_aliases;
241+
failed_aliases.reserve(step_aliases_.size());
242+
for (const StepAlias& alias : step_aliases_) {
243+
absl::Status unmap_status = vmm_allocator_->UnMap(
244+
step_device_ordinal_, remapping_->va_reservation.get(),
245+
alias.reservation_offset, alias.mapping_size);
246+
if (!unmap_status.ok()) {
247+
failed_aliases.push_back(alias);
248+
status.Update(unmap_status);
249+
}
250+
}
251+
step_aliases_ = std::move(failed_aliases);
252+
if (step_aliases_.empty()) {
253+
step_device_ordinal_ = -1;
254+
}
255+
return status;
256+
}
257+
258+
absl::StatusOr<BufferAllocations> GpuExecutableVaRemapAllocator::
259+
VaRemapExecutionScope::GetRemappedBufferAllocations(
260+
const BufferAllocations& owning_buffer_allocations,
261+
int device_ordinal) {
262+
// This is a non-owning copy of the finalized address table. Output donation
263+
// and copy protection run before this method and may replace parameter
264+
// entries, so parameter aliases must be installed from these current
265+
// addresses instead of during GenerateBufferAllocations.
266+
BufferAllocations execution_allocations(
267+
owning_buffer_allocations.buffers(),
268+
owning_buffer_allocations.device_ordinal(),
269+
owning_buffer_allocations.memory_allocator());
270+
271+
for (BufferAllocation::Index index : owner_->va_remapped_alloc_indices_) {
272+
const BufferAllocation& allocation = *owner_->allocations()[index];
273+
if (!allocation.is_entry_computation_parameter()) {
274+
continue;
275+
}
276+
const se::DeviceAddressBase buffer =
277+
owning_buffer_allocations.GetDeviceAddress(index);
278+
if (buffer.is_null()) {
279+
return Internal(
280+
"Command buffer VA remapping selected parameter allocation %d, but "
281+
"the parameter buffer is null for this execution",
282+
allocation.index());
283+
}
284+
ASSIGN_OR_RETURN(uint64_t va_offset,
285+
remapping_->GetReservationOffset(allocation.index()));
286+
ASSIGN_OR_RETURN(uint64_t mapping_size,
287+
remapping_->GetMappingSize(allocation.index()));
288+
// Map() reactivates a matching stale mapping from the previous execution,
289+
// so a parameter that keeps its address across executions performs no VMM
290+
// driver calls here.
291+
RETURN_IF_ERROR(vmm_allocator_->Map(device_ordinal, buffer,
292+
remapping_->va_reservation.get(),
293+
va_offset, mapping_size));
294+
RecordStepAlias(device_ordinal, va_offset, mapping_size);
295+
execution_allocations.GetMutableDeviceAddress(index) =
296+
ReservationSlice(va_offset, mapping_size);
297+
}
298+
return execution_allocations;
299+
}
300+
166301
absl::StatusOr<se::ScopedDeviceAddress<uint8_t>>
167302
GpuExecutableVaRemapAllocator::VaRemapExecutionScope::AllocateBuffer(
168303
int device_ordinal, const BufferAllocation& allocation,
@@ -202,8 +337,33 @@ absl::Status GpuExecutableVaRemapAllocator::VaRemapExecutionScope::
202337
"VA remapping: module %s executing with %d command buffer "
203338
"allocation(s)",
204339
owner_->module_name(), owner_->va_remapped_alloc_indices_.size());
205-
return execute(owning_buffer_allocations,
206-
absl::MakeConstSpan(owner_->persistent_alloc_indices()));
340+
341+
absl::StatusOr<BufferAllocations> execution_allocations =
342+
GetRemappedBufferAllocations(owning_buffer_allocations, device_ordinal);
343+
if (!execution_allocations.ok()) {
344+
absl::Status release_status = ReleaseStepAliases();
345+
if (!release_status.ok()) {
346+
LOG(ERROR) << "Failed to release command buffer VA remapping aliases "
347+
"after mapping failed for module "
348+
<< owner_->module_name() << ": " << release_status;
349+
}
350+
return execution_allocations.status();
351+
}
352+
353+
absl::Status execute_status =
354+
execute(*execution_allocations,
355+
absl::MakeConstSpan(owner_->persistent_alloc_indices()));
356+
// Release per-execution aliases even when execution failed. The owning
357+
// address table was never modified, so result handling and TearDown continue
358+
// to see caller- or allocator-owned addresses.
359+
absl::Status release_status = ReleaseStepAliases();
360+
if (!execute_status.ok() && !release_status.ok()) {
361+
LOG(ERROR) << "Failed to release command buffer VA remapping aliases "
362+
"after execution failed for module "
363+
<< owner_->module_name() << ": " << release_status;
364+
}
365+
RETURN_IF_ERROR(execute_status);
366+
return release_status;
207367
}
208368

209369
GpuExecutableVaRemapAllocator::GpuExecutableVaRemapAllocator(
@@ -248,6 +408,17 @@ GpuExecutableVaRemapAllocator::GpuExecutableVaRemapAllocator(
248408
<< this->module_name();
249409
}
250410

411+
void GpuExecutableVaRemapAllocator::AddVaRemappedAllocationForTesting(
412+
BufferAllocation::Index index) {
413+
va_remapped_alloc_indices_.insert(index);
414+
AllocationIndexSet persistent(constant_alloc_indices().begin(),
415+
constant_alloc_indices().end());
416+
persistent.insert(va_remapped_alloc_indices_.begin(),
417+
va_remapped_alloc_indices_.end());
418+
set_persistent_alloc_indices(std::vector<BufferAllocation::Index>(
419+
persistent.begin(), persistent.end()));
420+
}
421+
251422
GpuExecutableVaRemapAllocator::~GpuExecutableVaRemapAllocator() {
252423
absl::MutexLock lock(remappings_mutex_);
253424
for (auto& [executor, remapping] : remappings_) {

‎third_party/xla/xla/service/gpu/gpu_executable_va_remap_allocator.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,12 @@ class GpuExecutableVaRemapAllocator : public GpuExecutableBufferAllocator {
7171
se::DeviceAddressAllocator* memory_allocator,
7272
int device_ordinal) override;
7373

74+
// Test-only: adds `index` to the VA-remapped allocation set (and the
75+
// persistent set), as if it had been selected at construction time. Lets
76+
// unit tests exercise remapping of allocation kinds that no production
77+
// update mode selects yet.
78+
void AddVaRemappedAllocationForTesting(BufferAllocation::Index index);
79+
7480
private:
7581
// Per-executor VA remapping state, owned by the allocator and shared by the
7682
// execution scopes created for that executor.
@@ -80,11 +86,14 @@ class GpuExecutableVaRemapAllocator : public GpuExecutableBufferAllocator {
8086
uint64_t total_size = 0;
8187
absl::flat_hash_map<BufferAllocation::Index, uint64_t>
8288
allocation_to_reservation_offset;
89+
absl::flat_hash_map<BufferAllocation::Index, uint64_t>
90+
allocation_to_mapping_size;
8391
std::unique_ptr<se::MemoryReservation> va_reservation;
8492
se::DeviceAddressVmmAllocator* vmm_allocator = nullptr;
8593

8694
absl::StatusOr<uint64_t> GetReservationOffset(
8795
BufferAllocation::Index idx) const;
96+
absl::StatusOr<uint64_t> GetMappingSize(BufferAllocation::Index idx) const;
8897
};
8998

9099
// The VA-remapping ExecutionScope, defined in the .cc file.

0 commit comments

Comments
 (0)