Skip to content

Commit 36bd2ec

Browse files
shawnwang18tensorflower-gardener
authored andcommitted
PR #45596: [XLA:GPU] Support VA remapping of output buffer allocations
Imported from GitHub PR openxla/xla#45596 ## 📝 Summary of Changes This PR extends command-buffer virtual-address remapping to allocations that escape to the caller (`maybe_live_out`) while preserving the const ownership model introduced by #45595. - Keeps selected live-out allocations at normal allocator-owned addresses in the owning `BufferAllocations` table. - Extends `GetRemappedBufferAllocations` to map finalized selected input/output addresses into their stable reservation slices immediately before execution. - Rewrites only the non-owning execution copy, so result handling and teardown continue to see external, deallocatable addresses. - Handles donated and copy-protected aliases by mapping after output setup has finalized the owning address table. - Extends the host-only fake-VMM tests to cover stable execution aliases, logical external sizes, output deallocation, and deferred alias cleanup. This PR is stacked on #45595, **[XLA:GPU] Support VA remapping of parameter buffer allocations**. It is part 2 of 3 splitting the former #45568 implementation; #45597 adds the `SKIP_PROFILED` mode. ## 🎯 Justification Live-out buffers can outlive an execution scope, so the caller must retain a normal allocator-owned address that remains valid and can be passed to `Deallocate()`. At the same time, recorded command buffers need a stable reservation address. Building a separate execution allocation table provides both properties without mutating the owning table or removing its const qualifier. Deferring the mapping until execution also ensures donation and copy protection have already selected the final external address. ## 🚀 Kind of Contribution ✨ New Feature ## 📊 Benchmark (for Performance Improvements) Not applicable as a standalone change. This PR adds output-buffer remapping support but does not independently enable a new command-buffer update mode. Performance evaluation belongs to the final `SKIP_PROFILED` policy in #45597. ## 🧪 Unit Tests: - Extends the host-only fake-VMM allocator test with a non-granularity-aligned remapped live-out allocation. - Verifies the owning table contains the logical-size external address before and after execution. - Verifies execution receives the same stable reservation address across runs and maps the correct raw physical allocation. - Verifies the external address is deallocatable and all aliases are released. ## 🧪 Execution Tests: No standalone execution test is added for this enabling layer. End-to-end live-out selection, execution, and output deallocation are covered by the stacked `SKIP_PROFILED` change in #45597. Copybara import of the project: -- b7b79af2174271e65f2c7d8e8bac527a72e97af9 by Shawn Wang <[email protected]>: [XLA:GPU] Support VA remapping of output buffer allocations Keep selected maybe_live_out allocations at normal allocator-owned addresses in the owning BufferAllocations table. Immediately before execution, GetRemappedBufferAllocations maps the finalized selected input/output addresses into their stable reservation slices and rewrites only a non-owning execution copy. This preserves const ownership semantics, handles donation and copy protection before mapping, and leaves output addresses deallocatable by callers. Extend the host-only VMM test to verify stable execution addresses, logical external sizes, unchanged owning buffers, and alias cleanup across executions. Merging this change closes #45596 Reverts changelist 949495726 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45596 from shawnwang18:shawnw/cmd_buffer_va_remap_outputs b7b79af2174271e65f2c7d8e8bac527a72e97af9 PiperOrigin-RevId: 950641350
1 parent 4f1b7de commit 36bd2ec

192 files changed

Lines changed: 5452 additions & 1991 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎MODULE.bazel‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ tsl_extension = use_extension("@xla//third_party/extensions:tsl.bzl", "tsl_exten
101101
use_repo(tsl_extension, tsl = "tsl")
102102

103103
llvm = use_extension("@xla//third_party/extensions:llvm.bzl", "llvm_extension")
104-
use_repo(llvm, "llvm-project", llvm = "llvm-project")
104+
use_repo(llvm, "llvm-project")
105105

106106
xla_third_party = use_extension("@xla//third_party/extensions:third_party.bzl", "third_party_ext")
107107
use_repo(

‎tensorflow/c/eager/parallel_device/parallel_device_lib.cc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -304,7 +304,7 @@ void DeviceThread::Execute(TFE_Context* context, const char* operation_name,
304304
ParallelDevice::ParallelDevice(const std::vector<std::string>& devices,
305305
bool is_async, int in_flight_nodes_limit)
306306
: underlying_devices_(devices),
307-
default_cancellation_manager_(absl::make_unique<CancellationManager>()) {
307+
default_cancellation_manager_(std::make_unique<CancellationManager>()) {
308308
device_threads_.reserve(devices.size());
309309
for (int device_index = 0; device_index < devices.size(); ++device_index) {
310310
device_threads_.emplace_back(new DeviceThread(
@@ -358,7 +358,7 @@ ParallelDevice::Execute(TFE_Context* context,
358358
TFE_ContextAsyncWait(context, await_status.get());
359359
// Reset the cancellation manager on a bad status. Otherwise we'll cancel
360360
// all future operations.
361-
default_cancellation_manager_ = absl::make_unique<CancellationManager>();
361+
default_cancellation_manager_ = std::make_unique<CancellationManager>();
362362
}
363363
return result;
364364
}

‎tensorflow/c/experimental/saved_model/core/saved_variable_loading_test.cc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ class SavedVariableLoadingTest
5252
std::vector<std::unique_ptr<Device>> devices;
5353
TF_CHECK_OK(DeviceFactory::AddDevices(
5454
options, "/job:localhost/replica:0/task:0", &devices));
55-
device_mgr_ = absl::make_unique<StaticDeviceMgr>(std::move(devices));
55+
device_mgr_ = std::make_unique<StaticDeviceMgr>(std::move(devices));
5656
ctx_ = testing::CreateTestingEagerContext(device_mgr_.get());
5757
}
5858

‎tensorflow/c/experimental/stream_executor/stream_executor_test.cc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ class StreamExecutorTest : public ::testing::Test {
162162

163163
StreamExecutor* GetExecutor(int ordinal) {
164164
if (!cplatform_) {
165-
cplatform_ = absl::make_unique<CPlatform>(
165+
cplatform_ = std::make_unique<CPlatform>(
166166
platform_, test_util::DestroyPlatform, platform_fns_,
167167
test_util::DestroyPlatformFns, device_fns_, se_, timer_fns_);
168168
}

‎tensorflow/cc/client/client_session_test.cc‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -155,11 +155,11 @@ TEST(ClientSessionTest, MultiThreadedWithCustomThreadpool) {
155155
ClientSession session(root, GetSessionOptions());
156156

157157
auto inter_op_threadpool =
158-
absl::make_unique<CustomThreadPoolImpl>(num_threads);
158+
std::make_unique<CustomThreadPoolImpl>(num_threads);
159159
ASSERT_EQ(inter_op_threadpool->GetNumScheduleCalled(), 0);
160160

161161
auto intra_op_threadpool =
162-
absl::make_unique<CustomThreadPoolImpl>(num_threads);
162+
std::make_unique<CustomThreadPoolImpl>(num_threads);
163163
ASSERT_EQ(intra_op_threadpool->GetNumScheduleCalled(), 0);
164164

165165
tensorflow::thread::ThreadPoolOptions threadPoolOptions;
@@ -223,11 +223,11 @@ TEST(ClientSessionTest, CallableWithCustomThreadPool) {
223223
std::vector<Tensor> outputs;
224224

225225
auto inter_op_threadpool =
226-
absl::make_unique<CustomThreadPoolImpl>(num_threads);
226+
std::make_unique<CustomThreadPoolImpl>(num_threads);
227227
ASSERT_EQ(inter_op_threadpool->GetNumScheduleCalled(), 0);
228228

229229
auto intra_op_threadpool =
230-
absl::make_unique<CustomThreadPoolImpl>(num_threads);
230+
std::make_unique<CustomThreadPoolImpl>(num_threads);
231231
ASSERT_EQ(intra_op_threadpool->GetNumScheduleCalled(), 0);
232232

233233
tensorflow::thread::ThreadPoolOptions threadPoolOptions;

‎tensorflow/cc/gradients/linalg_grad.cc‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ limitations under the License.
2727
#include "absl/strings/str_cat.h"
2828
#include "absl/strings/str_split.h"
2929
#include "absl/strings/string_view.h"
30-
#include "absl/types/optional.h"
3130
#include "tensorflow/cc/framework/grad_op_registry.h"
3231
#include "tensorflow/cc/framework/gradients.h"
3332
#include "tensorflow/cc/gradients/grad_helper.h"

‎tensorflow/cc/saved_model/bundle_v2.cc‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ absl::Status SavedModelV2Bundle::Load(const std::string& export_dir,
8686
if (saved_model_proto.meta_graphs_size() != 1) {
8787
return absl::Status(
8888
absl::StatusCode::kInvalidArgument,
89-
strings::StrCat(
89+
absl::StrCat(
9090
"SavedModelV2 should have exactly one MetaGraphDef but actually ",
9191
"contains ", saved_model_proto.meta_graphs_size()));
9292
}
@@ -185,7 +185,7 @@ absl::Status SavedModelV2Bundle::RecurseObjectsToRestore(
185185
if (object_name.empty()) {
186186
child_name = local_name;
187187
} else {
188-
child_name = strings::StrCat(object_name, ".", local_name);
188+
child_name = absl::StrCat(object_name, ".", local_name);
189189
}
190190

191191
// Descend down the trackable graph.
@@ -197,7 +197,7 @@ absl::Status SavedModelV2Bundle::RecurseObjectsToRestore(
197197
if (trackable_child_node_id < 0 ||
198198
trackable_child_node_id >= trackable_object_graph().nodes_size()) {
199199
return absl::FailedPreconditionError(
200-
strings::StrCat("Illegal trackable child node id for ", child_name));
200+
absl::StrCat("Illegal trackable child node id for ", child_name));
201201
}
202202
const auto* trackable_child =
203203
&trackable_object_graph().nodes(trackable_child_node_id);
@@ -220,8 +220,8 @@ absl::Status SavedModelV2Bundle::RecurseObjectsToRestore(
220220
if (!saved_child) {
221221
return absl::Status(
222222
absl::StatusCode::kFailedPrecondition,
223-
strings::StrCat("Could not find saved object to restore for ",
224-
child_name));
223+
absl::StrCat("Could not find saved object to restore for ",
224+
child_name));
225225
}
226226

227227
TF_RETURN_IF_ERROR(RecurseObjectsToRestore(

‎tensorflow/compiler/jit/device_executable_persistor.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -228,7 +228,7 @@ DeviceExecutablePersistor<ExecutableType, ClientType>::TryToReadSerializedEntry(
228228
}
229229

230230
XlaSerializedCacheEntry entry;
231-
TF_RETURN_IF_ERROR(ReadTextOrBinaryProto(env, file_path, &entry));
231+
TF_RETURN_IF_ERROR(ReadBinaryProto(env, file_path, &entry));
232232
return std::optional<XlaSerializedCacheEntry>(entry);
233233
}
234234

‎tensorflow/compiler/mlir/lite/flatbuffer_import.cc‎

Lines changed: 66 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,56 @@ Status ParseDebugMetadata(Builder builder, const char* data, size_t size,
403403
return absl::OkStatus();
404404
}
405405

406+
std::string GetTfliteOpNameForLoc(const tflite::OperatorCodeT& op_code) {
407+
auto builtin_code = tflite::GetBuiltinCode(&op_code);
408+
if (builtin_code == tflite::BuiltinOperator_CUSTOM) {
409+
if (!op_code.custom_code.empty()) return op_code.custom_code;
410+
return "CUSTOM";
411+
}
412+
return tflite::EnumNameBuiltinOperator(builtin_code);
413+
}
414+
415+
Location AttachTfliteOpLoc(Builder builder, int subgraph_index, int op_index,
416+
const tflite::OperatorCodeT& op_code,
417+
Location child_loc) {
418+
std::string op_name = GetTfliteOpNameForLoc(op_code);
419+
std::string loc_name = absl::StrCat("tflite.subgraph=", subgraph_index,
420+
".op=", op_index, ":", op_name);
421+
// Attach as FusedLoc metadata so name extractors (e.g. GetNameFromLoc used
422+
// by the exporter to derive tensor names) keep resolving names from the
423+
// original child location.
424+
return mlir::FusedLoc::get(child_loc.getContext(), {child_loc},
425+
builder.getStringAttr(loc_name));
426+
}
427+
428+
Location AttachTfliteTensorLoc(Builder builder, int subgraph_index,
429+
int tensor_index, Location child_loc) {
430+
std::string loc_name = absl::StrCat("tflite.subgraph=", subgraph_index,
431+
".tensor=", tensor_index, ":CONST");
432+
return mlir::FusedLoc::get(child_loc.getContext(), {child_loc},
433+
builder.getStringAttr(loc_name));
434+
}
435+
436+
Location MaybeAttachFusedActivationLoc(Location loc, Builder builder,
437+
int subgraph_index, int op_index,
438+
const tflite::OperatorCodeT& op_code,
439+
Location child_loc, Operation* mlir_op) {
440+
auto fused_attr =
441+
mlir_op->getAttrOfType<mlir::StringAttr>("fused_activation_function");
442+
if (!fused_attr) return loc;
443+
444+
llvm::StringRef fused = fused_attr.getValue();
445+
if (fused.empty() || fused == "NONE") return loc;
446+
447+
std::string op_name = GetTfliteOpNameForLoc(op_code);
448+
std::string fused_name = fused.str();
449+
std::string loc_name =
450+
absl::StrCat("tflite.subgraph=", subgraph_index, ".op=", op_index, ":",
451+
op_name, "_fused_", fused_name);
452+
return mlir::FusedLoc::get(child_loc.getContext(), {child_loc},
453+
builder.getStringAttr(loc_name));
454+
}
455+
406456
// Return MLIR location if it exists in the debug metadata. Otherwise, create a
407457
// MLIR location by fusing its output tensor names.
408458
Location OpLoc(const OperatorT& op, Builder builder,
@@ -1470,7 +1520,7 @@ mlir::ResultRange MaybeWrapInControlNode(mlir::Operation* op,
14701520
// inputs/outputs in signature will be attached to the FuncOp.
14711521
StatusOr<FuncOp> ConvertSubgraph(
14721522
const tflite::ModelT& model, const tflite::SubGraphT& subgraph,
1473-
llvm::StringRef name,
1523+
int subgraph_index, llvm::StringRef name,
14741524
const std::vector<std::unique_ptr<tflite::OperatorCodeT>>& op_codes,
14751525
const std::vector<std::string>& func_names,
14761526
const std::vector<std::unique_ptr<tflite::BufferT>>& buffers,
@@ -1672,6 +1722,8 @@ StatusOr<FuncOp> ConvertSubgraph(
16721722
return emitError(const_loc, op_or_err.status().ToString()),
16731723
op_or_err.status();
16741724
}
1725+
op_or_err.value()->setLoc(AttachTfliteTensorLoc(builder, subgraph_index,
1726+
input_num, const_loc));
16751727
vals_map[input_num] = op_or_err.value()->getResult(0);
16761728
}
16771729
}
@@ -1689,7 +1741,11 @@ StatusOr<FuncOp> ConvertSubgraph(
16891741
intermediate_types.emplace_back(type);
16901742
}
16911743

1692-
auto op_loc = OpLoc(*op, builder, debug_metadata, subgraph, base_loc);
1744+
const tflite::OperatorCodeT& op_code = *op_codes.at(op->opcode_index);
1745+
auto original_op_loc =
1746+
OpLoc(*op, builder, debug_metadata, subgraph, base_loc);
1747+
auto op_loc = AttachTfliteOpLoc(builder, subgraph_index, it.index(),
1748+
op_code, original_op_loc);
16931749

16941750
// If there's an optional argument, maybe_optional_arg_marker has been set
16951751
// to a valid Value
@@ -1698,6 +1754,10 @@ StatusOr<FuncOp> ConvertSubgraph(
16981754
ConvertOp(*op, vals_map, intermediate_types, maybe_optional_arg_marker,
16991755
op_codes, func_names, subgraph.tensors, op_loc, op_builder,
17001756
model_ptr));
1757+
op_loc = MaybeAttachFusedActivationLoc(op_loc, builder, subgraph_index,
1758+
it.index(), op_code, original_op_loc,
1759+
mlir_op);
1760+
mlir_op->setLoc(op_loc);
17011761

17021762
// Add the results to the value maps. There are two cases: 1. the result
17031763
// tensor does not have min/max values, the original op result is used
@@ -1762,6 +1822,8 @@ StatusOr<FuncOp> ConvertSubgraph(
17621822
return emitError(const_loc, op_or_err.status().ToString()),
17631823
op_or_err.status();
17641824
}
1825+
op_or_err.value()->setLoc(
1826+
AttachTfliteTensorLoc(builder, subgraph_index, index, const_loc));
17651827
vals_map[index] = op_or_err.value()->getResult(0);
17661828
}
17671829
return_operands.push_back(vals_map[index]);
@@ -2039,8 +2101,8 @@ OwningOpRef<mlir::ModuleOp> tflite::FlatBufferToMlir(
20392101
SubgraphName(set_implicit_main_func, e.index(), *subgraph);
20402102
uint32_t subgraph_index = static_cast<uint32_t>(e.index());
20412103
auto func_or_error = ConvertSubgraph(
2042-
*model, *subgraph, name, model->operator_codes, func_names,
2043-
model->buffers, base_loc, builder,
2104+
*model, *subgraph, static_cast<int>(subgraph_index), name,
2105+
model->operator_codes, func_names, model->buffers, base_loc, builder,
20442106
/*is_entry_point=*/
20452107
set_implicit_main_func
20462108
? e.index() == 0

‎tensorflow/compiler/mlir/lite/quantization/common/quantization_lib/quantization_utils.h‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -490,6 +490,27 @@ class QuantizationPattern : public RewritePattern {
490490
(nodes_blocklist.find(sloc) != nodes_blocklist.end())) {
491491
return failure();
492492
}
493+
} else if (auto fused_loc =
494+
llvm::dyn_cast<FusedLoc>(quantizing_op->getLoc())) {
495+
llvm::SmallVector<mlir::Location, 4> stack;
496+
stack.push_back(fused_loc);
497+
bool blocked = false;
498+
while (!stack.empty()) {
499+
mlir::Location l = stack.pop_back_val();
500+
if (auto name_loc = llvm::dyn_cast<NameLoc>(l)) {
501+
std::string sloc = name_loc.getName().str();
502+
if (!sloc.empty() &&
503+
(nodes_blocklist.find(sloc) != nodes_blocklist.end())) {
504+
blocked = true;
505+
break;
506+
}
507+
} else if (auto fused = llvm::dyn_cast<FusedLoc>(l)) {
508+
for (auto child_loc : fused.getLocations()) {
509+
stack.push_back(child_loc);
510+
}
511+
}
512+
}
513+
if (blocked) return failure();
493514
}
494515
}
495516

0 commit comments

Comments
 (0)