Skip to content

Commit 617a41b

Browse files
shawnwang18tensorflower-gardener
authored andcommitted
PR #45597: [XLA:GPU] Add SKIP_PROFILED command buffer update mode
Imported from GitHub PR openxla/xla#45597 ## 📝 Summary of Changes This PR adds the `SKIP_PROFILED` command-buffer update mode. The mode automatically includes command-buffer-referenced preallocated temp allocations in the final VA-remap set while profiling other eligible non-constant allocations during the first three successful executions. Stable profile candidates are combined with the temp allocations so subsequent command-buffer executions can treat the final set as persistent and avoid unnecessary address updates. - Adds `DebugOptions::SKIP_PROFILED` and routes it through the GPU executable buffer-allocation policy. - Adds a per-executor profile state machine to `GpuExecutableVaRemapAllocator`: `kProfiling` transitions exactly once to `kActive` when an automatic temp or safe profile candidate exists, or to `kDisabled` when the final remap set is empty. - Automatically selects preallocated temp buffers and profiles eligible parameter and live-out candidates while filtering unstable addresses, non-VMM-backed parameters, and incompatible shared parameter buffers. - Counts only successful executions toward the profiling window and passes `std::nullopt` for persistent allocation indices while profiling, followed by one absent-to-present transition to the final persistent set. - Uses the parameter-remapping support from #45595 and output-buffer remapping support from #45596. Parameters and outputs remain externally owned in the const owning allocation table; finalized external addresses are mapped into an execution-only allocation-table copy immediately before execution. - Preserves the base allocator fallback when VMM remapping is unavailable or there are neither automatic temp buffers nor stable profile candidates. This PR is stacked on #45596, **[XLA:GPU] Support VA remapping of output buffer allocations**. It is part 3 of 3 and, together with #45595 and #45596, supersedes the closed pre-split PR #45568. ## 🎯 Justification `SKIP_TEMP` can stabilize preallocated temporary buffers, but parameters and live-out buffers may also keep stable physical addresses across repeated executions. Updating command-buffer node addresses for those stable allocations adds avoidable runtime overhead. `SKIP_PROFILED` discovers address stability for each executable/executor pair instead of assuming every dynamic allocation is stable. Workloads with repeatable allocation behavior can therefore stabilize more command-buffer-visible addresses, while unstable or unsupported workloads retain the existing fallback behavior. Mapping finalized owning addresses immediately before execution also keeps donation and copy-protection changes compatible with the const owning-allocation API introduced by the lower PRs. ## 🚀 Kind of Contribution ✨ New Feature ## 📊 Benchmark (for Performance Improvements) No benchmark result is claimed in this draft. The PR adds the policy and correctness coverage; a public-HLO performance benchmark can be added separately when reporting workload-specific speedups. ## 🧪 Unit Tests: - Extended `GpuExecutableTest.CommandBufferAllocationPolicy` expectations for automatic temp-buffer selection in `SKIP_PROFILED`. - Added host-only fake-VMM coverage for profiling, transition, active remapping, and disabled fallback states. - Verifies a temp-only executable transitions to an active persistent temp set, and an unstable parameter cannot disqualify an automatically selected temp. - Verifies stable parameter and output selection, including non-granularity-sized externally owned outputs. - Verifies failed executions do not advance the three-successful-execution profiling window. - Verifies fallback for unstable and non-VMM-backed parameters and incompatible shared parameter buffers. - Verifies copy protection both during profiling and after transition: the owning table retains the finalized external address while execution uses the stable reservation address. ## 🧪 Execution Tests: - Added one-device `VmmTest` coverage using a two-GEMM command-buffer workload across profiling, transition, and remapped executions, checking numerical correctness on every run. - Verifies one-time profile selection, automatic temp inclusion, VA-reservation creation and reuse, and successful deallocation of externally returned output buffers. - Added a non-VMM allocator execution test verifying the workload remains correct and creates no VA reservation when remapping is unavailable. Copybara import of the project: -- 8b5c4d766fdb20611c575203a9851ff7c67c27e8 by Shawn Wang <[email protected]>: [XLA:GPU] Add SKIP_PROFILED command buffer update mode Add SKIP_PROFILED to profile command-buffer-referenced non-constant allocation addresses for three successful executions, then VA-remap the stable candidates so recorded command buffers can skip updating them. The per-executor profile transitions once from profiling to active or disabled, preserving the required absent-to-present persistent-index transition. Active executions allocate selected temps at reservation addresses and map finalized selected input/output addresses into an execution-only BufferAllocations copy, leaving the owning table unchanged. Filter unstable candidates and parameter addresses that are not VMM-backed or are shared by multiple parameter allocations. Add flag parsing, allocation-policy coverage, GPU end-to-end coverage, and fallback behavior without a VMM allocator. -- a099d0ccb63bb1c45c39fb54a40e6431ba806253 by Shawn Wang <[email protected]>: Add host-only unit tests for SKIP_PROFILED Cover the SKIP_PROFILED profile state machine with the fake-VMM harness through a test-only profile-candidate seam. Verify stable parameters and outputs transition to execution-only reservation addresses while owning buffers remain external, failed executions do not advance profiling, and unstable, non-VMM-backed, or shared parameter buffers disable remapping. Verify copy-protected addresses are observed after output finalization and can be remapped both during a stable profile and when copy protection first appears after the transition. -- 153df493569ae521e8e0058d55241298ead297aa by Shawn Wang <[email protected]>: Remove profile candidate count accessor -- 16473a0c5bacc88055d839393b332223781dbb3e by Shawn Wang <[email protected]>: [XLA:GPU] Automatically remap temp buffers in SKIP_PROFILED -- 760026228f016ed3b0a6214b7286c1cda8bd5ab9 by Shawn Wang <[email protected]>: [XLA:GPU] Add direct VA remap allocator includes Merging this change closes #45597 Reverts changelist 949495726 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45597 from shawnwang18:shawnw/cmd_buffer_skip_profiled_split 760026228f016ed3b0a6214b7286c1cda8bd5ab9 PiperOrigin-RevId: 950773148
1 parent 4f1b7de commit 617a41b

207 files changed

Lines changed: 7124 additions & 2379 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎MODULE.bazel‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ tsl_extension = use_extension("@xla//third_party/extensions:tsl.bzl", "tsl_exten
101101
use_repo(tsl_extension, tsl = "tsl")
102102

103103
llvm = use_extension("@xla//third_party/extensions:llvm.bzl", "llvm_extension")
104-
use_repo(llvm, "llvm-project", llvm = "llvm-project")
104+
use_repo(llvm, "llvm-project")
105105

106106
xla_third_party = use_extension("@xla//third_party/extensions:third_party.bzl", "third_party_ext")
107107
use_repo(

‎tensorflow/c/eager/parallel_device/parallel_device_lib.cc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -304,7 +304,7 @@ void DeviceThread::Execute(TFE_Context* context, const char* operation_name,
304304
ParallelDevice::ParallelDevice(const std::vector<std::string>& devices,
305305
bool is_async, int in_flight_nodes_limit)
306306
: underlying_devices_(devices),
307-
default_cancellation_manager_(absl::make_unique<CancellationManager>()) {
307+
default_cancellation_manager_(std::make_unique<CancellationManager>()) {
308308
device_threads_.reserve(devices.size());
309309
for (int device_index = 0; device_index < devices.size(); ++device_index) {
310310
device_threads_.emplace_back(new DeviceThread(
@@ -358,7 +358,7 @@ ParallelDevice::Execute(TFE_Context* context,
358358
TFE_ContextAsyncWait(context, await_status.get());
359359
// Reset the cancellation manager on a bad status. Otherwise we'll cancel
360360
// all future operations.
361-
default_cancellation_manager_ = absl::make_unique<CancellationManager>();
361+
default_cancellation_manager_ = std::make_unique<CancellationManager>();
362362
}
363363
return result;
364364
}

‎tensorflow/c/experimental/saved_model/core/saved_variable_loading_test.cc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ class SavedVariableLoadingTest
5252
std::vector<std::unique_ptr<Device>> devices;
5353
TF_CHECK_OK(DeviceFactory::AddDevices(
5454
options, "/job:localhost/replica:0/task:0", &devices));
55-
device_mgr_ = absl::make_unique<StaticDeviceMgr>(std::move(devices));
55+
device_mgr_ = std::make_unique<StaticDeviceMgr>(std::move(devices));
5656
ctx_ = testing::CreateTestingEagerContext(device_mgr_.get());
5757
}
5858

‎tensorflow/c/experimental/stream_executor/stream_executor_test.cc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ class StreamExecutorTest : public ::testing::Test {
162162

163163
StreamExecutor* GetExecutor(int ordinal) {
164164
if (!cplatform_) {
165-
cplatform_ = absl::make_unique<CPlatform>(
165+
cplatform_ = std::make_unique<CPlatform>(
166166
platform_, test_util::DestroyPlatform, platform_fns_,
167167
test_util::DestroyPlatformFns, device_fns_, se_, timer_fns_);
168168
}

‎tensorflow/cc/client/client_session_test.cc‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -155,11 +155,11 @@ TEST(ClientSessionTest, MultiThreadedWithCustomThreadpool) {
155155
ClientSession session(root, GetSessionOptions());
156156

157157
auto inter_op_threadpool =
158-
absl::make_unique<CustomThreadPoolImpl>(num_threads);
158+
std::make_unique<CustomThreadPoolImpl>(num_threads);
159159
ASSERT_EQ(inter_op_threadpool->GetNumScheduleCalled(), 0);
160160

161161
auto intra_op_threadpool =
162-
absl::make_unique<CustomThreadPoolImpl>(num_threads);
162+
std::make_unique<CustomThreadPoolImpl>(num_threads);
163163
ASSERT_EQ(intra_op_threadpool->GetNumScheduleCalled(), 0);
164164

165165
tensorflow::thread::ThreadPoolOptions threadPoolOptions;
@@ -223,11 +223,11 @@ TEST(ClientSessionTest, CallableWithCustomThreadPool) {
223223
std::vector<Tensor> outputs;
224224

225225
auto inter_op_threadpool =
226-
absl::make_unique<CustomThreadPoolImpl>(num_threads);
226+
std::make_unique<CustomThreadPoolImpl>(num_threads);
227227
ASSERT_EQ(inter_op_threadpool->GetNumScheduleCalled(), 0);
228228

229229
auto intra_op_threadpool =
230-
absl::make_unique<CustomThreadPoolImpl>(num_threads);
230+
std::make_unique<CustomThreadPoolImpl>(num_threads);
231231
ASSERT_EQ(intra_op_threadpool->GetNumScheduleCalled(), 0);
232232

233233
tensorflow::thread::ThreadPoolOptions threadPoolOptions;

‎tensorflow/cc/gradients/linalg_grad.cc‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ limitations under the License.
2727
#include "absl/strings/str_cat.h"
2828
#include "absl/strings/str_split.h"
2929
#include "absl/strings/string_view.h"
30-
#include "absl/types/optional.h"
3130
#include "tensorflow/cc/framework/grad_op_registry.h"
3231
#include "tensorflow/cc/framework/gradients.h"
3332
#include "tensorflow/cc/gradients/grad_helper.h"

‎tensorflow/cc/saved_model/bundle_v2.cc‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ absl::Status SavedModelV2Bundle::Load(const std::string& export_dir,
8686
if (saved_model_proto.meta_graphs_size() != 1) {
8787
return absl::Status(
8888
absl::StatusCode::kInvalidArgument,
89-
strings::StrCat(
89+
absl::StrCat(
9090
"SavedModelV2 should have exactly one MetaGraphDef but actually ",
9191
"contains ", saved_model_proto.meta_graphs_size()));
9292
}
@@ -185,7 +185,7 @@ absl::Status SavedModelV2Bundle::RecurseObjectsToRestore(
185185
if (object_name.empty()) {
186186
child_name = local_name;
187187
} else {
188-
child_name = strings::StrCat(object_name, ".", local_name);
188+
child_name = absl::StrCat(object_name, ".", local_name);
189189
}
190190

191191
// Descend down the trackable graph.
@@ -197,7 +197,7 @@ absl::Status SavedModelV2Bundle::RecurseObjectsToRestore(
197197
if (trackable_child_node_id < 0 ||
198198
trackable_child_node_id >= trackable_object_graph().nodes_size()) {
199199
return absl::FailedPreconditionError(
200-
strings::StrCat("Illegal trackable child node id for ", child_name));
200+
absl::StrCat("Illegal trackable child node id for ", child_name));
201201
}
202202
const auto* trackable_child =
203203
&trackable_object_graph().nodes(trackable_child_node_id);
@@ -220,8 +220,8 @@ absl::Status SavedModelV2Bundle::RecurseObjectsToRestore(
220220
if (!saved_child) {
221221
return absl::Status(
222222
absl::StatusCode::kFailedPrecondition,
223-
strings::StrCat("Could not find saved object to restore for ",
224-
child_name));
223+
absl::StrCat("Could not find saved object to restore for ",
224+
child_name));
225225
}
226226

227227
TF_RETURN_IF_ERROR(RecurseObjectsToRestore(

‎tensorflow/compiler/jit/device_executable_persistor.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -228,7 +228,7 @@ DeviceExecutablePersistor<ExecutableType, ClientType>::TryToReadSerializedEntry(
228228
}
229229

230230
XlaSerializedCacheEntry entry;
231-
TF_RETURN_IF_ERROR(ReadTextOrBinaryProto(env, file_path, &entry));
231+
TF_RETURN_IF_ERROR(ReadBinaryProto(env, file_path, &entry));
232232
return std::optional<XlaSerializedCacheEntry>(entry);
233233
}
234234

‎tensorflow/compiler/mlir/lite/flatbuffer_import.cc‎

Lines changed: 66 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,56 @@ Status ParseDebugMetadata(Builder builder, const char* data, size_t size,
403403
return absl::OkStatus();
404404
}
405405

406+
std::string GetTfliteOpNameForLoc(const tflite::OperatorCodeT& op_code) {
407+
auto builtin_code = tflite::GetBuiltinCode(&op_code);
408+
if (builtin_code == tflite::BuiltinOperator_CUSTOM) {
409+
if (!op_code.custom_code.empty()) return op_code.custom_code;
410+
return "CUSTOM";
411+
}
412+
return tflite::EnumNameBuiltinOperator(builtin_code);
413+
}
414+
415+
Location AttachTfliteOpLoc(Builder builder, int subgraph_index, int op_index,
416+
const tflite::OperatorCodeT& op_code,
417+
Location child_loc) {
418+
std::string op_name = GetTfliteOpNameForLoc(op_code);
419+
std::string loc_name = absl::StrCat("tflite.subgraph=", subgraph_index,
420+
".op=", op_index, ":", op_name);
421+
// Attach as FusedLoc metadata so name extractors (e.g. GetNameFromLoc used
422+
// by the exporter to derive tensor names) keep resolving names from the
423+
// original child location.
424+
return mlir::FusedLoc::get(child_loc.getContext(), {child_loc},
425+
builder.getStringAttr(loc_name));
426+
}
427+
428+
Location AttachTfliteTensorLoc(Builder builder, int subgraph_index,
429+
int tensor_index, Location child_loc) {
430+
std::string loc_name = absl::StrCat("tflite.subgraph=", subgraph_index,
431+
".tensor=", tensor_index, ":CONST");
432+
return mlir::FusedLoc::get(child_loc.getContext(), {child_loc},
433+
builder.getStringAttr(loc_name));
434+
}
435+
436+
Location MaybeAttachFusedActivationLoc(Location loc, Builder builder,
437+
int subgraph_index, int op_index,
438+
const tflite::OperatorCodeT& op_code,
439+
Location child_loc, Operation* mlir_op) {
440+
auto fused_attr =
441+
mlir_op->getAttrOfType<mlir::StringAttr>("fused_activation_function");
442+
if (!fused_attr) return loc;
443+
444+
llvm::StringRef fused = fused_attr.getValue();
445+
if (fused.empty() || fused == "NONE") return loc;
446+
447+
std::string op_name = GetTfliteOpNameForLoc(op_code);
448+
std::string fused_name = fused.str();
449+
std::string loc_name =
450+
absl::StrCat("tflite.subgraph=", subgraph_index, ".op=", op_index, ":",
451+
op_name, "_fused_", fused_name);
452+
return mlir::FusedLoc::get(child_loc.getContext(), {child_loc},
453+
builder.getStringAttr(loc_name));
454+
}
455+
406456
// Return MLIR location if it exists in the debug metadata. Otherwise, create a
407457
// MLIR location by fusing its output tensor names.
408458
Location OpLoc(const OperatorT& op, Builder builder,
@@ -1470,7 +1520,7 @@ mlir::ResultRange MaybeWrapInControlNode(mlir::Operation* op,
14701520
// inputs/outputs in signature will be attached to the FuncOp.
14711521
StatusOr<FuncOp> ConvertSubgraph(
14721522
const tflite::ModelT& model, const tflite::SubGraphT& subgraph,
1473-
llvm::StringRef name,
1523+
int subgraph_index, llvm::StringRef name,
14741524
const std::vector<std::unique_ptr<tflite::OperatorCodeT>>& op_codes,
14751525
const std::vector<std::string>& func_names,
14761526
const std::vector<std::unique_ptr<tflite::BufferT>>& buffers,
@@ -1672,6 +1722,8 @@ StatusOr<FuncOp> ConvertSubgraph(
16721722
return emitError(const_loc, op_or_err.status().ToString()),
16731723
op_or_err.status();
16741724
}
1725+
op_or_err.value()->setLoc(AttachTfliteTensorLoc(builder, subgraph_index,
1726+
input_num, const_loc));
16751727
vals_map[input_num] = op_or_err.value()->getResult(0);
16761728
}
16771729
}
@@ -1689,7 +1741,11 @@ StatusOr<FuncOp> ConvertSubgraph(
16891741
intermediate_types.emplace_back(type);
16901742
}
16911743

1692-
auto op_loc = OpLoc(*op, builder, debug_metadata, subgraph, base_loc);
1744+
const tflite::OperatorCodeT& op_code = *op_codes.at(op->opcode_index);
1745+
auto original_op_loc =
1746+
OpLoc(*op, builder, debug_metadata, subgraph, base_loc);
1747+
auto op_loc = AttachTfliteOpLoc(builder, subgraph_index, it.index(),
1748+
op_code, original_op_loc);
16931749

16941750
// If there's an optional argument, maybe_optional_arg_marker has been set
16951751
// to a valid Value
@@ -1698,6 +1754,10 @@ StatusOr<FuncOp> ConvertSubgraph(
16981754
ConvertOp(*op, vals_map, intermediate_types, maybe_optional_arg_marker,
16991755
op_codes, func_names, subgraph.tensors, op_loc, op_builder,
17001756
model_ptr));
1757+
op_loc = MaybeAttachFusedActivationLoc(op_loc, builder, subgraph_index,
1758+
it.index(), op_code, original_op_loc,
1759+
mlir_op);
1760+
mlir_op->setLoc(op_loc);
17011761

17021762
// Add the results to the value maps. There are two cases: 1. the result
17031763
// tensor does not have min/max values, the original op result is used
@@ -1762,6 +1822,8 @@ StatusOr<FuncOp> ConvertSubgraph(
17621822
return emitError(const_loc, op_or_err.status().ToString()),
17631823
op_or_err.status();
17641824
}
1825+
op_or_err.value()->setLoc(
1826+
AttachTfliteTensorLoc(builder, subgraph_index, index, const_loc));
17651827
vals_map[index] = op_or_err.value()->getResult(0);
17661828
}
17671829
return_operands.push_back(vals_map[index]);
@@ -2039,8 +2101,8 @@ OwningOpRef<mlir::ModuleOp> tflite::FlatBufferToMlir(
20392101
SubgraphName(set_implicit_main_func, e.index(), *subgraph);
20402102
uint32_t subgraph_index = static_cast<uint32_t>(e.index());
20412103
auto func_or_error = ConvertSubgraph(
2042-
*model, *subgraph, name, model->operator_codes, func_names,
2043-
model->buffers, base_loc, builder,
2104+
*model, *subgraph, static_cast<int>(subgraph_index), name,
2105+
model->operator_codes, func_names, model->buffers, base_loc, builder,
20442106
/*is_entry_point=*/
20452107
set_implicit_main_func
20462108
? e.index() == 0

‎tensorflow/compiler/mlir/lite/quantization/common/quantization_lib/quantization_utils.h‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -490,6 +490,27 @@ class QuantizationPattern : public RewritePattern {
490490
(nodes_blocklist.find(sloc) != nodes_blocklist.end())) {
491491
return failure();
492492
}
493+
} else if (auto fused_loc =
494+
llvm::dyn_cast<FusedLoc>(quantizing_op->getLoc())) {
495+
llvm::SmallVector<mlir::Location, 4> stack;
496+
stack.push_back(fused_loc);
497+
bool blocked = false;
498+
while (!stack.empty()) {
499+
mlir::Location l = stack.pop_back_val();
500+
if (auto name_loc = llvm::dyn_cast<NameLoc>(l)) {
501+
std::string sloc = name_loc.getName().str();
502+
if (!sloc.empty() &&
503+
(nodes_blocklist.find(sloc) != nodes_blocklist.end())) {
504+
blocked = true;
505+
break;
506+
}
507+
} else if (auto fused = llvm::dyn_cast<FusedLoc>(l)) {
508+
for (auto child_loc : fused.getLocations()) {
509+
stack.push_back(child_loc);
510+
}
511+
}
512+
}
513+
if (blocked) return failure();
493514
}
494515
}
495516

0 commit comments

Comments
 (0)