Skip to content

Conversation

@preetha-intel
Copy link

Description

Support to set EPdynamic options in OVEP

Motivation and Context

relate to microsoft#22282

if (dynamic_workload_type == "EFFICIENT") {
auto ov_execution_network = concrete_backend_->GetOVCompiledModel();
ov_execution_network.set_property({{"WORKLOAD_TYPE", "EFFICIENT"}});
LOGS_DEFAULT(VERBOSE) << "[OpenVINO-EP]" << " OV WORKLOAD_TYPE property is set to EFFICIENT mode for OV inference";

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix cpp lint issues

@hylandk-movidius
Copy link

hylandk-movidius commented Nov 6, 2024

Hi Preetha -

1] the string comparison needed an explicit cast to std::string for it to work in my testcase.
ORIG - if ((std::find(keys.begin(), keys.end(), "ep.dynamic.workload_type") != keys.end()) &&
(std::find(values.begin(), values.end(),"Efficient") != values.end())) {
backend_manager_->dynamic_workload_type = "EFFICIENT";
}
MOD - if ((std::find(keys.begin(), keys.end(), std::string("ep.dynamic.workload_type")) != keys.end()) &&
(std::find(values.begin(), values.end(), std::string("Efficient")) != values.end())) {
backend_manager_->dynamic_workload_type = "EFFICIENT";
}

2] The requirement from MSFT is that the paramter is not coupled to a inference request. If you recall, they detailed a concern in the original code they provided where they wanted to move from efficient to default for work that had not returned. To do that they moved from changing the parameter in conjuction with an infer request to an asyncronous setting. To do that, we need to set the cmpiled model parameter from the openvin_execution_provider directly (via a pointer to the compiled model for that session). See sample code here - https://github.com/intel-sandbox/AIPPG/tree/main/src/PowerPerformanceGuide/assets/Toolbox/efficient_cpp

@preetha-intel
Copy link
Author

Hi Preetha -

1] the string comparison needed an explicit cast to std::string for it to work in my testcase. ORIG - if ((std::find(keys.begin(), keys.end(), "ep.dynamic.workload_type") != keys.end()) && (std::find(values.begin(), values.end(),"Efficient") != values.end())) { backend_manager_->dynamic_workload_type = "EFFICIENT"; } MOD - if ((std::find(keys.begin(), keys.end(), std::string("ep.dynamic.workload_type")) != keys.end()) && (std::find(values.begin(), values.end(), std::string("Efficient")) != values.end())) { backend_manager_->dynamic_workload_type = "EFFICIENT"; }

2] The requirement from MSFT is that the paramter is not coupled to a inference request. If you recall, they detailed a concern in the original code they provided where they wanted to move from efficient to default for work that had not returned. To do that they moved from changing the parameter in conjuction with an infer request to an asyncronous setting. To do that, we need to set the cmpiled model parameter from the openvin_execution_provider directly (via a pointer to the compiled model for that session). See sample code here - https://github.com/intel-sandbox/AIPPG/tree/main/src/PowerPerformanceGuide/assets/Toolbox/efficient_cpp

Hi @hylandk-movidius

Yeah you are right with the string comparison logic. I found it during validation. I have modified it to locate the values independent of the keys location. As they are two separate char arrays an end user may not be assigning values and keys in same order. If we want to restrict I can modify this.

@preetha-intel preetha-intel merged commit ddd84c7 into ovep-develop-lnl-1.4 Nov 7, 2024
if (strcmp(key, "ep.dynamic.workload_type") == 0) {
for (auto val : values) {
if (strcmp(val, "Efficient") == 0) {
backend_manager_->dynamic_workload_type = "EFFICIENT";

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@preetha-intel , Can we set the OV WorkloadType property inside this function directly and not rely for Compute/ Infer to be called which is why MSFT made the API change

Below are 2 lines of code in backendmanager.cc->Compute which we need to call in SetEpDynamicOptions function for both Default and Efficient.
auto ov_execution_network = concrete_backend_->GetOVCompiledModel();
ov_execution_network.set_property({{"WORKLOAD_TYPE", "DEFAULT"}});

ankitm3k pushed a commit that referenced this pull request Dec 11, 2024
* Add support for dynmaic workload type

* Fix the iteration on gsl::span dynamic keys and values

* Fix lint issues
ankitm3k pushed a commit that referenced this pull request Dec 11, 2024
* Add support for dynmaic workload type

* Fix the iteration on gsl::span dynamic keys and values

* Fix lint issues
ankitm3k pushed a commit that referenced this pull request Dec 11, 2024
* Add support for dynmaic workload type

* Fix the iteration on gsl::span dynamic keys and values

* Fix lint issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants