-
Notifications
You must be signed in to change notification settings - Fork 56
Add support for dynamic workload type #489
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for dynamic workload type #489
Conversation
| if (dynamic_workload_type == "EFFICIENT") { | ||
| auto ov_execution_network = concrete_backend_->GetOVCompiledModel(); | ||
| ov_execution_network.set_property({{"WORKLOAD_TYPE", "EFFICIENT"}}); | ||
| LOGS_DEFAULT(VERBOSE) << "[OpenVINO-EP]" << " OV WORKLOAD_TYPE property is set to EFFICIENT mode for OV inference"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please fix cpp lint issues
|
Hi Preetha - 1] the string comparison needed an explicit cast to std::string for it to work in my testcase. 2] The requirement from MSFT is that the paramter is not coupled to a inference request. If you recall, they detailed a concern in the original code they provided where they wanted to move from efficient to default for work that had not returned. To do that they moved from changing the parameter in conjuction with an infer request to an asyncronous setting. To do that, we need to set the cmpiled model parameter from the openvin_execution_provider directly (via a pointer to the compiled model for that session). See sample code here - https://github.com/intel-sandbox/AIPPG/tree/main/src/PowerPerformanceGuide/assets/Toolbox/efficient_cpp |
Yeah you are right with the string comparison logic. I found it during validation. I have modified it to locate the values independent of the keys location. As they are two separate char arrays an end user may not be assigning values and keys in same order. If we want to restrict I can modify this. |
| if (strcmp(key, "ep.dynamic.workload_type") == 0) { | ||
| for (auto val : values) { | ||
| if (strcmp(val, "Efficient") == 0) { | ||
| backend_manager_->dynamic_workload_type = "EFFICIENT"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@preetha-intel , Can we set the OV WorkloadType property inside this function directly and not rely for Compute/ Infer to be called which is why MSFT made the API change
Below are 2 lines of code in backendmanager.cc->Compute which we need to call in SetEpDynamicOptions function for both Default and Efficient.
auto ov_execution_network = concrete_backend_->GetOVCompiledModel();
ov_execution_network.set_property({{"WORKLOAD_TYPE", "DEFAULT"}});
* Add support for dynmaic workload type * Fix the iteration on gsl::span dynamic keys and values * Fix lint issues
* Add support for dynmaic workload type * Fix the iteration on gsl::span dynamic keys and values * Fix lint issues
* Add support for dynmaic workload type * Fix the iteration on gsl::span dynamic keys and values * Fix lint issues
Description
Support to set EPdynamic options in OVEP
Motivation and Context
relate to microsoft#22282