[NNAPI QDQ] Add QDQReshape op support#10533
Conversation
| if (IsQuantizedOp(node_unit)) { | ||
| AddQuantizationScaleAndZeroPointToSkip(model_builder, *node_unit.Inputs()[0].quant_param); // x_scale, x_zp | ||
| AddQuantizationScaleAndZeroPointToSkip(model_builder, *node_unit.Outputs()[0].quant_param); // y_scale, y_zp | ||
| } else { |
There was a problem hiding this comment.
The perm initializer needs always to be ignored here
| ORT_RETURN_IF_ERROR(IsValidInputQuantizedType(model_builder, input, x_scale, x_zero_point)); | ||
| } | ||
|
|
||
| return AddReshapeOperator(model_builder, node_unit, input, shape); |
There was a problem hiding this comment.
You need to pass the x_scale and x_zero_point into AddReshapeOperator, otherwise those will be set to 0
| if (!GetType(node_unit.Inputs()[0].node_arg, input_type)) | ||
| return false; | ||
|
|
||
| if (input_type != ONNX_NAMESPACE::TensorProto_DataType_FLOAT && |
There was a problem hiding this comment.
These are not required here, this is only for these operators
onnxruntime/onnxruntime/core/providers/nnapi/nnapi_builtin/builders/helper.cc
Lines 286 to 293 in bfb20b3
| Status AddToModelBuilderImpl(ModelBuilder& model_builder, const NodeUnit& node_unit) const override; | ||
| static bool CanSkipReshape(const ModelBuilder& model_builder, const NodeUnit& node_unit, | ||
| size_t input_rank, size_t output_rank); | ||
| static bool IsQuantizedOp(const NodeUnit& node_unit) ORT_MUST_USE_RESULT; // TODO, see if we want to move this to BaseOpBuilder |
There was a problem hiding this comment.
yes, @gwang-msft do you think we should move it to baseopbuilder now or keep it here at individual opbuilder level as we don't have that many qdq ops supported?
There was a problem hiding this comment.
We can move it as a virtual function for BaseOpBuilder and by default return false, each individual builder will override this if necessary
Same for BaseOpSupportChecker
| Shape shape_dimen = {static_cast<uint32_t>(shape.size())}; | ||
| std::string shape_name = model_builder.GetUniqueName(node_unit.Name() + input + "newshape"); | ||
| OperandType shape_operand_type(Type::TENSOR_INT32, shape_dimen); | ||
| OperandType shape_operand_type(Type::TENSOR_INT32, shape_dimen, scale, zero_point); |
There was a problem hiding this comment.
This is the new shape of the reshape op, it does not need scale and zero_point
| const std::string& input, | ||
| const std::vector<int32_t>& shape) { | ||
| const std::vector<int32_t>& shape, | ||
| float scale, int32_t zero_point) { |
There was a problem hiding this comment.
It will be easier to get the scale and zero_point from the input inside this function, instead of passing them in, use something like this
+ // For reshape, the output type should be the same as the input type except the shape is different
+ auto output_operand_type = operand_types.at(input);
+ output_operand_type.SetDimensions(shaper[output]);
+
// Since Reshape is not running using hardware in NNAPI for some CPU (e.g. Qualcomm SD for now)
// We will try to see if we the skip the Reshape to prevent context switching between
// NNAPI CPU impl and NNAPI hardware accelerator impl
if (CanSkipReshape(model_builder, node_unit, input_rank, output_rank)) {
// Since reshape can be skipped, only register the dimension and type, with same index and new name
- const OperandType output_operand_type(operand_types.at(input).type, shaper[output], scale, zero_point);
model_builder.RegisterOperand(output, operand_indices.at(input), output_operand_type, false);
} else {
// We still need to perform a reshape here
// Add new shape
Shape shape_dimen = {static_cast<uint32_t>(shape.size())};
std::string shape_name = model_builder.GetUniqueName(node_unit.Name() + input + "newshape");
- OperandType shape_operand_type(Type::TENSOR_INT32, shape_dimen, scale, zero_point);
+ OperandType shape_operand_type(Type::TENSOR_INT32, shape_dimen);
ORT_RETURN_IF_ERROR(model_builder.AddOperandFromPersistMemoryBuffer(shape_name, shape.data(), shape_operand_type));
input_indices.push_back(operand_indices.at(shape_name));
-
- const OperandType output_operand_type(operand_types.at(input).type, shaper[output], scale, zero_point);
ORT_RETURN_IF_ERROR(model_builder.AddOperation(ANEURALNETWORKS_RESHAPE, input_indices, {output}, {output_operand_type}, {false}));
}
3 participants
Description: Describe your changes.
Add QDQReshape op support and basic test case
Motivation and Context
More operator support