Academia.eduAcademia.edu

Figure from:Neural Network Quantization with AI Model Efficiency Toolkit (AIMET)

See full PDFdownloadDownload figure
Figure 2.2: A schematic of matrix-multiply logic in an neural network accelerator for quantized inference. Once the three quantization parameters are defined we can proceed with the quantization operation Starting from a real-valued vector x we first map it to the unsigned integer grid {0,...,2° — 1}:

Figure 2 2: A schematic of matrix-multiply logic in an neural network accelerator for quantized inference. Once the three quantization parameters are defined we can proceed with the quantization operation Starting from a real-valued vector x we first map it to the unsigned integer grid {0,...,2° — 1}:

Related Figures (14)

Figure 1.1: High-level view of AIMET architecture.
Figure 1.1: High-level view of AIMET architecture.
Figure 2.1: A schematic overview of matrix-multiply logic in neural network accelerator hardware. where sx is a floating-point scale factor and xin, is an integer vector, e.g., INT8. We denote this quantized version of the vector as X. By quantizing the weights and activations we can write the quantized version of the accumulation equation:
Figure 2.1: A schematic overview of matrix-multiply logic in neural network accelerator hardware. where sx is a floating-point scale factor and xin, is an integer vector, e.g., INT8. We denote this quantized version of the vector as X. By quantizing the weights and activations we can write the quantized version of the accumulation equation:
Figure 2.3: A visual explanation of the different uniform quantization grids for a bit-width of 8. s is the scaling factor, z the zero-point. The floating-point grid is in black, the integer quantized grid in blue.
Figure 2.3: A visual explanation of the different uniform quantization grids for a bit-width of 8. s is the scaling factor, z the zero-point. The floating-point grid is in black, the integer quantized grid in blue.
Figure 3.1: Schematic overview of quantized forward pass for a convolutional layer. To test how well a neural network would run on a quantized device, we often simulate the quantized behavior on the same general-purpose machine we use for training neural networks. This is called quantization simulation. Quantization simulation allows the user to efficiently test various quantization options. In this section, we first explain the fundamentals of this simulation process and then discuss techniques that help bridge the gap between simulated and the actual on-device performance. AIMET enables a user to automatically create a quantization simulation model, given a PyTorch or TensorFlow model. AIMET will add the quantizer nodes shown in figure 3.1 to the PyTorch or TensorFlow model graph. The resulting model graph can be used as-is in the user’s evaluation or training pipeline. Quantization Simulation
Figure 3.1: Schematic overview of quantized forward pass for a convolutional layer. To test how well a neural network would run on a quantized device, we often simulate the quantized behavior on the same general-purpose machine we use for training neural networks. This is called quantization simulation. Quantization simulation allows the user to efficiently test various quantization options. In this section, we first explain the fundamentals of this simulation process and then discuss techniques that help bridge the gap between simulated and the actual on-device performance. AIMET enables a user to automatically create a quantization simulation model, given a PyTorch or TensorFlow model. AIMET will add the quantizer nodes shown in figure 3.1 to the PyTorch or TensorFlow model graph. The resulting model graph can be used as-is in the user’s evaluation or training pipeline. Quantization Simulation
Figure 3.2: AIMET quantization simulation workflow. Workflow Figure 3.2 shows the workflow for using AIMET quantization simulation to simulate on-target quantized accuracy.
Figure 3.2: AIMET quantization simulation workflow. Workflow Figure 3.2 shows the workflow for using AIMET quantization simulation to simulate on-target quantized accuracy.
Figure 3.3: Workflow to export AIMET quantization simulation encodings. Quantization simulation does not change the model parameters itself. However, as part of creating the simulation model, AIMET computes optimal scale/offset quantization encodings for both parameters and activations in the model. These quantization encodings can be exported from AIMET. On-target runtimes, such as the Qualcomm Neural Processing SDK, can import these optimized quantization encodings and use them instead of computing quantization encodings independently.
Figure 3.3: Workflow to export AIMET quantization simulation encodings. Quantization simulation does not change the model parameters itself. However, as part of creating the simulation model, AIMET computes optimal scale/offset quantization encodings for both parameters and activations in the model. These quantization encodings can be exported from AIMET. On-target runtimes, such as the Qualcomm Neural Processing SDK, can import these optimized quantization encodings and use them instead of computing quantization encodings independently.
Activation range setting As the final step, we determine the quantization ranges of all data- dependent tensors in the network, such as activation tensors using SQNR based criteria for most of the layers. A small calibration dataset is required to perform this step.
Activation range setting As the final step, we determine the quantization ranges of all data- dependent tensors in the network, such as activation tensors using SQNR based criteria for most of the layers. A small calibration dataset is required to perform this step.
Figure 4.2: AIMET visualization example: per (output) channel weight ranges of the first depthwise- separable layer in MobileNetV2 after BN folding. The boxplots show the min and max value range for each channel before CLE.
Figure 4.2: AIMET visualization example: per (output) channel weight ranges of the first depthwise- separable layer in MobileNetV2 after BN folding. The boxplots show the min and max value range for each channel before CLE.
Figure 4.3: AIMET visualization example: per (output) channel weight ranges of the first depthwise- separable layer in MobileNetV2. The boxplots show the min and max value range for each channel after CLE. 4.3.1 Caveats
Figure 4.3: AIMET visualization example: per (output) channel weight ranges of the first depthwise- separable layer in MobileNetV2. The boxplots show the min and max value range for each channel after CLE. 4.3.1 Caveats
Figure 4.4: AdaRound versus round-to-nearest approach. operation has a significant impact on the performance of the network. AdaRound (adaptive rounding provides a theoretically sound, computationally fast method for rounding weights. It requires onl a small number of unlabeled data samples (example : 500 - 2,000 samples were used on visio tasks), no hyperparameter tuning or end-to-end finetuning, and can be applied to fully connecte and convolutional layers of any neural network. AdaRound optimizes a local loss function usin the unlabeled training or calibration data to adaptively decide whether to quantize a specific weigh to the integer value below or above. Figure 4.4 illustrates schematically the difference betwee round-to-nearest and AdaRound. During quantization simulation (cf. section 3.1), AIMET by defaul rounds the weights of a layer to the nearest value. However, for performance enhancement especiall for low bit-width integer quantization, AdaRound should be invoked.
Figure 4.4: AdaRound versus round-to-nearest approach. operation has a significant impact on the performance of the network. AdaRound (adaptive rounding provides a theoretically sound, computationally fast method for rounding weights. It requires onl a small number of unlabeled data samples (example : 500 - 2,000 samples were used on visio tasks), no hyperparameter tuning or end-to-end finetuning, and can be applied to fully connecte and convolutional layers of any neural network. AdaRound optimizes a local loss function usin the unlabeled training or calibration data to adaptively decide whether to quantize a specific weigh to the integer value below or above. Figure 4.4 illustrates schematically the difference betwee round-to-nearest and AdaRound. During quantization simulation (cf. section 3.1), AIMET by defaul rounds the weights of a layer to the nearest value. However, for performance enhancement especiall for low bit-width integer quantization, AdaRound should be invoked.
Figure 4.5: PTQ debugging flow chart. Error is the difference between floating-point and quantized model accuracy.
Figure 4.5: PTQ debugging flow chart. Error is the difference between floating-point and quantized model accuracy.
Figure 5.1: Forward and backward computation graph for quantization aware training with STE assumption. The post-training quantization techniques covered in the previous chapters are the first go-to tool i our quantization toolkit. They are very effective and fast to implement because they do not requir retraining the network with labeled data. However, for some models or use cases, they may b insufficient. Post-training techniques may not be enough to mitigate the significant quantization erro incurred by low-bit quantization. In these cases, we resort to quantization-aware training (QAT QAT models the quantization noise during training and allows the model to find better solution than post-training quantization. However, the higher accuracy comes with the usual costs of neura network training, i.e. longer training times, need for labeled data and hyperparameter search. In thi chapter, we briefly explore how back-propagation works in networks with simulated quantization an provide a standard pipeline for training models with QAT effectively.
Figure 5.1: Forward and backward computation graph for quantization aware training with STE assumption. The post-training quantization techniques covered in the previous chapters are the first go-to tool i our quantization toolkit. They are very effective and fast to implement because they do not requir retraining the network with labeled data. However, for some models or use cases, they may b insufficient. Post-training techniques may not be enough to mitigate the significant quantization erro incurred by low-bit quantization. In these cases, we resort to quantization-aware training (QAT QAT models the quantization noise during training and allows the model to find better solution than post-training quantization. However, the higher accuracy comes with the usual costs of neura network training, i.e. longer training times, need for labeled data and hyperparameter search. In thi chapter, we briefly explore how back-propagation works in networks with simulated quantization an provide a standard pipeline for training models with QAT effectively.
Figure 5.2: Quantization-aware training pipeline. The blue boxes represent the steps and the turquoise boxes recommended choices and grey box is an optional step.
Figure 5.2: Quantization-aware training pipeline. The blue boxes represent the steps and the turquoise boxes recommended choices and grey box is an optional step.