-
Notifications
You must be signed in to change notification settings - Fork 5k
3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release #16042
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
* Fix rpc channel leak due to concurrent operation * Throw channel create failed exception --------- Co-authored-by: Rick Cheng <[email protected]>
…apache#15540) * Reduce the size of tarball to continue ASF release for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt # Conflicts: # tools/dependencies/known-dependencies.txt
Co-authored-by: Eric Gao <[email protected]>
…ers (apache#16033) Co-authored-by: Rick Cheng <[email protected]>
…/project parameters data type (apache#15967) * [Improvement][parameter] New data types and type filtering * [Improvement][parameter] Improvement startup parameters/global parameters data type * fix api interfaces compatible * add project parameter data type default value * [Improvement][parameter] New data types and type filtering * [Improvement][parameter] Improvement startup parameters/global parameters data type * fix api interfaces compatible * add project parameter data type default value * improvement project code * remove useless imports * remove method onClearSearchTaskType * add parameter doc * optimisation logic * code conflict resolution * code conflict resolution * [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (apache#15978) * update * test * add monitor enhance ui * update * update * update doc * fix spotless * update * update * Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java Co-authored-by: Wenjun Ruan <[email protected]> * Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml Co-authored-by: Wenjun Ruan <[email protected]> * update * fix spotless * update --------- Co-authored-by: Wenjun Ruan <[email protected]> * [Improvement][Monitor] Add UT for montor (apache#15998) * formatting Code
| # This is the version number of the application being deployed. This version number should be | ||
| # incremented each time you make changes to the application. | ||
| appVersion: 3.1.0 | ||
| appVersion: 3.2.2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this appVersion should be changed.
dolphinscheduler.master.labels quotes appVersion
dolphinscheduler/deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl
Lines 51 to 64 in 56ecec7
| {{- define "dolphinscheduler.common.labels" -}} | |
| app.kubernetes.io/instance: {{ .Release.Name }} | |
| app.kubernetes.io/managed-by: {{ .Release.Service }} | |
| app.kubernetes.io/version: {{ .Chart.AppVersion }} | |
| {{- end -}} | |
| {{/* | |
| Create a master labels. | |
| */}} | |
| {{- define "dolphinscheduler.master.labels" -}} | |
| app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-master | |
| app.kubernetes.io/component: master | |
| {{ include "dolphinscheduler.common.labels" . }} | |
| {{- end -}} |
And dolphinscheduler.master.labels as master's matchLabels, but matchLabels are immutable, which means that, Updating helm charts for an already deployed dolphinscheduler will fail
Lines 27 to 29 in 56ecec7
| selector: | |
| matchLabels: | |
| {{- include "dolphinscheduler.master.labels" . | nindent 6 }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we still want to modify the appVersion, then we should remove the app.kubernetes.io/version: {{ .Chart.AppVersion }} from dolphinscheduler.common.labels first
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| <dependency> | ||
| <groupId>org.apache.httpcomponents</groupId> | ||
| <artifactId>httpclient</artifactId> | ||
| <version>4.5.13</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.httpcomponents</groupId> | ||
| <artifactId>httpmime</artifactId> | ||
| <version>4.5.13</version> | ||
| </dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see any related test cases being added. Why add this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
delete it
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-all</artifactId> | ||
| <artifactId>dolphinscheduler-task-all-prune</artifactId> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reduce release package size
Co-authored-by: xiangzihao <[email protected]>
Fix the PreparedStatement parameter is TIME, set it to java.sql.Time Co-authored-by: Aaron Wang <[email protected]> Co-authored-by: Rick Cheng <[email protected]>
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-datax</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-flink</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-flink-stream</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-http</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-mr</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-pigeon</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-procedure</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-python</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-seatunnel</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-shell</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-spark</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-sql</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-sqoop</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-emr</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-zeppelin</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-jupyter</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-k8s</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-mlflow</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-openmldb</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-dvc</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-dinky</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-java</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-sagemaker</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-pytorch</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-hivecli</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-chunjun</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-dms</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-datasync</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-kubeflow</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-linkis</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-datafactory</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.dolphinscheduler</groupId> | ||
| <artifactId>dolphinscheduler-task-remoteshell</artifactId> | ||
| <version>${project.version}</version> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>*</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we do this, please inform the user in the documentation that this is an incompatible change. This may lead to serious production accidents. cc @ruanwenjun @EricGao888 @Radeity
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is a sad solution from 3.2.1, since no-one want to refactor the release architecture. Right now, the release package size is 1.x g, large than ASF limit. I create #15960 to do this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 3.2.2-prepare #16042 +/- ##
===================================================
+ Coverage 40.58% 40.66% +0.07%
- Complexity 5216 5225 +9
===================================================
Files 1380 1380
Lines 46056 46011 -45
Branches 4916 4919 +3
===================================================
+ Hits 18693 18710 +17
+ Misses 25437 25376 -61
+ Partials 1926 1925 -1 ☔ View full report in Codecov by Sentry. |
|
Please retry analysis of this Pull-Request directly on SonarCloud |

Purpose of the pull request
It is prepared for version 3.2.2, mainly to reduce the size of the release package and fix the helm version problem.
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(or)
If your pull request contain incompatible change, you should also add it to
docs/docs/en/guide/upgrede/incompatible.md