Skip to content

Conversation

@ruanwenjun
Copy link
Member

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

kezhenxu94
kezhenxu94 previously approved these changes Apr 8, 2023
davidzollo
davidzollo previously approved these changes Apr 8, 2023
Copy link
Contributor

@davidzollo davidzollo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@ruanwenjun ruanwenjun dismissed stale reviews from davidzollo and kezhenxu94 via 05c9980 April 8, 2023 14:38
@ruanwenjun ruanwenjun force-pushed the dev_wenjun_usePercentageInCPUMemory branch from eb662f6 to 05c9980 Compare April 8, 2023 14:38
@ruanwenjun ruanwenjun added 3.2.0 for 3.2.0 version feature new feature labels Apr 8, 2023
@ruanwenjun ruanwenjun added this to the 3.2.0 milestone Apr 8, 2023
@codecov-commenter
Copy link

codecov-commenter commented Apr 8, 2023

Codecov Report

Merging #13896 (8c78a54) into dev (d772f24) will decrease coverage by 0.05%.
The diff coverage is 48.71%.

❗ Current head 8c78a54 differs from pull request most recent head 7d164f5. Consider uploading reports for the commit 7d164f5 to get more accurate results

@@             Coverage Diff              @@
##                dev   #13896      +/-   ##
============================================
- Coverage     38.91%   38.87%   -0.05%     
+ Complexity     4465     4453      -12     
============================================
  Files          1158     1158              
  Lines         42424    42414      -10     
  Branches       4772     4776       +4     
============================================
- Hits          16510    16488      -22     
- Misses        24097    24106       +9     
- Partials       1817     1820       +3     
Impacted Files Coverage Δ
...inscheduler/alert/registry/AlertHeartbeatTask.java 0.00% <0.00%> (ø)
...api/service/impl/ProcessDefinitionServiceImpl.java 35.09% <0.00%> (ø)
...inscheduler/server/master/config/MasterConfig.java 1.88% <0.00%> (-0.08%) ⬇️
...eduler/server/master/task/MasterHeartBeatTask.java 4.34% <0.00%> (+0.34%) ⬆️
...inscheduler/server/worker/config/WorkerConfig.java 3.84% <ø> (ø)
.../apache/dolphinscheduler/common/utils/OSUtils.java 29.93% <18.18%> (-3.80%) ⬇️
...heduler/api/service/impl/SchedulerServiceImpl.java 30.50% <33.33%> (ø)
...eduler/server/worker/task/WorkerHeartBeatTask.java 80.43% <90.90%> (-0.82%) ⬇️
...pache/dolphinscheduler/service/cron/CronUtils.java 53.78% <100.00%> (+2.02%) ⬆️

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@Radeity Radeity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @ruanwenjun , i just wonder why we use percentage representation? For me, physical memory is better.

@ruanwenjun ruanwenjun force-pushed the dev_wenjun_usePercentageInCPUMemory branch 2 times, most recently from 0a8dfbc to 60cdf22 Compare April 10, 2023 03:28
@ruanwenjun
Copy link
Member Author

Hi, @ruanwenjun , i just wonder why we use percentage representation? For me, physical memory is better.

Since the instance's memory/cpu might different, use percentage is more suitable, otherwise we need to set different config.

@ruanwenjun ruanwenjun force-pushed the dev_wenjun_usePercentageInCPUMemory branch from 60cdf22 to 7d164f5 Compare April 10, 2023 03:31
@sonarqubecloud
Copy link

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

26.5% 26.5% Coverage
0.0% 0.0% Duplication

@Radeity
Copy link
Member

Radeity commented Apr 10, 2023

Since the instance's memory/cpu might different, use percentage is more suitable, otherwise we need to set different config.

If physical memory of different worker vary significantly, such as workerA has 8G memory and workerB has 64G, it's better to set different config, WDYT?

Anyway, we can still use different percentage to represent it if you think percentage is more intuitive :D

@ruanwenjun
Copy link
Member Author

Since the instance's memory/cpu might different, use percentage is more suitable, otherwise we need to set different config.

If physical memory of different worker vary significantly, such as workerA has 8G memory and workerB has 64G, it's better to set different config, WDYT?

Anyway, we can still use different percentage to represent it if you think percentage is more intuitive :D

I think we may more concern about the percentage of usage.

@EricGao888
Copy link
Member

Since the instance's memory/cpu might different, use percentage is more suitable, otherwise we need to set different config.

If physical memory of different worker vary significantly, such as workerA has 8G memory and workerB has 64G, it's better to set different config, WDYT?

Anyway, we can still use different percentage to represent it if you think percentage is more intuitive :D

Actually I think both make sense to me. May I ask whether it is possible to keep both of them and give users an option to configure whether by physical memory or by percentage?

@ruanwenjun
Copy link
Member Author

Since the instance's memory/cpu might different, use percentage is more suitable, otherwise we need to set different config.

If physical memory of different worker vary significantly, such as workerA has 8G memory and workerB has 64G, it's better to set different config, WDYT?
Anyway, we can still use different percentage to represent it if you think percentage is more intuitive :D

Actually I think both make sense to me. May I ask whether it is possible to keep both of them and give users an option to configure whether by physical memory or by percentage?

This is possible, we can provide different strategy, e.g percentage, absolute value. This will need to change the whole config.

avoid-overload:
    strategy: Percentage
    max-memory-usage: 70%
    max-cpu-usage: 80%    
    # strategy: AbsoluteValue 
    # max-memory-usage: 6G
    # max-cpu-usage: 6C

If this is needed, I will improve this in the next PR.

@Radeity
Copy link
Member

Radeity commented Apr 10, 2023

This is possible, we can provide different strategy, e.g percentage, absolute value. This will need to change the whole config.

avoid-overload:
    strategy: Percentage
    max-memory-usage: 70%
    max-cpu-usage: 80%    
    # strategy: AbsoluteValue 
    # max-memory-usage: 6G
    # max-cpu-usage: 6C

If this is needed, I will improve this in the next PR.

Maybe we can simply provide two settings, if satisfy both, we consider the node is overload, like the memory usage is higher than 70% and reserved memory is less than 6G.

@EricGao888
Copy link
Member

Since the instance's memory/cpu might different, use percentage is more suitable, otherwise we need to set different config.

If physical memory of different worker vary significantly, such as workerA has 8G memory and workerB has 64G, it's better to set different config, WDYT?
Anyway, we can still use different percentage to represent it if you think percentage is more intuitive :D

Actually I think both make sense to me. May I ask whether it is possible to keep both of them and give users an option to configure whether by physical memory or by percentage?

This is possible, we can provide different strategy, e.g percentage, absolute value. This will need to change the whole config.

avoid-overload:
    strategy: Percentage
    max-memory-usage: 70%
    max-cpu-usage: 80%    
    # strategy: AbsoluteValue 
    # max-memory-usage: 6G
    # max-cpu-usage: 6C

If this is needed, I will improve this in the next PR.

+1

Copy link
Contributor

@caishunfeng caishunfeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@caishunfeng caishunfeng merged commit 61a689a into apache:dev Apr 13, 2023
@ruanwenjun ruanwenjun deleted the dev_wenjun_usePercentageInCPUMemory branch January 6, 2024 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.2.0 for 3.2.0 version backend document feature new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants