Skip to content

Add log for new task count computation in CostBasedAutoScaler#18929

Merged
cryptoe merged 4 commits into
apache:masterfrom
Fly-Style:cost-autoscaler-logs
Jan 20, 2026
Merged

Add log for new task count computation in CostBasedAutoScaler#18929
cryptoe merged 4 commits into
apache:masterfrom
Fly-Style:cost-autoscaler-logs

Conversation

@Fly-Style

@Fly-Style Fly-Style commented Jan 18, 2026

Copy link
Copy Markdown
Contributor

Also this patch includes a mock-based tests for computeTaskCountForScaleAction

@Fly-Style Fly-Style changed the title Add intemediate log for new task count computation in CostBasedAutoScaler Add log for new task count computation in CostBasedAutoScaler Jan 18, 2026
@Fly-Style
Fly-Style force-pushed the cost-autoscaler-logs branch from 041382e to dd5f8ab Compare January 18, 2026 13:36
int currentTaskCount = 10;
int expectedOptimalCount = 17; // Higher than current

CostMetrics metrics = createMetrics(5000.0, currentTaskCount, PARTITION_COUNT, 0.1);

Check notice

Code scanning / CodeQL

Unread local variable Note test

Variable 'CostMetrics metrics' is never read.
);
}

private void setupMocksForMetricsCollection(int taskCount, double avgLag, double pollIdleRatio)

Check notice

Code scanning / CodeQL

Useless parameter Note test

The parameter 'pollIdleRatio' is never used.
int taskCount = -1;
if (optimalTaskCount >= currentTaskCount) {
taskCount = optimalTaskCount;
log.info("New task count [%d]", taskCount);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should we also log the supervisor id for this ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

taskCount = optimalTaskCount;
log.info("New task count [%d] on supervisor [%s]", taskCount, supervisorId);
} else {
log.info("No scaling required for supervisor [%s]", supervisorId);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How often does this func run ?
Worried a bit about log spam.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once per N minutes

@cryptoe
cryptoe merged commit dba356c into apache:master Jan 20, 2026
64 of 65 checks passed
@Fly-Style
Fly-Style deleted the cost-autoscaler-logs branch January 20, 2026 09:31
@kgyrtkirk kgyrtkirk added this to the 37.0.0 milestone Jan 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants