-
Notifications
You must be signed in to change notification settings - Fork 8k
Open
Labels
Description
Look at this beautiful CI overview page ... all checks are green:
https://s3.amazonaws.com/clickhouse-test-reports/65761/3ee741bd5e33d16b2f5711a8f2b06fca1a64b7bc/ci_running.html
Then click the link "Performance Comparison (release) [1/4]"
https://s3.amazonaws.com/clickhouse-test-reports/65761/3ee741bd5e33d16b2f5711a8f2b06fca1a64b7bc/performance_comparison__release__[1_4]/report.html
and find that lower_upper_utf8 regressed by 16x.
Currently, only timeouts in performance tests let the CI check fail. In case of performance regressions > N%, we should also fail.
(for context, see #68711)
antaljanosbenjamin