Skip to content

Performance barrier tests are too easy to deteriorate #68726

@rschu1ze

Description

@rschu1ze

Look at this beautiful CI overview page ... all checks are green:
https://s3.amazonaws.com/clickhouse-test-reports/65761/3ee741bd5e33d16b2f5711a8f2b06fca1a64b7bc/ci_running.html

Then click the link "Performance Comparison (release) [1/4]"
https://s3.amazonaws.com/clickhouse-test-reports/65761/3ee741bd5e33d16b2f5711a8f2b06fca1a64b7bc/performance_comparison__release__[1_4]/report.html
and find that lower_upper_utf8 regressed by 16x.

Currently, only timeouts in performance tests let the CI check fail. In case of performance regressions > N%, we should also fail.

(for context, see #68711)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions