I've seen lots of time when there should be zero difference in performance tests, while it shows unstable queries, i.e.:

• https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=78440&sha=1acce546888cdec7d7c77b87be7ee2e6b08203e0&name_0=PR&name_1=Performance%20Comparison%20%28amd_release%2Cmaster_head%2C2%2F3%29
• https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=78460&sha=1c3503b0fcf736c64a9ba82ff11c5c0d8d0bb349&name_0=PR&name_1=Performance%20Comparison%20%28amd_release%2Cmaster_head%2C2%2F3%29

Possible issues from the top of my head (from likely to unlikely):

• using disks over ramfs/tmpfs as before (mmay introduces latency, although if the data just written it should be in page cache)
• very unlikely, but ensure that there is only one numa node on the CI workers only 1
• maybe convert some tests from unnecessary data on disk
  • CREATE TABLE ... ENGINE = File() to INSERT INTO FUNCTION file() (i.e. number_formatting_formats.xml)
• maybe some automatic external aggregations?
• after refactoring zzz-perf-comparison-tweaks-config.xml is not used (sigh)

Also some other issues that does not looks good:

• right now performance tests runs against latest version from master

Referenced issues:

• Performance barrier tests are too easy to deteriorate #68726
• Analyzer: Broken performance tests #62245

Some observations/cases:

• Do not share async_read_counters between queries #83423 (comment)
• MemoryShared is 2x bigger - https://s3.amazonaws.com/clickhouse-test-reports/PRs/82441/308985c016e559f5f367261a5e5ea20e0c52cc68//performance_comparison_amd_release_master_head_1_3/report.html (due to debug symbols likely)

Old issues:

• perf: disable remap_executable to avoid iTLB multihit mitigation cost #31543