Revert "Merge pull request #38212 from azat/no-stress" by azat · Pull Request #38750 · ClickHouse/ClickHouse

azat · 2022-07-03T11:54:34Z

This reverts commit 24d5be1, reversing
changes made to a756b4b.

After #38717 there is no need in --no-stress, since there will be no
such checks for previous releases that fails.

Reverts: #38212 (cc @qoega @tavplubix )
Requires: #38717

Changelog category (leave one):

Not for changelog (changelog entry is not required)

azat · 2022-07-08T06:48:46Z

Stateless tests flaky check (address, actions) — fail: 1, passed: 99

2022-07-08 00:53:09 [fec491aa32e2] 2022.07.08 00:53:09.155936 [ 602 ] {0253a950-8580-48ce-bf2f-277b90f92e2f}  executeQuery: Code: 57. DB::Exception: Cannot attach table with UUID a19baf4a-a905-4a9f-a8d9-c85e6aefb9b5, because it was detached but still used by some query. Retry later. (TABLE_ALREADY_EXISTS) (version 22.7.1.1) (from [::1]:36882) (comment: 01646_system_restart_replicas_smoke.sql) (in query: SYSTEM RESTART REPLICAS;), Stack trace (when copying this message, always include the lines below):

Stress test (memory, actions) — OOM killer (or signal 9) in clickhouse-server.log

Fix possible deadlock with OvercommitTracker and send_logs_level #38990

Stress test (undefined, actions) — Backward compatibility check: Error message in clickhouse-server.log (see bc_check_error_messages.txt)

2022.07.08 00:51:54.714243 [ 1236430 ] {} <Error> DatabaseCatalog: Cannot load partially dropped table test_35.ttl_table2 (bee8350b-7238-40b0-88ad-511bd1936dbd) from: /var/lib/clickhouse/metadata_dropped/test_35.ttl_table2.bee8350b-7238-40b0-88ad-511bd1936dbd.sql. Parsed query: ATTACH TABLE test_35.ttl_table2 UUID 'bee8350b-7238-40b0-88ad-511bd1936dbd' (`key` DateTime) ENGINE = ReplicatedMergeTree('/test/01921_concurrent_ttl_and_normal_merges/01921_concurrent_ttl_and_normal_merges_zookeeper_long_test_35/ttl_table', '2') ORDER BY tuple() TTL key + toIntervalSecond(1) SETTINGS merge_with_ttl_timeout = 1, max_replicated_merges_with_ttl_in_queue = 0, max_number_of_merges_with_ttl_in_pool = 100, cleanup_delay_period = 1, cleanup_delay_period_random_add = 0, index_granularity = 8192. Will remove metadata and store/bee/bee8350b-7238-40b0-88ad-511bd1936dbd/. Garbage may be left in ZooKeeper.: Code: 231. DB::Exception: The local set of parts of table test_35.ttl_table2 (bee8350b-7238-40b0-88ad-511bd1936dbd) doesn't look like the set of parts in ZooKeeper: 20.00 rows of 35.00 total rows in filesystem are suspicious. There are 4 uncovered unexpected parts with 20 rows (0 of them is not just-written with 0 rows), 0 missing parts (with 0 blocks), 0 covered unexpected parts (with 0 rows). (TOO_MANY_UNEXPECTED_DATA_PARTS), Stack trace (when copying this message, always include the lines below):

Try to improve backward compatibility check #38717 initially removes that Error messages check in logs, however now it is there again...

tavplubix · 2022-07-08T11:06:51Z

See also #38629 (comment)

alexey-milovidov · 2022-07-30T22:30:02Z

@azat It still fails:

Stress test (undefined, actions) — Backward compatibility check: Error message in clickhouse-server.log

azat · 2022-07-31T11:32:41Z

Originally I though that #38717 will remove that additional check against error log, since usual (non bc check) does not have it, and since bc check uses test from particular release, what is the purpose if this check then? If it is important, then why there is no such check for non-bc check?

Also this check already has lots of excludes, is it really useful?

ClickHouse/docker/test/stress/run.sh

Lines 365 to 392 in eaeb044

    
           zgrep -Fav -e "Code: 236. DB::Exception: Cancelled merging parts" \ 
        
                      -e "Code: 236. DB::Exception: Cancelled mutating parts" \ 
        
                      -e "REPLICA_IS_ALREADY_ACTIVE" \ 
        
                      -e "REPLICA_IS_ALREADY_EXIST" \ 
        
                      -e "ALL_REPLICAS_LOST" \ 
        
                      -e "DDLWorker: Cannot parse DDL task query" \ 
        
                      -e "RaftInstance: failed to accept a rpc connection due to error 125" \ 
        
                      -e "UNKNOWN_DATABASE" \ 
        
                      -e "NETWORK_ERROR" \ 
        
                      -e "UNKNOWN_TABLE" \ 
        
                      -e "ZooKeeperClient" \ 
        
                      -e "KEEPER_EXCEPTION" \ 
        
                      -e "DirectoryMonitor" \ 
        
                      -e "TABLE_IS_READ_ONLY" \ 
        
                      -e "Code: 1000, e.code() = 111, Connection refused" \ 
        
                      -e "UNFINISHED" \ 
        
                      -e "Renaming unexpected part" \ 
        
                      -e "PART_IS_TEMPORARILY_LOCKED" \ 
        
                      -e "and a merge is impossible: we didn't find" \ 
        
                      -e "found in queue and some source parts for it was lost" \ 
        
                      -e "is lost forever." \ 
        
                      -e "Unknown index: idx." \ 
        
                      -e "Cannot parse string 'Hello' as UInt64" \ 
        
                      -e "} <Error> TCPHandler: Code:" \ 
        
                      -e "} <Error> executeQuery: Code:" \ 
        
                      -e "Missing columns: 'v3' while processing query: 'v3, k, v1, v2, p'" \ 
        
                      -e "This engine is deprecated and is not supported in transactions" \ 
        
                      -e "[Queue = DB::MergeMutateRuntimeQueue]: Code: 235. DB::Exception: Part" \

alexey-milovidov · 2022-10-07T23:34:17Z

@Avogar can answer this question.

alexey-milovidov · 2023-01-29T16:45:20Z

@azat it does not work.

alexey-milovidov · 2023-02-15T14:11:52Z

@azat it does not work.

This reverts commit 24d5be1, reversing changes made to a756b4b. After ClickHouse#38717 there is no need in --no-stress, since there will be no such checks for previous releases that fails. Reverts: ClickHouse#38212 Requires: ClickHouse#38717

azat · 2023-02-15T21:14:21Z

/run.sh: line 285: ./stress: Permission denied

Fixed.

But, note, that I don't like this patch, since without no-stress tag, test 01646_system_restart_replicas_smoke will likely break BC check, since this test introduce too much various issues.

alexey-milovidov · 2023-02-16T19:14:10Z

FYI people from GitHub confirmed a bug that editing a file in the UI can reset its executable mode.

tavplubix · 2023-02-21T16:09:17Z

usage: clickhouse-test [-h] [-q QUERIES] [--tmp TMP] [-b BINARY] [-c CLIENT]
                       [--extract_from_config EXTRACT_FROM_CONFIG]
                       [--configclient CONFIGCLIENT]
                       [--configserver CONFIGSERVER] [-o OUTPUT] [-t TIMEOUT]
                       [--global_time_limit GLOBAL_TIME_LIMIT] [-d] [--stop]
                       [--order {asc,desc,random}] [--testname] [--hung-check]
                       [--no-left-queries-check] [--force-color]
                       [--database DATABASE] [--no-drop-if-fail]
                       [--hide-db-name] [--parallel PARALLEL] [-j [JOBS]]
                       [--test-runs [TEST_RUNS]] [-U UNIFIED]
                       [-r SERVER_CHECK_RETRIES] [--db-engine DB_ENGINE]
                       [--replicated-database] [--fast-tests-only]
                       [--no-stateless] [--no-stateful]
                       [--skip SKIP [SKIP ...]]
                       [--sequential SEQUENTIAL [SEQUENTIAL ...]] [--no-long]
                       [--client-option CLIENT_OPTION [CLIENT_OPTION ...]]
                       [--print-time] [--check-zookeeper-session]
                       [--s3-storage] [--no-random-settings]
                       [--no-random-merge-tree-settings]
                       [--run-by-hash-num RUN_BY_HASH_NUM]
                       [--run-by-hash-total RUN_BY_HASH_TOTAL]
                       [--zookeeper | --no-zookeeper]
                       [--shard | --no-shard | --backward-compatibility-check]
                       [--secure] [--max-failures-chain MAX_FAILURES_CHAIN]
                       [--report-coverage] [--report-logs-stats]
                       [--no-parallel-replicas]
                       [test [test ...]]
clickhouse-test: error: unrecognized arguments: --stress

So stress tests were testing nothing for the last 4 days: https://play.clickhouse.com/play?user=play#d2l0aCAKJyUlJyBhcyBuYW1lX3BhdHRlcm4sCicyMDIyLTA5LTAxJyBhcyBzdGFydF9kYXRlLAooJ1N0YXRlbGVzcyB0ZXN0cyAoYXNhbiknLCAnU3RhdGVsZXNzIHRlc3RzIChhZGRyZXNzKScsICdTdGF0ZWxlc3MgdGVzdHMgKGFkZHJlc3MsIGFjdGlvbnMpJykgYXMgYmFja3BvcnRfYW5kX3JlbGVhc2Vfc3BlY2lmaWNfY2hlY2tzCnNlbGVjdCAKdG9TdGFydE9mRGF5KGNoZWNrX3N0YXJ0X3RpbWUpIGFzIGQsCmNvdW50KCksICBncm91cFVuaXFBcnJheShwdWxsX3JlcXVlc3RfbnVtYmVyKSwgYW55KHJlcG9ydF91cmwpCmZyb20gY2hlY2tzIHdoZXJlIHN0YXJ0X2RhdGUgPD0gY2hlY2tfc3RhcnRfdGltZSBhbmQgcHVsbF9yZXF1ZXN0X251bWJlciBub3QgaW4gCihzZWxlY3QgcHVsbF9yZXF1ZXN0X251bWJlciBhcyBwcm4gZnJvbSBjaGVja3Mgd2hlcmUgcHJuIT0wIGFuZCBzdGFydF9kYXRlIDw9IGNoZWNrX3N0YXJ0X3RpbWUgYW5kIGNoZWNrX25hbWUgaW4gYmFja3BvcnRfYW5kX3JlbGVhc2Vfc3BlY2lmaWNfY2hlY2tzKSAKYW5kIHRlc3RfbmFtZSBsaWtlIG5hbWVfcGF0dGVybiBhbmQgdGVzdF9jb250ZXh0X3JhdyBub3QgbGlrZSAnJUNhbm5vdFNlbmRSZXF1ZXN0JScgYW5kIGNoZWNrX25hbWUgaWxpa2UgJyVzdHJlc3MlJwphbmQgdGVzdF9zdGF0dXMgaW4gKCdGQUlMJywgJ0ZMQUtZJykgZ3JvdXAgYnkgZCBvcmRlciBieSBkIGRlc2M=

azat · 2023-02-21T16:46:38Z

@tavplubix thank you! Too bad that I haven't noticed this, since in my mind stress tests may periodically fails because of 01646_system_restart_replicas_smoke test (as I have repeatedly noticed), and it could fail even in this PR, and then it will not be merged.

But let's see, maybe it will work...

azat marked this pull request as draft July 3, 2022 11:54

robot-ch-test-poll2 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jul 3, 2022

azat force-pushed the revert-no-stress branch from 86d67c4 to 40db496 Compare July 7, 2022 18:41

azat marked this pull request as ready for review July 7, 2022 18:41

tavplubix approved these changes Jul 7, 2022

View reviewed changes

tavplubix self-assigned this Jul 7, 2022

azat marked this pull request as draft July 8, 2022 05:25

azat mentioned this pull request Jul 12, 2022

tests: enable back 02232_dist_insert_send_logs_level_hung #39124

Merged

alexey-milovidov approved these changes Jul 30, 2022

View reviewed changes

alexey-milovidov approved these changes Jan 29, 2023

View reviewed changes

alexey-milovidov marked this pull request as ready for review January 29, 2023 03:17

alexey-milovidov assigned alexey-milovidov and unassigned tavplubix Jan 29, 2023

azat force-pushed the revert-no-stress branch from f937030 to bddead1 Compare February 15, 2023 21:12

alexey-milovidov merged commit e66c05e into ClickHouse:master Feb 16, 2023

azat deleted the revert-no-stress branch February 16, 2023 19:20

tavplubix mentioned this pull request Feb 21, 2023

Break Stress tests #46663

Merged

tavplubix mentioned this pull request Feb 28, 2023

Possible deadlock on shutdown #47040

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert "Merge pull request #38212 from azat/no-stress"#38750

Revert "Merge pull request #38212 from azat/no-stress"#38750
alexey-milovidov merged 1 commit intoClickHouse:masterfrom
azat:revert-no-stress

azat commented Jul 3, 2022

Uh oh!

azat commented Jul 8, 2022

Uh oh!

tavplubix commented Jul 8, 2022

Uh oh!

alexey-milovidov commented Jul 30, 2022

Uh oh!

azat commented Jul 31, 2022

Uh oh!

alexey-milovidov commented Oct 7, 2022

Uh oh!

alexey-milovidov commented Jan 29, 2023

Uh oh!

alexey-milovidov commented Feb 15, 2023

Uh oh!

azat commented Feb 15, 2023

Uh oh!

alexey-milovidov commented Feb 16, 2023

Uh oh!

tavplubix commented Feb 21, 2023

Uh oh!

azat commented Feb 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

azat commented Jul 3, 2022

Changelog category (leave one):

Uh oh!

azat commented Jul 8, 2022

Uh oh!

tavplubix commented Jul 8, 2022

Uh oh!

alexey-milovidov commented Jul 30, 2022

Uh oh!

azat commented Jul 31, 2022

Uh oh!

alexey-milovidov commented Oct 7, 2022

Uh oh!

alexey-milovidov commented Jan 29, 2023

Uh oh!

alexey-milovidov commented Feb 15, 2023

Uh oh!

azat commented Feb 15, 2023

Uh oh!

alexey-milovidov commented Feb 16, 2023

Uh oh!

tavplubix commented Feb 21, 2023

Uh oh!

azat commented Feb 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants