Skip to content

Fix flaky test_version_update_after_mutation/test.py::test_upgrade_while_mutation#48783

Merged
kssenii merged 3 commits intoClickHouse:masterfrom
kssenii:fix-flaky-test_upgrade_while_mutation
Apr 15, 2023
Merged

Fix flaky test_version_update_after_mutation/test.py::test_upgrade_while_mutation#48783
kssenii merged 3 commits intoClickHouse:masterfrom
kssenii:fix-flaky-test_upgrade_while_mutation

Conversation

@kssenii
Copy link
Copy Markdown
Member

@kssenii kssenii commented Apr 14, 2023 •
edited
Loading

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Closes #48377.

the test test_version_update_after_mutation/test.py::test_upgrade_while_mutation does
node3.restart_with_latest_version(signal=9, fix_metadata=True)
then on server startup it sees that data/system is not empty
actually there are only some leftovers

data/system
└── part_log
    ├── detached
    └── format_version.txt

because right before server was shutdown, it managed to create only part_log.sql.tmp, but did not manage to rename to part_log.sql

<Debug> SystemLog (system.part_log): Creating new table system.part_log for PartLog

then on server restart it removed part_log.sql.tmp :
<Information> DatabaseOrdinary (system): Removing file /var/lib/clickhouse/metadata/system/part_log.sql.tmp
and then started to move system table from ordinary to atomic db
but did not delete data/system/part_log/detached and data/system/part_log/format_version.txt because it transferred to atomic db only data of tables which were successfully loaded.
Therefore there was an exception on server startup

Cannot drop: filesystem error: in remove: Directory not empty ["/var/lib/clickhouse/data/system/"].

@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Apr 14, 2023
@tavplubix tavplubix self-assigned this Apr 14, 2023
alesapin
alesapin reviewed Apr 14, 2023
View reviewed changes
@kssenii
Copy link
Copy Markdown
Member Author

kssenii commented Apr 15, 2023

ClickHouse special build check — 7/8 artifact groups are OK

Apr 14 15:29:46 [10005/11759] Building CXX object src/CMakeFiles/dbms.dir/Processors/Formats/IOutputFormat.cpp.o
Apr 14 15:29:49 [10006/11759] Building CXX object src/CMakeFiles/dbms.dir/Processors/Formats/PullingOutputFormat.cpp.o
Apr 14 15:29:51 [10007/11759] Building CXX object src/CMakeFiles/dbms.dir/Processors/Executors/PipelineExecutor.cpp.o
time="2023-04-14T15:29:56Z" level=error msg="error waiting for container: unexpected EOF"
Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/_temp/build_check/ClickHouse/docker/packager/./packager", line 495, in <module>
    main()
  File "/home/ubuntu/actions-runner/_work/_temp/build_check/ClickHouse/docker/packager/./packager", line 483, in main
    run_docker_image_with_env(
  File "/home/ubuntu/actions-runner/_work/_temp/build_check/ClickHouse/docker/packager/./packager", line 112, in run_docker_image_with_env
    subprocess.check_call(cmd, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker run --network=host --user=1000:1000 --rm --volume=/home/ubuntu/actions-runner/_work/_temp/build_check/binary_tidy:/output --volume=/home/ubuntu/actions-runner/_work/_temp/build_check/ClickHouse:/build  -e OUTPUT_DIR=/output -e DEB_ARCH=amd64 -e CC=clang-15 -e CXX=clang++-15 -e BUILD_TYPE=Debug -e SCCACHE_BUCKET=clickhouse-builds -e SCCACHE_S3_KEY_PREFIX=ccache/sccache -e CTCACHE_S3_BUCKET=clickhouse-builds -e CTCACHE_S3_FOLDER=ccache/clang-tidy-cache -e NINJA_FLAGS=-k0 -e VERSION_STRING='23.4.1.1' -e CMAKE_FLAGS="$CMAKE_FLAGS -DCMAKE_C_COMPILER=clang-15 -DCMAKE_CXX_COMPILER=clang++-15 -DCOMPILER_CACHE=sccache -DENABLE_CLANG_TIDY=1 -DENABLE_TESTS=1 -DENABLE_EXAMPLES=1 -DENABLE_UTILS=1" -e BUILD_TARGET='all'  clickhouse/binary-builder:latest' returned non-zero exit status 125.

:/

@kssenii kssenii merged commit e312c4c into ClickHouse:master Apr 15, 2023
@KochetovNicolai KochetovNicolai mentioned this pull request Jul 31, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alesapin alesapin alesapin left review comments

@tavplubix tavplubix tavplubix approved these changes

Assignees

@tavplubix tavplubix

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Flaky test test_version_update_after_mutation/test.py::test_upgrade_while_mutation

4 participants

@kssenii @alesapin @tavplubix @robot-ch-test-poll1