Skip to content

Improve backtrace catching on server failures in CI for stress tests#20462

Merged
alexey-milovidov merged 3 commits intoClickHouse:masterfrom
azat:stress-backtrace-on-failure
Feb 19, 2021
Merged

Improve backtrace catching on server failures in CI for stress tests#20462
alexey-milovidov merged 3 commits intoClickHouse:masterfrom
azat:stress-backtrace-on-failure

Conversation

@azat
Copy link
Copy Markdown
Member

@azat azat commented Feb 13, 2021

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Follow-up for: #19524
Cc: @alexey-milovidov

@robot-clickhouse robot-clickhouse added the pr-not-for-changelog This PR should not be mentioned in the changelog label Feb 13, 2021
@alexey-milovidov
Copy link
Copy Markdown
Member

#19524 does not handle SIGABRT deliberately.
Because there should not be the cases when we cannot print our own diagnostics.

@alexey-milovidov alexey-milovidov self-assigned this Feb 13, 2021
@alexey-milovidov
Copy link
Copy Markdown
Member

I see the motivation here: #19580 (comment), let's try...

@azat azat force-pushed the stress-backtrace-on-failure branch from 025a217 to 42f7ca6 Compare February 14, 2021 08:38
@azat azat marked this pull request as draft February 14, 2021 15:42
@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 14, 2021

@azat azat changed the title stress: improve backtrace catching on server failures ci: improve backtrace catching on server failures Feb 14, 2021
@azat azat force-pushed the stress-backtrace-on-failure branch from 42f7ca6 to 8c9b507 Compare February 14, 2021 20:36
@alexey-milovidov
Copy link
Copy Markdown
Member

Note that it should not prevent receiving fatal messages with stack trace in client.

@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 14, 2021

Note that it should not prevent receiving fatal messages with stack trace in client.

Right. Actually I don't think that this should break this, since the server will continue after gdb will print stacktraces

@azat azat force-pushed the stress-backtrace-on-failure branch from 8c9b507 to 4b92b31 Compare February 15, 2021 07:24
@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 16, 2021

Stress tests are okay, but fasttest not:

https://clickhouse-test-reports.s3.yandex.net/20462/014c11af7fcaa60d62c10b1497bbdc02526d180c/fast_test/runlog.out.log

/ClickHouse/docker/test/fasttest/run.sh: line 121: gdb: command not found

By some reason fasttest image wasn't updated, because that layer was in cache https://clickhouse-test-reports.s3.yandex.net/20462/014c11af7fcaa60d62c10b1497bbdc02526d180c/push_to_dockerhub/taglatest.out.1.log

 ---> Using cache
 ---> 9446b5f83627
Step 6/17 : RUN apt-get update     && apt-get install         apt-transport-https         bash         brotli         build-essential         ca-certificates         ccache         clang-${LLVM_VERSION}         clang-tidy-${LLVM_VERSION}         cmake         curl         lsof         expect         fakeroot         git         gdb         gperf         lld-${LLVM_VERSION}         llvm-${LLVM_VERSION}         moreutils         ninja-build         psmisc         python3         python3-pip         python3-lxml         python3-requests         python3-termcolor         rename         software-properties-common         tzdata         unixodbc        --yes --no-install-recommends
 ---> Using cache

@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 16, 2021

72c61da does not helps with avoiding using docker cache for "new" layers, maybe it should be reset somehow? (@alesapin ?)

@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 17, 2021

@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 17, 2021

72c61da does not helps with avoiding using docker cache for "new" layers, maybe it should be reset somehow? (@alesapin ?)

Let's try install it as a separate layer then - c879719e7c1ed188cdd81a73148575cdbbacaf13

@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 17, 2021

Let's try install it as a separate layer then - c879719

Still no luck, moved into separate PR - #20864

@azat azat force-pushed the stress-backtrace-on-failure branch from c879719 to 770c340 Compare February 17, 2021 21:13
@azat azat changed the title ci: improve backtrace catching on server failures Improve backtrace catching on server failures in CI for stress tests Feb 17, 2021
@azat azat marked this pull request as ready for review February 17, 2021 21:13
@azat
Copy link
Copy Markdown
Member Author

azat commented Feb 19, 2021

Marker check Pending — Starting checks (No special conditions apply)

Looks hanged

@alexey-milovidov can you merge this? (maybe it will reproduce the problem and will have full stacktrace, in the meantime I will look into this #16481 (comment))

@alexey-milovidov alexey-milovidov merged commit ceacae4 into ClickHouse:master Feb 19, 2021
@azat azat deleted the stress-backtrace-on-failure branch February 19, 2021 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants