Cassandra container 3.0.26 fails to start on CI #23523
Description
Apache Airflow version
main (development)
What happened
Cassandra released a new image (3.0.26) on 05.05.2022 and it broke our builds, for example:
- https://github.com/apache/airflow/runs/6320170343?check_suite_focus=true#step:10:6651
- https://github.com/apache/airflow/runs/6319805534?check_suite_focus=true#step:10:12629
- https://github.com/apache/airflow/runs/6319710486?check_suite_focus=true#step:10:6759
The problem was that container for cassandra did not cleanly start:
ERROR: for airflow Container "3bd115315ba7" is unhealthy.
Encountered errors while bringing up the project.
3bd115315ba7 cassandra:3.0 "docker-entrypoint.s…" 5 minutes ago Up 5 minutes (unhealthy) 7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp airflow-integration-postgres_cassandra_1
The logs of cassandra container do not show anything suspected, cassandra seems to start ok, but the health-checks for the :
INFO 08:45:22 Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
INFO 08:45:22 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO 08:45:23 Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO 08:45:23 Startup complete
INFO 08:45:24 Created default superuser role ‘cassandra’
We mitigated it by #23522 and pinned cassandra to 3.0.25 version but more investigation/reachout is needed.
What you think should happen instead
Cassandra should start properly.
How to reproduce
Revert #23522 and make. PR. The builds will start to fail with "cassandra unhealthy"
Operating System
Github Actions
Versions of Apache Airflow Providers
not relevant
Deployment
Other
Deployment details
CI
Anything else
Always.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct