fix(modules.kafka): Use broker container IP instead of host IP for advertised broker listener#1989
Merged
mdelapenya merged 10 commits intotestcontainers:mainfrom May 8, 2024
Merged
Conversation
✅ Deploy Preview for testcontainers-go ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
mdelapenya
previously approved these changes
Dec 11, 2023
Member
mdelapenya
left a comment
mdelapenya
left a comment
There was a problem hiding this comment.
LGTM! Although I'd like to have @eddumelendez 's eyes on this one too before merging.
Thanks for your time in any case, will merge it once the review and the CI pass.
port 9092 is not exposed to the host, so we can not connect to it via host ip, but need to use the broker container ip
Contributor
Author
|
@mdelapenya @eddumelendez |
Member
|
@eddumelendez, as our kafka expert, what are your thoughts on this? |
mdelapenya
reviewed
Apr 5, 2024
* main: (44 commits) feat: expose JSON representation of a container with Inspect (testcontainers#2534) chore(deps): bump test-summary action to v2.3 (testcontainers#2535) chore(deps): bump jinja2 from 3.1.3 to 3.1.4 (testcontainers#2533) Update devcontainer image (testcontainers#2531) chore(influxdb): include more characters in wait for log regex (testcontainers#2532) fix(compose): avoid race conditions when caching services (testcontainers#2528) chore(deps): bump golangci/golangci-lint-action from 3.7.0 to 5.1.0 (testcontainers#2525) chore(deps): bump mkdocs-material from 8.2.7 to 9.1.21 (testcontainers#2524) chore(compose): return error in options (testcontainers#2520) chore(deps): bump github.com/compose-spec/compose-go/v2 from v2.0.0-rc8 to v2.1.0 (testcontainers#2519) chore(deps): bump github.com/containerd/containerd from 1.7.12 to 1.7.15 (testcontainers#2517) break: return error from Customize request option (testcontainers#2267) fix: wrong copy paste (testcontainers#2515) docs: add documentation for Exec method (testcontainers#2451) docs: document the SSHd tunnel (testcontainers#2514) fix: enhance host configuration port binding (testcontainers#2512) feat: forward host ports to a container using an SSH tunnel (testcontainers#2471) Update follow_logs.md with adding missing package (testcontainers#2513) fix: don't retry on permanent APIClient errors (testcontainers#2506) feat: support overriding the default recreate options for compose (testcontainers#2511) ...
Member
Contributor
Author
|
@mdelapenya looks good, thanks :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR changes the advertised broker ip from being the host ip to container ip. Port 9092 is not exposed to the host, so we can not connect to it via host IP, but need to use the broker container IP.
The current setup works with a normal desktop setup, because there the returned host IP is "localhost".
Alternatively, it should also be sufficient to hardcode the advertised broker as "localhost", because there is only 1 broker that needs to be able to connect to itself.
Why is it important?
This makes it possible to use the Kafka module inside another Docker container or other setups where Host() does not return "localhost". This enables the use of this module in CI/CD pipelines that run in Docker.
Currently it leads to errors in the TransactionCoordinator:
WARN [TransactionCoordinator id=1] Connection to node 1 (/172.17.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)Related issues
This PR tackles the same problem, but using the container name instead of ip breaks the normal Desktop setup, as the system doesn't know the container name as a hostname.
How to test this PR
After starting a Kafka container using the Kafka module, the KAFKA_ADVERTISED_LISTENERS should contain the kafka container ip.
E.g.
the /usr/sbin/testcontainers_start.sh script should contain the following value:
export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:51702,BROKER://172.17.0.6:9092