Podman support

It would be really nice if TestContainers supported podman.
This is a real game changer to no longer depend on Docker daemons to run.

Thanks for asking! Since this is not the first time somebody asks for Podman, I would like to post a (non complete, most probably) list of requirements we need to do the Testcontainers' magic:

Cross-platform support. It should work exactly the same on Linux, Mac and Windows, without having us do the VM management
Docker networks, ability to start/connect/disconnect containers with/to Networks
Image management - pulling (including the various authorisation mechanisms), listing, building
Long running processes that survive the test run
Access to /var/run/docker.sock from inside the container, ability to do regular Docker commands with it
Log access/streaming
Stdin/Stdout/Stderr streaming
There are many Docker images that require root user, these should work too
Volumes management

I am not a Podman expert, and if somebody from the community can help clarifying these, I would be happy to learn, and, as long as every point is clarified (the list is not 100% complete and we may need to add more in future) we can consider adding the support for Podman.

and BTW, "daemonless" here makes it harder for Testcontainers :)

@rhatdan

From my Podman experience:

Access to /var/run/docker.sock : I don't think so, since there is no daemon;
Cross Platform support: Podman only works in Linux. I haven't checked the internals of Testcontainers yet, but Podman offers an alias to replace the docker command, so Testcontainers could use Podman in Linux with the same CLI syntax as used in Docker (if the CLI is used);
Image Building: is performed by Buildah

I wish @rhatdan could chime in the discussion to clarify or correct my statements 😃

if the CLI is used

We're not using the CLI, but the REST API.

It looks to me that we should rather focus on waiting for the rootless Docker (an ongoing effort upstream), Podman as it is (plus Buildah, plus whatever else is needed) cannot provide the same functionality we need to do what we're doing in Testcontainers.

Dropping this here because I found it useful and relevant to the discussion :)

https://medium.com/nttlabs/cgroup-v2-596d035be4d7

We're not using the CLI, but the REST API.

Podman recently started offering a REST API: https://podman.io/blogs/2020/01/17/podman-new-api.html

@ppalaga

@ppalaga nice!

Do you have a compatibility table or something, to see which Docker's API endpoints are supported or not? Thanks!

@bsideup

@bsideup
I would help putting that table together. Is there a list of Docker API endpoints (and options) testcontainers uses or a test-suite we can run against podman's Docker compability API.

@crunchtime-ali

@crunchtime-ali I believe running Testcontainers' test suite is the best way to check whether all APIs are supported or not. Our tests should cover all types of Docker features we are using.

You can also start a TCP proxy between Testcontainers and Podman to collect the list of endpoints that are used.

Some more information that might be helpful:

https://github.com/containers/libpod/blob/master/docs/source/markdown/podman-system-service.1.md

https://github.com/containers/libpod/blob/master/API.md

The Podman API supposedly has a docker API compatible endpoint.

And it can also expose a sock socket file as well, which should meet the requirement of having a /var/run/docker.sock (or equivalent) inside the container.

Correct we are under heavy development and are looking at testcontainers to prove that we handle the Docker API correctly. We have an extended version to support advanced features of Podman as well.

I'm also looking at doing the reverse: Running testcontainers tests in podman CI. Solving the challenges here, will help me get to there. (subscribed)

how does one run the test suite? hints around how to specify the sock to connect to is a bonus. ill do the work.

@bsantanna

The workaround proposed by @bsantanna works for some containers but not others. Even ones that work I get this a lot:

ERRO[1205] accept tcp [::]:39779: use of closed network connection

I understand this issue is closed, although just for the record, here is a workaround I found that worked in my case:

My integration test was letting TestContainer launch docker-compose as follows:

      return new DockerComposeContainer(
          "dynamo-metrics-service-integration-tests",
          new File("./path/to/my/docker-compose.yml"),
          )
          .waitingFor("druid-zookeeper_1", Wait.forLogMessage(".*Started AdminServer.*", 1))
          .waitingFor("druid-postgres_1", Wait.forLogMessage(".*database system is ready to accept connections.*", 1))
          .waitingFor("druid-coordinator_1", Wait.forLogMessage(".*reportin' for duty.*", 1))
          .waitingFor("druid-broker_1", Wait.forLogMessage(".* Successfully started lifecycle.*", 1))
          .waitingFor("druid-router_1", Wait.forLogMessage(".* Successfully started lifecycle.*", 1))
          .waitingFor("druid-historical_1", Wait.forLogMessage(".* Successfully started lifecycle.*", 1))
          .waitingFor("druid-middlemanager_1", Wait.forLogMessage(".* Successfully started lifecycle.*", 1))

I'm using Podman 3.4.2 and TestContainer 1.6.3 on Linux with the following env variables:

DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
TESTCONTAINERS_CHECKS_DISABLE=true
TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=unix:///run/user/1000/podman/podman.sock
TESTCONTAINERS_RYUK_DISABLED=true

The DockerComposeContainer statement above failed with the following error message:

Status 500: {"cause":"incorrect volume format, should be [host-dir:]ctr-dir[:option]","message":"fill out specgen: /path/to/my/code/unix:/run/user/1000/podman/podman.socket:/docker.sock:rw: incorrect volume format, should be [host-dir:]ctr-dir[:option]","response":500}

I solved the issue by telling DockerComposeContainer to work with local compose :

          .withLocalCompose(true);

Here's my recipe. I was able to avoid the need to disable ryuk.

Edit ~/.testcontainers.properties and add the following line

ryuk.container.privileged=true

Then run the following

brew install podman
podman machine init -v $HOME:$HOME

If you're podman 4.1 or higher, you don't need the the -v $HOME:$HOME volume mount.

If you're on mac,

sudo /opt/homebrew/Cellar/podman/4.0.3/bin/podman-mac-helper install

Then (for all oses)

podman machine set --rootful
podman machine start

The rootful part makes ryuk behave ok, based on my testing (see podman-container-tools/podman#14238). If ryuk is still causing problems or you don't want to run rootful then export TESTCONTAINERS_RYUK_DISABLED=true also does the trick, but containers can sometimes hang around and cause problems.

Finally, if you're on Mac M1, update podman to behave better with images for a different architecture (see https://edofic.com/posts/2021-09-12-podman-m1-amd64/ )

podman machine ssh
sudo -i
rpm-ostree install qemu-user-static
systemctl reboot

Once the virtual machine restarts, you should be good to run testcontainers.

@holly-cummins

This is awesome @holly-cummins. One question - With podman 4.1.1 on apple silicon mac, did you have to downgrade qemu back to version 6.2.0?

@holly-cummins

This is awesome @holly-cummins. One question - With podman 4.1.1 on apple silicon mac, did you have to downgrade qemu back to version 6.2.0?

Ummm. I don't remember doing that, but I am indeed running 6.2.0:

holly@hcummins-mac ~ % podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 36.20220421.dev.0
[...]

[root@localhost ~]# rpm -q qemu-user-static
qemu-user-static-6.2.0-9.fc36.aarch64

I think I must have been lucky, and done my installation before v7 was released. Is 7 not so good on your machine?

When I did a brew update and it updated qemu to v7 it broke podman on my M1 mac. I haven't tried it again since, so maybe its fixed, but the issue below makes it seem like it isn't fixed.

Homebrew/homebrew-core#102242

I had to manually back qemu down to 6.2.0_1 using this procedure:

curl -L -H "Authorization: Bearer QQ==" -o qemu-6.2.0_1.monterey.bottle.tar.gz https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:fcc3b1a8139f70dae57f5449f3856f9b3b67448ee0623e64da1e47dc255b46f6

brew uninstall --ignore-dependencies qemu

brew install -f qemu-6.2.0_1.monterey.bottle.tar.gz

I found that procedure out on some GitHub issue somewhere - I just can't seem to remember where exactly...

@holly-cummins

@holly-cummins There's also this - podman-container-tools/podman#14303

@edeandrea

Oh, sorry, @edeandrea, I've got muddled between the qemu-user-static in the podman machine, and the qemu on my mac machine. My mac has podman 4.1.1 and qemu 7.0.0, but it seems happy. So either the issue is fixed, or some other environmental factor on my machine masks the issue.

@holly-cummins

Good to know @holly-cummins . Maybe I'll give it a try.

@holly-cummins

@holly-cummins thanks for the ryuk.container.privileged=true tip. testcontainers seems to work fine with that in the native rootless user podman on Fedora (not running in a "podman machine" VM), without needing the TESTCONTAINERS_RYUK_DISABLED env var.

Just wanted to add that in case the podman machine set --rootful might not be needed. I'm not really familiar with the "podman machine" stuff though.

Is there known issue with pulling image by testcontainers ? I keep getting below errors when image is not in local repo and testcontainers tries to pull it from remote:

2022-10-10 17:02:59,110 INFO  --- [                     docker-java-stream--] docker.io/postgres:14]  :  Starting to pull image 
2022-10-10 17:02:59,197 ERROR --- [                     docker-java-stream--] c.g.d.api.async.ResultCallbackTemplate  :  Error during callback 
java.lang.NullPointerException: Cannot invoke "String.matches(String)" because the return value of "com.github.dockerjava.api.model.PullResponseItem.getStatus()" is null
	at com.github.dockerjava.api.command.PullImageResultCallback.checkForDockerSwarmResponse(PullImageResultCallback.java:48)
	at com.github.dockerjava.api.command.PullImageResultCallback.onNext(PullImageResultCallback.java:35)
	at org.testcontainers.images.LoggedPullImageResultCallback.onNext(LoggedPullImageResultCallback.java:48)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:73)
	at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.onNext(TimeLimitedLoggedPullImageResultCallback.java:24)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onNext(AbstrAsyncDockerCmdExec.java:41)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:315)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$JsonSink.accept(DefaultInvocationBuilder.java:298)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:275)
	at java.base/java.lang.Thread.run(Thread.java:833)

@mario45211

@mario45211 I had the same issue initially. When I tried to pull the image manually, I realized that Podman did not have access to that repo yet where the image "repohost:8080/path/to/image:latest" was located. So I did podman login repohost:8080 in the console first and afterwards I was able to pull the image using testcontainers.

It seems there is a bug in docker-java-api that triggers the NPE in such a case. At the code where the NPE occurrs in the class PullImageResultCallback, it tries to compare the result of item.getStatus() without checking it against null first, but the method is clearly marked with @CheckForNull to indicate that the return value could be null. Why the status is null when the login failed, I don't know. Maybe such a problem should be caught at another place.
I'd have to check if there is an issue in their Guthub repo already relating to that.

To follow up, I did not find a tracked issue regarding the NPE. As the problem lies in the docker-java-api code and not with testcontainerst itself, I created an issue in their issue tracker: docker-java/docker-java#1998

For those landing here hunting for the solution for NPE, here is one workaround until the docker-java bug is fixed:
docker-java/docker-java#1998 (comment)

As far as I can tell, nowadays Podman works fine with Testcontainers, as long as you enable the Podman socket (podman.socket systemd unit) and set DOCKER_HOST appropriately.

The reason why some need to run Ryuk as a privileged container (or disable it altogether), as far as I can tell, is not because of Podman, but rather because of SELinux. If SELinux is enabled, Ryuk will be unable to connect to the Docker/Podman UNIX socket which is bind-mounted inside the container. See this and this discussion for more detail.

Running Ryuk as a privileged container works around this problem because privileged containers are not isolated through SELinux. Similarly, non-privileged Ryuk also works fine with Podman on systems without SELinux. And also, the same problem with Ryuk reproduces if you run the Docker daemon with SELinux enabled (#7177).

I suspect the reason this issue comes up in discussions with Podman because (1) Podman has SELinux support enabled by default and most adopters are likely on Red Hat distros which ship with SELinux enabled and (2) Docker has SELinux support disabled by default, even on distributions with SELinux enabled.

Navigation Menu

Repository navigation

Podman support #2088

78 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Podman support #2088

Description

Activity

bsideup commented on Nov 20, 2019

bsideup commented on Nov 20, 2019

gastaldi commented on Nov 20, 2019

bsideup commented on Nov 20, 2019

Kidlike commented on Dec 18, 2019

ppalaga commented on Jan 28, 2020

bsideup commented on Jan 28, 2020

crunchtime-ali commented on Jan 28, 2020

bsideup commented on Jan 28, 2020

rocketraman commented on Mar 3, 2020

rocketraman commented on Mar 3, 2020

rhatdan commented on Mar 3, 2020

cevich commented on Mar 3, 2020

baude commented on Mar 3, 2020

78 remaining items

edeandrea commented on Sep 7, 2021

sv3ndk commented on Apr 6, 2022

holly-cummins commented on Jun 29, 2022

edeandrea commented on Jun 29, 2022

holly-cummins commented on Jun 29, 2022

edeandrea commented on Jun 29, 2022

edeandrea commented on Jun 29, 2022

holly-cummins commented on Jun 29, 2022

edeandrea commented on Jun 29, 2022

grdryn commented on Jul 5, 2022

mario45211 commented on Oct 10, 2022

TheHaf commented on Nov 10, 2022

TheHaf commented on Nov 15, 2022

AshwinPrabhuB commented on Jan 6, 2023

joanbm commented on Jun 28, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions