Problems with Docker version 28.0.1 on Jetson Orin NX

[user117398]

First i am aware of this thread: Error with “Nvidia Container Runtime with Docker Integration” on AGX Orin with JP6.2

However we ran into a problem that has not been solved by upgrading to Docker version 28.0.1.

We have a reverse proxy container that stops instantly with the following error message:

Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint reverse_proxy-1 (...): Unable to enable DIRECT ACCESS FILTERING - DROP rule:  (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.18.0.4 --dport 80 ! -i br-51c7e31b90d4 -j DROP: iptables v1.8.7 (legacy): can't initialize iptables table `raw': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

Same container (native amd64 and emulated arm64) works on our amd64 pcs with the same docker version.

Doesnt work on our Jetsons with JP 6.1 (custom RT kernel, The ip_set etc. kernel modules were already enabled since we are using k8s).
Just happens with this one container other containers seem fine.

Downgrading Docker verion 27 solves the problem, however thats just a temporary solution ofc.

Has anbody encountered the same problem and has a solution?

[ToeiRei]

It all started with docker having a problem with the nvidia runtime and vice versa and now it seems like we need a new kernel.

[AastaLLL]

Hi,

Do you have a custom kernel?

After the docker 28.0.1 release, it should be able to run on the JetPack 6.1/6.2 BSP without changes.
If it doesn’t work in your environment, could you share the following info with us?

$ sudo docker info
$ journalctl -xu docker.service

Thanks.

[sean.pan]

Hi,
I have been used the latest Jetson Linux BSP package 36.4.3, and also face the same problem,
my docker version is as follow:
$ docker --version
Docker version 28.0.1, build 068a01e

my docker info message is as follow:
$ sudo docker info
Client: Docker Engine - Community
Version: 28.0.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.21.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.33.1
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 2
Server Version: 28.0.1
Storage Driver: overlay2
Backing Filesystem: btrfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: journald
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 nvidia
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.148-tegra
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 7.443GiB
Name: tegra-ubuntu
ID: f98d7454-51e3-4b46-9d72-38580bf5c20e
Docker Root Dir: /opt/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 172.100.0.0/16, Size: 24

And the journalctl message from docker is as follow:
$ journalctl -xu docker.service
Feb 12 06:05:25 tegra-ubuntu dockerd[905]: time=“2025-02-12T06:05:25.854595681Z” level=error msg=“Handler for POST /v1.48/containers/e2ee74fea501e8620ddde770b9112cb5d89f17525fff53075f945bf722b1697f/start returned error: failed to set up container networking: driver failed programming external connectivity on endpoint 6c1e26a7-22c5-452d-9bd5-ca8ed7ac90cb-web1-1 (bb880b0c080de050631b7bf953b5dd49b0cba20c4abcd22f0ccc455efef4d163): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.100.3.3 --dport 80 ! -i br-3ee37c5f8364 -j DROP: iptables v1.8.7 (legacy): can’t initialize iptables table `raw’: Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n (exit status 3))” spanID=5a7d7e7132330817 traceID=53ea434e468c36e7988c272b10914852

[bashaku]

Docker 28.0.1 depends on the kernel module iptable_raw, but it doesn’t seem to be included in Jetson Linux.

$ cat /etc/nv_tegra_release
# R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan  8 01:49:37 UTC 2025
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia

$ docker --version
Docker version 28.0.1, build 068a01e

$ docker run -dit -p 80:80 --rm --name alpine alpine:latest
44de6abaa53736b9ffa4eb5171b631475772b015aa9dc8f92005065c016685d8
docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint alpine (d373025c7fbf1f880fe55171c6673b2b8f036ee84160128712ffde9980c02266): Unable to enable DIRECT ACCESS FILTERING - DROP rule:  (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.17.0.2 --dport 80 ! -i docker0 -j DROP: iptables v1.8.7 (legacy): can't initialize iptables table `raw': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3))

$ find /lib/modules/$(uname -r) -name '*iptable_raw*'

$ lsmod | grep -i iptable_raw

$ zgrep IP_NF_RAW /proc/config.gz
# CONFIG_IP_NF_RAW is not set

[bashaku]

To resolve this, a kernel rebuild was required.

I added the following line to the end of defconfig

CONFIG_IP_NF_RAW=m

re-build kernel and destination for iptable_raw.ko is

/lib/modules/5.15.148-tegra/kernel/net/ipv4/netfilter/

$ docker --version
Docker version 28.0.1, build 068a01e

$ docker run -dit -p 80:80 --rm --name alpine alpine:latest
c2210a7f8d3be7e5326b2b3773dc0d92834f7c4db9c779ee0ee9910edf8df462

$ lsmod | grep -i iptable_raw
iptable_raw            16384  1
ip_tables              32768  3 iptable_filter,iptable_raw,iptable_nat
x_tables               45056  12 ip6table_filter,xt_conntrack,iptable_filter,ip6table_nat,xt_tcpudp,xt_addrtype,xt_nat,ip6_tables,iptable_raw,ip_tables,iptable_nat,xt_MASQUERADE

[AastaLLL]

Hi,

Is the docker 28.0.1 working normally after the configure is enabled?
Thanks.

[bashaku]

Hi, @AastaLLL

I enabled only CONFIG_IP_NF_RAW.
and deployed new kernal image file and iptable_raw.ko to my Jetson.

then, the docker 28.0.1 working normally.

[user117398]

Hi, @bashaku

Your solution works for me as well. Thanks!

[john432]

I have this problem as well. I turned iptables off in /etc/docker/daemon.json and have been running my containers in host with JP62 which isn’t my ideal scenario. I really don’t want to manage custom kernels. @AastaLLL will iptable_raw be added to the Jetpack release?

[arampulla]

In addition to knowing when this will be fixed in the current JetPack releases, are there detailed instructions anywhere on building the kernel? Normally this is done via the SDK Manager and I’ve never had to build it by hand, so instructions would be helpful.

[AastaLLL]

Hi, all

We are discussing to enable CONFIG_IP_NF_RAW internally.
Will share more info with you later.

Please find below the build from the source tutorial.

Kernel Customization — NVIDIA Jetson Linux Developer Guide

You can also check the below comment for the detailed steps.
(please update the kernel config accordingly)

Error with “Nvidia Container Runtime with Docker Integration” on AGX Orin with JP6.2

Hi, all Update: The new 28.0.1 docker release has fixed this issue. SDKmanager can work normally now. $ sudo apt-get install docker-ce docker-ce-cli Thanks for your patience. According to this issue: $ sudo docker run hello-world docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? $ journalctl -xu docker.service ... Feb 24 02:48:46 tegra-ubuntu dockerd[76713]: failed to start daemon: Error initializing network controller: error obtainin…

Thanks.

[petlock]

Those of us who don’t rebuild linux kernels in their sleep might need a refresher on some of the steps. If I’m on an Orin Nano AGX what’s the path for the toolchain referenced in step 3???

I was following along until this step

export CROSS_COMPILE=<toolchain-path>/bin/aarch64-buildroot-linux-gnu-

This user crashes trying to resolve whatever toolchain-path actually is…

I have no clue what the toolchain path would be. I’m on my jetson orin nano agx if that helps.

Is it this? Jetson Linux Toolchain — Jetson Linux Developer Guide documentation

and then I’d use this link ? https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v3.0/toolchain/aarch64--glibc--stable-2022.08-1.tar.bz2

[alex585]

@AastaLLL

e are discussing to enable CONFIG_IP_NF_RAW internally.
Will share more info with you later.

That would be VERY nice…
Im currently getting stuck bilding new Kernel and now blocked due to the docker iptables issue…

[duane.pike]

I agree, having a fleet of these and not being able to do the upgrade without issues is very concerning. I will do the kernal update on our dev machine. However will have to hold back on all the others.

[Kangalow]

@petlock @alex585 I’ve put up a precompiled version of the IP_NV_RAW module on Github. Could you test it and see if it works for you? It’s for JetPack 6.2: jetson-orin-kernel-builder/prebuilt/jetpack-6.2 at main · jetsonhacks/jetson-orin-kernel-builder · GitHub
You should know that the Docker folks have a pull request in to remove the requirement for the module that will probably happen pretty soon.

[petlock]

I’ll try it tonight.

And while I have Nvidia’s attention I wish I had known I was forever locking my machine out of frigate by upgrading to the latest jetpack when I bought it. Hell of a realization after dropping two grand.

[alex585]

Hmm, i yesterday used your Scripts (jetson-orin-kernel-builder/scripts at main · jetsonhacks/jetson-orin-kernel-builder · GitHub) to compile a new Kernel and kernel Moduls :-D
Indeed, found them yesterday. Then i copied the new created Kernel with enabled IP_NV_RAW over /boot/Image, et voila my Docker is back in the Game.

Hopefully this setting will be standard in NV delivered Kernels…

[Kangalow]

I’m glad you found the scripts useful. My understanding from the Docker issues in their repository is someone was working on optimization and pulled in the IP tables to help. This broke a good number of devices, including Jetsons and many Windows machines. They tried to pull the IP tables back out in 28.01, but missed one. They release 28.0.2 today, which they say fixes the IP table problem.
The overall plan includes a different solution entirely for the original optimization issue which does not include the IP table modules. Coming to a future Docker release near you. Docker was well intentioned, but just happened to break a lot of things in the process.
So even if NVIDIA includes IP_NV_RAW in another release, that ship has already sailed.

[alex.coulcher]

We simply rolled back to 27.3 to resolve this issue.

apt-get install docker-ce-cli=5:27.3.0-1~ubuntu.22.04~jammy
apt-get install docker-ce=5:27.3.0-1~ubuntu.22.04~jammy