fix GetOutboundIp auto-detection on ipv6-only systems #1356

Merged

mfenniak merged 1 commit from cristicbz/forgejo-runner:try-ipv6-for-outbound into main 2026-02-07 18:19:07 +00:00

Conversation 10 Commits 1 Files changed 1 +11 -4

cristicbz commented 2026-02-07 15:29:49 +00:00

Contributor

Copy link

On an IPv6-only node, the daemon would fail to enable the cache, with

Could not start the cache server, cache will be disabled: unable to determine outbound IP address

and, similarly, exec would just crash on start-up with Error: unable to determine outbound IP address.

The issue stems from GetOutboundIp assuming 8.8.8.8 being reachable is equivalent to the node having internet access (which is not true for IPv6 nodes of course). I changed this (in the first commit) to try the IPv6 of Google's public DNS as well, and took the opportunity to fallback to cloudflare's as well.

A related issue (handled in the second commit) is that artifactcache.StartHandler was always passed "" (forcing auto-detection) even if cache.host was set in the config. I don't claim to fully understand what the different components are, but forwarding cache.host to that seems to work?

Scaleway offers dirt cheap (<1 EUR/month) IPv6-only nodes. I wanted to use some for my runners.

• other
  • PR: fix GetOutboundIp auto-detection on ipv6-only systems

On an IPv6-only node, the `daemon` would fail to enable the cache, with ``` Could not start the cache server, cache will be disabled: unable to determine outbound IP address ``` and, similarly, `exec` would just crash on start-up with `Error: unable to determine outbound IP address`. The issue stems from `GetOutboundIp` assuming 8.8.8.8 being reachable is equivalent to the node having internet access (which is not true for IPv6 nodes of course). I changed this (in the first commit) to try the IPv6 of Google's public DNS as well, and took the opportunity to fallback to cloudflare's as well. ~A related issue (handled in the second commit) is that `artifactcache.StartHandler` was always passed `""` (forcing auto-detection) even if `cache.host` was set in the config. I don't claim to fully understand what the different components are, but forwarding `cache.host` to that seems to work?~ Scaleway offers dirt cheap (<1 EUR/month) IPv6-only nodes. I wanted to use some for my runners. <!--start release-notes-assistant--> <!--URL:https://code.forgejo.org/forgejo/runner--> - other - [PR](https://code.forgejo.org/forgejo/runner/pulls/1356): <!--number 1356 --><!--line 0 --><!--description Zml4IEdldE91dGJvdW5kSXAgYXV0by1kZXRlY3Rpb24gb24gaXB2Ni1vbmx5IHN5c3RlbXM=-->fix GetOutboundIp auto-detection on ipv6-only systems<!--description--> <!--end release-notes-assistant-->

cristicbz requested review from mfenniak 2026-02-07 15:51:41 +00:00

cristicbz changed title from fix GetOutbooundIp auto-detection on ipv6-only systems to fix GetOutboundIp auto-detection on ipv6-only systems 2026-02-07 15:53:18 +00:00

mfenniak commented 2026-02-07 16:54:06 +00:00

Owner

Copy link

I think the IPv6-capable detection seems like a good change. The overall approach to detection is a bit too clever for my liking, but adding additional IPv6 addresses seems like an OK workaround for the gap you've identified. I don't think we can make a practical automated test for this, so this part of the change is good as-is.

The cache config change would be a regression of #1088. There are two different servers being run by the runner; a proxy that the actions connect to, and a cache server that the proxy connects to. You're changing the cache server, which isn't what cfg.Cache.Host is intended & documented to control.

I think the IPv6-capable detection seems like a good change. The overall approach to detection is a bit too clever for my liking, but adding additional IPv6 addresses seems like an OK workaround for the gap you've identified. I don't think we can make a practical automated test for this, so this part of the change is good as-is. The cache config change would be a regression of https://code.forgejo.org/forgejo/runner/pulls/1088. There are two different servers being run by the runner; a proxy that the actions connect to, and a cache server that the proxy connects to. You're changing the cache server, which isn't what `cfg.Cache.Host` is intended & documented to control.

cristicbz force-pushed try-ipv6-for-outbound from 23840a560b to f2200fee29 2026-02-07 17:20:08 +00:00 Compare

cristicbz commented 2026-02-07 17:21:50 +00:00

Author

Contributor

Copy link

@mfenniak thanks for the explanation, I reverted the second commit.

There's something about the overall architecture i don't get---why any of these internal services need an external IP adress vs being able to just use loopback; it seems to extend the attack surface unnecessarily to listen on public interfaces.

@mfenniak thanks for the explanation, I reverted the second commit. There's something about the overall architecture i don't get---why any of these internal services need an external IP adress vs being able to just use loopback; it seems to extend the attack surface unnecessarily to listen on public interfaces.

mfenniak commented 2026-02-07 17:38:42 +00:00

Owner

Copy link

@cristicbz wrote in #1356 (comment):

There's something about the overall architecture i don't get---why any of these internal services need an external IP adress vs being able to just use loopback; it seems to extend the attack surface unnecessarily to listen on public interfaces.

It's complicated. Here's my understanding, which may be incomplete.

The first requirement is to create a proxy network port which can be accessed by a job container. A loopback address wouldn't work for this because localhost would reference the container, not the runner. The runner itself may be within a container, and therefore we need the container's IP address and not an IP from the host.

The "outbound" IP is not necessarily a public IP. It's the IP address of the interface which the default route is applied to; in an IPv4 network that's usually a non-routable RFC1918 address. It isn't really important that it is used for internet access... it serves more of a role of "this is a reasonable default IP address which I can access both from the runner, and from the

The second requirement is to create a cache server network port which can be accessed by the runner (after the proxy is hit). This has different requirements depending on whether the cache server is internal, or external. If it's external, it needs to be accessible from the cache clients (the proxies). If it's internal, it could probably be a loopback address.

@cristicbz wrote in https://code.forgejo.org/forgejo/runner/pulls/1356#issuecomment-77199: > There's something about the overall architecture i don't get---why any of these internal services need an external IP adress vs being able to just use loopback; it seems to extend the attack surface unnecessarily to listen on public interfaces. It's complicated. Here's my understanding, which may be incomplete. The first requirement is to create a proxy network port which can be accessed by a job container. A loopback address wouldn't work for this because `localhost` would reference the container, not the runner. The runner itself may be within a container, and therefore we need the container's IP address and not an IP from the host. The "outbound" IP is not necessarily a public IP. It's the IP address of the interface which the default route is applied to; in an IPv4 network that's usually a non-routable RFC1918 address. It isn't really important that it is used for internet access... it serves more of a role of "this is a reasonable default IP address which I can access *both* from the runner, and from the The second requirement is to create a cache server network port which can be accessed by the runner (after the proxy is hit). This has different requirements depending on whether the cache server is internal, or external. If it's external, it needs to be accessible from the cache clients (the proxies). If it's internal, it could probably be a loopback address.

mfenniak commented 2026-02-07 17:40:47 +00:00

Owner

Copy link

This patch is failing CI as-is and needs a make fmt run on it.

This patch is failing CI as-is and needs a `make fmt` run on it.

cristicbz force-pushed try-ipv6-for-outbound from f2200fee29 to 4d573ee4de 2026-02-07 17:41:36 +00:00 Compare

cristicbz commented 2026-02-07 17:42:09 +00:00

Author

Contributor

Copy link

Done. Sorry, I ran go fmt, which didn't complain, didn't realize this had stricter linting requirements

Done. Sorry, I ran `go fmt`, which didn't complain, didn't realize this had stricter linting requirements

mfenniak merged commit e07587daa6 into main 2026-02-07 18:19:07 +00:00

mfenniak referenced this pull request from a commit 2026-02-07 18:19:08 +00:00

fix GetOutboundIp auto-detection on ipv6-only systems (#1356)

mfenniak commented 2026-02-07 18:21:03 +00:00

Owner

Copy link

Because we're not actually doing anything with the UDP network connection -- it's UDP which is stateless, and nothing is sent, and it's not even to a DNS port 🤣 -- in retrospect adding in Cloudflare's addresses is probably not meaningful. It's really just a hack to get the outbound IP which does nothing but resolve the route table. But I think it has no harm either, so all good. 👍

Because we're not actually doing anything with the UDP network connection -- it's UDP which is stateless, and nothing is sent, and it's not even to a DNS port 🤣 -- in retrospect adding in Cloudflare's addresses is probably not meaningful. It's really just a hack to get the outbound IP which does nothing but resolve the route table. But I think it has no harm either, so all good. 👍

cristicbz commented 2026-02-07 19:38:00 +00:00

Author

Contributor

Copy link

@mfenniak

The "outbound" IP is not necessarily a public IP. It's the IP address of the interface which the default route is applied to; in an IPv4 network that's usually a non-routable RFC1918 address. It isn't really important that it is used for internet access... it serves more of a role of "this is a reasonable default IP address which I can access both from the runner, and from the

I'm not set up to trivially check this for an IPv4 node, but for my little IPv6 only node at least, it does seem to resolve to the public IP address:

root@***:~# ss -lntup | grep forgejo
tcp   LISTEN 0      4096               *:42689            *:*    users:(("forgejo-runner",pid=943,fd=8))
tcp   LISTEN 0      4096               *:33341            *:*    users:(("forgejo-runner",pid=943,fd=7))

And I can confirm that if I curl the VPS's public IP address from my machine, i get a 404 Not Found from the services running on there.

Don't know if you have an IPv4 node where you can check that assertion empirically, but this seems like a significant vulnerability if I'm right.

@mfenniak > The "outbound" IP is not necessarily a public IP. It's the IP address of the interface which the default route is applied to; in an IPv4 network that's usually a non-routable RFC1918 address. It isn't really important that it is used for internet access... it serves more of a role of "this is a reasonable default IP address which I can access both from the runner, and from the I'm not set up to trivially check this for an IPv4 node, but for my little IPv6 only node at least, it does seem to resolve to the public IP address: ``` root@***:~# ss -lntup | grep forgejo tcp LISTEN 0 4096 *:42689 *:* users:(("forgejo-runner",pid=943,fd=8)) tcp LISTEN 0 4096 *:33341 *:* users:(("forgejo-runner",pid=943,fd=7)) ``` And I can confirm that if I `curl` the VPS's public IP address from my machine, i get a 404 Not Found from the services running on there. Don't know if you have an IPv4 node where you can check that assertion empirically, but this seems like a significant vulnerability if I'm right.

cristicbz commented 2026-02-07 19:56:48 +00:00

Author

Contributor

Copy link

I can confirm that this seems to happen on a node with IPv4 as well, do you see something else if you try? I was expecting the runner to not listen on ANY ports on a public interface. I'm running:

1. Latest forgejo-runner built from main
2. podman 5.4.2 with rootless containers
3. No config.yml

Ok. This is clearly intended behaviour, I just don't understand how this is safe:

	listener, err := net.Listen("tcp", fmt.Sprintf(":%d", port)) // listen on all interfaces
	if err != nil {
		return nil, err
	}
	server := &http.Server{
		ReadHeaderTimeout: 2 * time.Second,
		Handler:           router,
	}

(in both cacheproxy & artifactcache)

I would expect these services to listen on the network's gateway, (i.e. docker network inspect bridge -f "{{ (index .IPAM.Config 0).Gateway }}"); of course this gets messier with ephemeral networks

I can confirm that this seems to happen on a node with IPv4 as well, do you see something else if you try? I was expecting the runner to not listen on ANY ports on a public interface. I'm running: 1. Latest `forgejo-runner` built from `main` 2. podman 5.4.2 with rootless containers 3. No `config.yml` Ok. This is clearly intended behaviour, I just don't understand how this is safe: ``` listener, err := net.Listen("tcp", fmt.Sprintf(":%d", port)) // listen on all interfaces if err != nil { return nil, err } server := &http.Server{ ReadHeaderTimeout: 2 * time.Second, Handler: router, } ``` (in both cacheproxy & artifactcache) I would expect these services to listen on the network's gateway, (i.e. `docker network inspect bridge -f "{{ (index .IPAM.Config 0).Gateway }}"`); of course this gets messier with ephemeral networks

mfenniak commented 2026-02-07 20:50:24 +00:00

Owner

Copy link

Ah, I see your concern. The cache handlers always bind themselves on all network interfaces. This doesn't relate to GetOutboundIP directly; that IP is just used as an address that can be used to access the network ports once they're open.

act/artifactcache/handler.go

Line 93 in 5276774

listener, err := net.Listen("tcp", fmt.Sprintf(":%d", port)) // listen on all interfaces

act/cacheproxy/handler.go

Line 98 in 5276774

listener, err := net.Listen("tcp", fmt.Sprintf(":%d", port)) // listen on all interfaces

Within the designed scope of the APIs that are exposed for caching, they are fairly safe for exposure -- there should be no way to access cached data without a credential that is embedded into the ACTIONS_CACHE_URL env variable. As a workaround if you want to maximize security against design flaws and vulnerabilities, you can consider a host-based firewall -- a default-DENY incoming connection on your public interface should address the issue, but you'll have to test that it doesn't affect traffic from your containers (or local jobs, if you're using a host-based executor model).

It would be difficult to design a better auto-detection solution that would meet the runner's needs, and so the only plausible direction to me to improve this would require administrators to manually configure the bind address. I think adding manual configuration options like that would be reasonable. I wish it could be automatic instead, but I can't imagine a plausible way.

This discussion should probably continue in a separate issue if you'd like to file one, in order to make it more searchable for future users.

Ah, I see your concern. The cache handlers always bind themselves on all network interfaces. This doesn't relate to `GetOutboundIP` directly; that IP is just used as an address that can be used to access the network ports once they're open. https://code.forgejo.org/forgejo/runner/src/commit/52767745cc696967439470176195b3b950107679/act/artifactcache/handler.go#L93 https://code.forgejo.org/forgejo/runner/src/commit/52767745cc696967439470176195b3b950107679/act/cacheproxy/handler.go#L98 Within the designed scope of the APIs that are exposed for caching, they are fairly safe for exposure -- there should be no way to access cached data without a credential that is embedded into the `ACTIONS_CACHE_URL` env variable. As a workaround if you want to maximize security against design flaws and vulnerabilities, you can consider a host-based firewall -- a default-DENY incoming connection on your public interface should address the issue, but you'll have to test that it doesn't affect traffic from your containers (or local jobs, if you're using a host-based executor model). It would be difficult to design a better auto-detection solution that would meet the runner's needs, and so the only plausible direction to me to improve this would require administrators to manually configure the bind address. I think adding manual configuration options like that would be reasonable. I wish it could be automatic instead, but I can't imagine a plausible way. This discussion should probably continue in a separate issue if you'd like to file one, in order to make it more searchable for future users.

aahlenst commented 2026-02-07 22:26:03 +00:00

Member

Copy link

I believe #229 requests the option to set a bind address.

I believe https://code.forgejo.org/forgejo/runner/issues/229 requests the option to set a bind address.

👍 2

viceice-bot referenced this pull request from actions/setup-forgejo 2026-02-08 03:35:39 +00:00

Update dependency forgejo/runner to v12.6.4 #876

mfenniak referenced this pull request from a commit 2026-02-08 04:15:22 +00:00

Update dependency forgejo/runner to v12.6.4 (#876)

viceice-bot referenced this pull request 2026-02-09 00:00:56 +00:00

Update data.forgejo.org/forgejo/runner Docker tag to v12.6.4 #1361

viceice-bot referenced this pull request 2026-02-09 00:00:59 +00:00

Update dependency forgejo/runner to v12.6.4 #1362

mfenniak referenced this pull request from a commit 2026-02-09 01:38:10 +00:00

Update dependency forgejo/runner to v12.6.4 (#1362)

mfenniak referenced this pull request from a commit 2026-02-09 01:38:32 +00:00

Update data.forgejo.org/forgejo/runner Docker tag to v12.6.4 (#1361)

forgejo-mirror referenced this pull request from a commit 2026-02-10 04:01:17 +00:00

Update module code.forgejo.org/forgejo/runner/v12 to v12.6.4 (forgejo) (#11222)

Sign in to join this conversation.

Reviewers

No reviewers

mfenniak

Labels

Clear labels   

FreeBSD

  

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Chore

It is no fun but it needs to be done, this is a chore.

  

Kind/DependencyUpdate

for pull requests that are created by renovate when updating a dependency

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

  

Windows

Windows ports is an independent project, see https://github.com/Crown0815/Forgejo-runner-windows-builder

  

linux-powerpc64le

powerpc64le

  

linux-riscv64

  

linux-s390x

  

run-end-to-end-tests

Run https://code.forgejo.org/forgejo/end-to-end with a runner compiled from this pull request

  

run-forgejo-tests

Run https://codeberg.org/forgejo/forgejo tests importing the runner from this pull request

No labels

FreeBSD

Kind/Breaking

Kind/Bug

Kind/Chore

Kind/DependencyUpdate

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Windows

linux-powerpc64le

linux-riscv64

linux-s390x

run-end-to-end-tests

run-forgejo-tests

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

forgejo/runner!1356

No description provided.