btw:

$ getconf OPEN_MAX
1073741816

2^30 - 8 - I would really like to know where that comes from..

@polarathene knows more about this and should decide whether this should be merged.

I don't know what's causing it but CI seems to be failing a lot recently, I think after @polarathene merged a PR that refactors tests (showing us that there are race conditions in tests?). Can we do something about that?

I've some questions regarding the problem and your provided workaround.

Fail2ban closes all fds on startup

Just for my understanding, why is fail2ban trying to close all(?) files. What files? Why so many? I think I miss something here 😉

Can your workaround (hard-coding 1024) introduce issues for existing setups?

We resolved this for our tests in #2730 we do not need to merge this.

It's been a while, but I think you are setting ulimit for the entire container like this and IIRC, docker doesn't like ulimit being used, if you run another ulimit later in the container/script it would fail I think.

1024 is excessively low as well and considered a legacy hard limit. You should find any modern linux system has 512k by default. The soft limit is still 1024 though (for compatibility reasons), and a process can ask to raise it so long as the hard limit isn't exceeded across all processes as far as I understand it.

What Docker uses can be much larger though, and varies by linux host, often by a systemd service config (more info here).

In docker-compose.yml, add the following ulimits:

ulimits:
  nofile:
    soft: $(ulimit -Sn)
    hard: $(ulimit -Hn)

The value probably won't work in compose like that, but if you run both ulimit commands on the shell, you should get the appropriate values for your Docker host to be set to.

A hard limit of a million appears to be minimal delay AFAIK, whereas 1 billion or more like I had caused a 10 minute start-up delay.

btw:

$ getconf OPEN_MAX
1073741816

2^30 - 8 - I would really like to know where that comes from..

You'll find that Docker is probably configured by systemd by your distro here.

It varies what value is set, and it's presumably because the Docker daemon needs to support a large amount across containers?

I had the same amount on my systems.

Thankfully old me documented this issue heavily, plenty of details can be found on the related f2b issue I originally raised.

Closing reason

Remember this does not only affect f2b, but also postsrsd.

I'm not aware of any other software we presently ship that creates a daemon with the same issue, but it could happen with future updates.

The correct approach is to set ulimits on the container itself.

cu around :-)

p.s.

The correct approach is to set ulimits on the container itself

i don't agree but future versions of fail2ban close only the fds that are actually open, so i'm happy anyway.