CRITICAL BUG // DefaultLimitNOFILESoft=1024  #5953

Activity

1. ladar

   added

   bug report

   needs triage

   on Jul 23, 2022
2. al-cheb commented on Jul 23, 2022

   

   al-cheb

   on Jul 23, 2022

   Contributor

   More actions

   Hey @ladar.
   You can change this limit:
   #4683 (comment)
   #3738 (comment)

   Default settings:
   

   - name: ulimit
     run: |
         sudo prlimit --pid $$ --nofile=500000:500000
         ulimit -n
   

   
3. al-cheb

   added

   OS: Ubuntu

   Area: OS/VM Configuration

   investigateCollect additional information, like space on disk, other tool incompatibilities etc.Collect additional information, like space on disk, other tool incompatibilities etc.

   and removed

   needs triage

   on Jul 23, 2022
4. ladar commented on Jul 23, 2022

   

   ladar

   on Jul 23, 2022

   Author

   More actions

   So I don't know definitively what caused the breakage between the 20220717 and 20220722 images. Nothing in the repo commit commit log looked like a culprit. So the issue could be at a lower level.

   I don't know which change fixed the problem, but I wasn't able to get the workflow to run reliability until I increased the swap size from 4096 to 8192. Some of my other changes may have also helped, as they reduced the memory footprint. In particular net.unix.max_dgram_qlen combines with net.core.wmem_default and net.core.wmem_max to limit the backlog for unix sockets. Critically though, the Linux kernel requires the entire memory block (qlen*wmem) to be a contiguous for unix sockets. If the host running the image faces memory contention, it might not be able to find a contiguous block, which can cause the symptoms I saw.

   The entire fixes:

   - name: Increase Limits
     run: |
       sudo sysctl -q vm.overcommit_ratio=100
       sudo sysctl -q net.unix.max_dgram_qlen=64
       sudo prlimit --pid $$ --nproc=65536:65536
       sudo prlimit --pid $$ --nofile=500000:500000
       printf "DefaultLimitNPROC=65536:65536\n" | sudo tee -a /etc/systemd/user.conf > /dev/null || exit 1
       printf "DefaultLimitNPROC=65536:65536\n" | sudo tee -a /etc/systemd/system.conf > /dev/null || exit 1
       printf "DefaultLimitNOFILE=500000:500000\n" | sudo tee -a /etc/systemd/user.conf > /dev/null || exit 1
       printf "DefaultLimitNOFILE=500000:500000\n" | sudo tee -a /etc/systemd/system.conf > /dev/null || exit 1
       sudo systemctl daemon-reload
       systemctl --user daemon-reload
   - name: Increase Swap
     run: |
       sudo dd if=/dev/zero of=/swap bs=1M count=4096 status=none
       sudo chmod 600 /swap
       sudo mkswap /swap
       sudo swapon /swap

   I had to rerun the prlimit increase commands for each build step like so:

   - name: Demonstrate Limit Increases for a Build Step
     env:
       GOGC: 50
       GOMAXPROCS: 1
     run: |
       date +"%nStarting a sample/example build step at %r on %x%n"
       sudo prlimit --pid $$ --nproc=65536:65536
       sudo prlimit --pid $$ --nofile=500000:500000
5. al-cheb

   closed this as completedon Jul 25, 2022
6. ladar commented on Jul 29, 2022

   

   ladar

   on Jul 29, 2022

   Author

   More actions

   @al-cheb I saw you closed the bug. Were you able to figure out what changed between the 20220717 and 20220721 images?
7. john-heinnickel commented on Jan 12, 2023

   

   john-heinnickel

   on Jan 12, 2023

   More actions

   I may be wrong, but I came across this issue while trying to remember how I helped a co-worker correct a problem with the open files soft limit previously. What I recall was that while the kernel was configured to support a sufficiently large ulimit maximum, the GitHub Runner Action was spawned from a process in some ancestor unit file that was either explicitly setting or implicitly accepting a reduced soft limit. The way these things inherit, you can always opt into a smaller limit, but you cannot raise your limit higher than what you inherit at process creation time.

   In order for our own set of sufficiently large ulimit -nfiles to be effective, we had to modify this ancestor unit file to not restrict itself and its children quite as much.

   I don't have that running example environment or access to one similar just now, so while I am sure we fixed his by raising the soft ulimit for open files in some ancestor systemd unit file, I cannot seem to recall which one we had to target for GitHub actions to inherit the correctly all the way back to PID=1.

   Anyhow, I see this thread is relatively recent so thought I'd share what I had retained in case it is of some help/value.
8. ladar commented on Feb 23, 2023

   

   ladar

   on Feb 23, 2023

   Author

   More actions

   This can be a hard issue to track down, because it runs contrary to how nix systems historically handled the open file limit (aka a hard max for the kernel and via the ulimit interface). It's also an issue that gets buried inside the morass that is systemd, which is still an opaque MacGuffin to the average system administrator.

   The quickest fix is:

   sudo sed -i '/DefaultLimitNOFILE=/d' /etc/systemd/user.conf
   printf "DefaultLimitNOFILE=65536\n" | sudo tee -a /etc/systemd/user.conf > /dev/null
   
   sudo sed -i '/DefaultLimitNOFILE=/d' /etc/systemd/system.conf
   printf "DefaultLimitNOFILE=65536:524288\n" | sudo tee -a /etc/systemd/system.conf > /dev/null
   

   This will increase the default limits for everything, and save you lots of grief. Of course this assumes your not on a system shared by many, that is resource constricted. If that is the case, you'll need a more granular approach.

   To see where the limits are, run this:

    echo 'user'; systemctl --user show | grep NOFILE ; echo 'system' ;sudo sudo systemctl show  | grep NOFILE

   And don't forget, the kernel maximum, and the ulimit values still apply, so update those as well, if necessary.