Thanks for this PR/bug fix. I'll take a look tomorrow and give more feedback, in the mean time I've added a few reviewers.

Thanks for reviewing @mathisrichter!

I will implement your suggestions during the coming days and will also think of a way to write some specific tests.

Some follow-up information

Below is a screenshot of my terminal emulator, showing proof of concept:

This does the following:

• define an alias to run test_learning_rule and filter for relevant output
• set a low ulimit for open file descriptors
• run test with main branch -> fails with "Too many open files"
• run test with fix branch -> passes
• set higher ulimit for open file descriptors
• run test with main branch -> passes now

This means that with low ulimit, only the fix branch avoids the files error, but with high ulimit, both avoid it. This result is deterministic. It can be run any number of times with the same result.

I am running:

• Python 3.10.9
• Linux kernel 6.1.12

However: I tested the fix today on my MacBook (macOS Ventura 13.2) and there it does not work. After going back to the Linux machine and taking a deeper look, I noticed that if I call the tests in a specific way (by repeating the same test a large number of times), some descriptors still leak, although much more slowly, eventually leading to the same error.

Conclusion: This change fixes some leaks, but not all of them yet. At least one remains. I will try to track them all down and come back with an update soon.

@ssgier Thanks for that deeper analysis! I brought up your PR internally and learned that there may be tests for this error in some branch. I asked @joyeshmishra to add them as a comment to this thread.
But I don't know the details of what those tests cover, so in the meantime, please continue with your own investigation.

Update: Found some more leaks and committed a draft of a fix here.

New behavior: With these fixes applied, all file descriptor leaks appear to be resolved on both Linux and macOs and the whole test suite runs through with low ulimit.

Next steps: I will implement @mathisrichter's suggestions and also write an explicit test. It turns out that there is a simple way of doing this:

import psutil
p = psutil.Process()
num_fds_before = p.num_fds()

<run subject under test>

num_fds_after = p.num_fds()
assert num_fds_after == num_fds_before

Details

1. The shared memory manager instance opens pipes internally, which are only closed when the Python object is reclaimed.
2. Some of the ports are never joined by the runtime.
3. This one is specific to macOS: the multiprocessing semaphore opens a special file descriptor: the POSIX Semaphore (type PSXSEM when running lsof). This file descriptor is only closed when the Python object is reclaimed.
4. Threads were leaking in pypychannel. They waited on the semaphore and never got to check the self._done condition. This is not a file descriptor leak, but it is related to fixing item 3, because the threads have a reference to the semaphore object. Cleanest would be to also join the threads, but this slows down the stopping process of the runtime.

I tried to keep the fixes as non-intrusive as possible. One shortcoming is that some of them rely on objects being reclaimed, but it works in practice and from what I can see, it does not break existing functionality.

@joyeshmishra thanks for reviewing! I did verification on Linux and macOS but not on Windows. Is resource leakage also a problem on Windows? It looked like a problem specific to Unix-like systems.

As discussed separately with @mathisrichter, removed the now obsolete hint from README and tutorial. The tag referred in the README would have to be updated in the next release.