This whole thing looks really scary and weird.

Can you please explain this? Why are we doing it?
We are only ever calling the function returned by filterWait immediately after getting it. How is this safe for concurrency?

This is a pattern to limit only one go routine to execute in that function at any point in time. The pattern works like this: When you call filterWait you try to write to a buffered channel of size 1. If nobody has written anything there the write will go through and you are allowed to execute the function. And filterWait returns a closure which will read from that channel which you call in defer i.e you read from the channel what you wrote when you are done. The channel buffer size determines how much concurrency you want. In this case you want only one single go routine to concurrently execute the caller function. You can use the same pattern to limit the concurrency to any number you want.

In this particular case it is to serialize the iptable plumbing function to be executed by only one go routine at any time to fix any race when multiple go routines try to execute these functions concurrently.

In general, I try to avoid using mutex for code serialization as it more often than not results in deadlocks. I use mutexes only to protect data not code which is exactly how they should be used.

IMO it has the same chance of a dealock and race as a mutex because there is no way to try or cancel. This is basically non-obvious, slow mutex.
However, it's probably up to @mrjana to decide what to use in libnetwork :)

I see.