[LLVM] Remove the requirement for named pipe in jobserver#169154
[LLVM] Remove the requirement for named pipe in jobserver#169154
Conversation
|
@llvm/pr-subscribers-llvm-support Author: Michał Górny (mgorny) ChangesRemove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead:
This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. CC @amonakov 1 Files Affected:
diff --git a/llvm/lib/Support/Unix/Jobserver.inc b/llvm/lib/Support/Unix/Jobserver.inc
index 53bf7f288ca1f..300fb5e55045e 100644
--- a/llvm/lib/Support/Unix/Jobserver.inc
+++ b/llvm/lib/Support/Unix/Jobserver.inc
@@ -19,14 +19,6 @@
#include <unistd.h>
namespace {
-/// Returns true if the given file descriptor is a FIFO (named pipe).
-bool isFifo(int FD) {
- struct stat StatBuf;
- if (::fstat(FD, &StatBuf) != 0)
- return false;
- return S_ISFIFO(StatBuf.st_mode);
-}
-
/// Returns true if the given file descriptors are valid.
bool areFdsValid(int ReadFD, int WriteFD) {
if (ReadFD == -1 || WriteFD == -1)
@@ -75,7 +67,7 @@ JobserverClientImpl::JobserverClientImpl(const JobserverConfig &Config) {
case JobserverConfig::PosixFifo:
// Open the FIFO for reading. It must be non-blocking and close-on-exec.
ReadFD = ::open(Config.Path.c_str(), O_RDONLY | O_NONBLOCK | O_CLOEXEC);
- if (ReadFD < 0 || !isFifo(ReadFD)) {
+ if (ReadFD < 0) {
if (ReadFD >= 0)
::close(ReadFD);
ReadFD = -1;
|
|
This change looks good functionally, but I think it leaves some confusing naming and comments behind. Previously, JobserverConfig::PosixFifo was tightly coupled to a real POSIX FIFO (we enforced S_ISFIFO), so the fifo naming and comments in this code matched the actual kernel object. Now that we intentionally accept any endpoint that implements the jobserver token protocol (including FUSE files or CUSE character devices), the FIFO terminology becomes misleading: it's really 'FIFO-compatible endpoint as described by fifo: in MAKEFLAGS', not necessarily a kernel FIFO. It would be clearer to future readers if we adjusted the naming/comments in a follow-up:
|
|
It might be worth adding a small regression test to lock in the new behavior around fifo: endpoints. Right now the Unix jobserver tests (JobserverClientTest.UnixClientFifo and the JobserverStrategyTest cases) only exercise the "happy path" with a real FIFO created by ScopedFifo. Before this patch, JobserverClientImpl would reject any --jobserver-auth=fifo:PATH that did not point to a S_ISFIFO object; after this patch we intentionally accept non-FIFO endpoints as long as open() succeeds. To make that intent explicit and prevent future regressions, I would suggest adding a test alongside UnixClientFifo that:
This would clearly document that fifo: is now a protocol tag rather than a strict POSIX FIFO requirement, and that we deliberately allow FUSE/CUSE-backed implementations presenting themselves under a fifo: URL. |
Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. Signed-off-by: Michał Górny <[email protected]>
Signed-off-by: Michał Górny <[email protected]>
762cdd3 to
32e2acb
Compare
|
Rebased and added a test so far. I've just copied the whole logic from |
|
@yxsamliu ping |
Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. CC @amonakov --------- Signed-off-by: Michał Górny <[email protected]>
Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. CC @amonakov --------- Signed-off-by: Michał Górny <[email protected]>
Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead:
nixos-jobserver (WIP) uses simple file on FUSE
steve uses a character device via CUSE
guildmaster uses a character device via CUSE
This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility.
CC @amonakov