poke @ncw :-)

Thanks for kicking CI! That seems to be coreos/go-systemd#440. Adding a replace directive to my go.mod gets a GOOS=plan9 go build ./cmd/serve to pass.

Can you stub it out on plan 9?

That's what I did upstream, in coreos/go-systemd#440.

If this doesn't get merged there, I could add a little wrapper package in rclone, and do the plan9 stubbing there, but it'd be more code on the rclone side.

Is this the only thing blocking this, or do you have other feedback on the code?

Well you could do the same thing in the rclone code with build tags - that is what I was suggesting. Though if your pr is merged that is the neatest way for rclone!

I have looked at the code which looks ok (I'll do a formal review at some point). I would like some explanation of how it works in the docs and example config of how you would use it with systemd would be very useful.

Ones thing that wasn't clear to me - does systemd start a new rclone instance for every incoming connection? Or does it keep things running for a while. I guess this might be in the systemd docs!

Yeah, I'm also hoping PR gets merged into go-systemd, as essentially everyone linking against the activation package and building or this arch would do stubbing on their own, and there already is a windows stub...

Regarding systemd configuration, there's a lot of different config knobs, which is why I only linked to the systemd docs. You can bind on Unix sockets, TCP ports, ...

rclone keeps running, and normally gets started lazily on the first connection. If rclone had a parameter to exit the process after a specific time of inactivity, it would also get fully shut down, but that's a bit out of scope for this PR.

Hmm, looks like we might have to go for build tags - not a lot of action on that PR...

Alright, let me take a look at that

I rebased this, and moved the wrapping into our own wrapper package, linking to the upstream issue. also added socket activation to the serve sftp command (which only has a single listener).

@ncw sorry for the noise, but can you kick off CI once again?

Poke - this should be ready now.

I just stubled over #7392, which tried doing the same thing. Some feedback on what'd be needed to get either of the two accepted would be appreciated.

Sorry for the delay on this @flokli

Can you rebase it on master please and fixup any problems (might not be any but there have been lots of commits to master since this started).

@ncw I rebased this, and successfully tested socket-activating a webdav and sftp server.

systemd-socket-activate -l 8088 -l 8089 --fdname=foo:bar -- ./rclone serve webdav :local:vfs/
systemd-socket-activate -l 2022 --fdname=foo:bar -- ./rclone serve sftp :local:vfs

… both from the root of the rclone checkout, after a go build ./rclone.go.

PS I think all the test failures are flakes so I've re-run them.

The windows lint is actually correct - will push a fix.

Can you put these commands in the docs please? It is great to have some commands people can cut and paste to try the feature.

It really isn't how I'd expect people to run this in production usecases - where they'll probably configure systemd .service and .socket files. I can try to find a wording that makes sure this only makes sense for testing purposes.

Auxillary question - do you think we should roll this a bit differently, essentially switch to be socket-activated if we detect to be in such an environment (and have a CLI flag to disable it), but keep the existing listener syntax mostly?

PS I think all the test failures are flakes so I've re-run them.

The windows lint is actually correct - will push a fix.

👍

Can you put these commands in the docs please? It is great to have some commands people can cut and paste to try the feature.

It really isn't how I'd expect people to run this in production usecases - where they'll probably configure systemd .service and .socket files. I can try to find a wording that makes sure this only makes sense for testing purposes.

Great.

Auxillary question - do you think we should roll this a bit differently, essentially switch to be socket-activated if we detect to be in such an environment (and have a CLI flag to disable it), but keep the existing listener syntax mostly?

How does this change the listener syntax? Do you mean where it ignores the command line args if it detects socket activation?

Not sure I understand the question (probably not enough coffee yet :-)

It really isn't how I'd expect people to run this in production usecases - where they'll probably configure systemd .service and .socket files.

right, which is why an example socket and service file would probably be useful to some in documentation, right?

It really isn't how I'd expect people to run this in production usecases - where they'll probably configure systemd .service and .socket files.

right, which is why an example socket and service file would probably be useful to some in documentation, right?

The docs already link to https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html, which links to a bunch of guides:

For more extensive descriptions see the "systemd for Developers" series: Socket Activation, Socket Activation, part II, Converting inetd Services, Socket Activated Internet Services and OS Containers.

There's plenty of explanation there, and I'm fairly certain people looking for this feature are already aware of how it works, or willing to do some research on it if they want it. Duplicating a small subset of examples into an individual rclone documentation item feels like a bit too much IMHO. This is the rclone documentation, we can mention we do support it, be helpful and link to some more upstream documentation, but we shouldn't try to document every other auxillary tool rclone can interface. We also don't describe how to use an sftp or http client - I see this in a similar league.

yeah i mean you and i find this topic immediately recognizable so it seems basic - i just wasn't sure this would be obvious to everyone

I'm fairly certain people looking for this feature are already aware of how it works

actually that's a great point... while this won't be obvious to everyone of course it will likely be obvious to everyone looking for it

on that note... thank you for your contribution here @flokli! i was just looking for this feature 2 days ago and found this PR... somehow i wasn't surprised to see your name 😉

Auxillary question - do you think we should roll this a bit differently, essentially switch to be socket-activated if we detect to be in such an environment (and have a CLI flag to disable it), but keep the existing listener syntax mostly?

How does this change the listener syntax? Do you mean where it ignores the command line args if it detects socket activation?

Not sure I understand the question (probably not enough coffee yet :-)

I think I confused myself too 😆

With this PR, lib/http/server.go supports both TLS and non-TLS listening (by looking at tlsConfig). It doesn't look at ListenAddr if socket-activated, and this is documented in the help.

I took another pass over the docs, mentioned .socket and .service unit files, and mentioned systemd-socket-activate for ad-hoc testing:

+It can be configured with .socket and .service unit files as described in                                                                    
+https://www.freedesktop.org/software/systemd/man/latest/systemd.socket.html                                                                  
+                                                                                                                                             
+Socket activation can be tested ad-hoc with the ` + "`systemd-socket-activate`" + `command:                                                  
+                                                                                                                                             
+       systemd-socket-activate -l 2222 -- rclone serve sftp :local:vfs/                                                                      
+                                                                                                                                             
+This will socket-activate rclone on the first connection to port 2222 over TCP.   

@ncw friendly ping :-) is there anything left you want me to change here, or can we land this?