Closes: #850

Adding some rudimentary signal propagation to allow testing graceful shutdown logic via cargo lambda watch.

As a refresher, SIGTERM and SIGINT are sent to functions if and only if at least one extension is registered. And then, the delay before SIGKILL is 500ms if there is an internal extension registered, or 300ms if only an external.

More information can be found here:
https://github.com/aws-samples/graceful-shutdown-with-aws-lambda

My use case is that I want to validate function graceful shutdown logic without relying on opaque AWS decisions on when to kill a running function.

I added some minimalist logic to somewhat mirror the above:

1. track the count of internal vs external extensions registered and stash it in the extension cache
2. on SIGINT or SIGTERM against the cargo lambda watch process, check whether either extension type was set, and return the corresponding shutdown delay if so
3. if graceful shutdown is needed, fire off the SIGTERM/SIGINT signal, and then race the process ending, against the maximum delay, and SIGKILL if not done in time. Either way, clean up the watchexec.

I did NOT implement the following:

• graceful shutdown on hot reload due to recompile - it was pretty klunky trying to chain watchexec action outcomes with non-watch-exec logic inbetween, I didn't bother futzing with it
• graceful shutdown (of external extensions) - internal extensions already get the same signals as the function, but external extensions are not currently hooked into watchexec. I didn't bother wiring it up.
• any sort of sequencing between the runtime shutting down and extensions receiving SHUTDOWN events / signals - this would have been much larger scope

These gaps are documented in the new doc section for this PR. If somebody needs any of those features, presumably they could layer them on top down the road.

Testing

My graceful shutdown testbed was this file.

With my new logic, Control-C-ing on my watch task in debug mode generated:

DEBUG watcher action received action=Action { events: [Event { tags: [Source(Keyboard), Signal(Interrupt)], metadata: {} }], outcome: OnceCell(Uninit) } signals=[Interrupt] has_paths=false empty_event=false
 INFO terminating lambda scheduler
 INFO terminating lambda function function="_"
DEBUG attempting graceful function shutdown signal_type=Interrupt max_delay=500m
 INFO [runtime] SIGINT received
 INFO [runtime] Graceful shutdown in progress ...
[runtime] Graceful shutdown completed

If I add a second-long sleep (longer than the 500ms delay before SIGKILL), I instead see the shutdown not complete:

INFO terminating lambda scheduler
DEBUG watcher action received action=Action { events: [Event { tags: [Source(Keyboard), Signal(Interrupt)], metadata: {} }], outcome: OnceCell(Uninit) } signals=[Interrupt] has_paths=false empty_event=false
DEBUG attempting graceful function shutdown signal_type=Interrupt max_delay=500m
 INFO terminating lambda function function="_"
 INFO [runtime] SIGINT received
 INFO [runtime] Graceful shutdown in progress ...

Some sort of warn message that time ran out would be nice, but as I mentioned, it's pretty klunky trying to chain non-watchexec logic inside the outcomes, so I didn't bother.

Finally, in a function with NO extensions registered, I don't see graceful shutdown attempted at all:

DEBUG watcher action received action=Action { events: [Event { tags: [Source(Keyboard), Signal(Interrupt)], metadata: {} }], outcome: OnceCell(Uninit) } signals=[Interrupt] has_paths=false empty_event=false
 INFO terminating lambda function function="_"
 INFO terminating lambda scheduler