Failure listeners and fallback sinks by nblumhardt · Pull Request #2108 · serilog/serilog

nblumhardt · 2024-08-30T12:03:06Z

Implements #1791.

The core addition is a new interface, ILoggingFailureListener, through which sinks can report temporary or permanent logging failures.

The most direct way to consume this is hooked up as WriteTo.Fallible(), which installs a provided failure listener for a given sink:

    .WriteTo.Fallible(
        wt => wt.Email(...),
        new MyFailureListener())

Failure listeners get enough information about failures that monitoring/metrics use cases should be well handled.

The common batching infrastructure has been updated to support failure listeners, so existing sinks that either a) throw exceptions synchronously on failure, or b) use the common batching infrastructure, will automatically support failure listeners.

Sinks that don't fall into either category can implement ISetLoggingFailureListener and receive a failure listener at start-up.

The second, higher-level way to consume the new functionality is through fallback chains: the WriteTo.FallbackChain() method accepts an ordered sequence of sinks to try, and using failure listeners, will pass failed writes on to the next sink in the chain:

    .WriteTo.FallbackChain(
        // First try writing to Seq
        wt => wt.Seq(...),
        // If that fails, try email
        wt => wt.Email(...),
        // Finally, write events to a file
        wt => wt.File(...))

A few APIs get added and so the feature needs some careful scrutiny. It's possible to construe this as bloat, but it's also proven very hard to implement anything similar in a general/cross-cutting way outside the core, so I'm leaning towards pushing ahead with this. Keen to streamline and refine it as much as we can.

nblumhardt · 2024-08-30T13:19:04Z

Making a note that Serilog.Sinks.Async will need an update to support this if/when merged.

alsi-lawr · 2024-08-30T21:39:39Z

@nblumhardt thanks for the effort that's gone into this, it's greatly appreciated.

If I understand correctly, would there essentially be 2 kinds of sinks that can be wrapped with a fallback chain:

Sinks that explicitly implement the ISetFailureLoggingSink interface get passed whatever failure listener configured during logger configuration, and then must explicitly call the OnLoggingFailed method based on the LogFailureKind
Sinks that don't implement the new ISetFailureLoggingSink interface get minimal support through exceptions only

Is there also support for state capture? Is that the goal of the object sender parameter of the OnLoggingFailed method?

alsi-lawr · 2024-08-30T21:41:06Z

+        LoggingFailureKind kind,
+        string message,
+        IReadOnlyCollection<LogEvent>? events,
+        Exception? exception);


Is there a reason that Exception is preferred over ExceptionDispatchInfo to preserve the stack trace?

Thanks for taking a look!

If the failure is to be forwarded on to another Serilog sink, an Exception will be needed; ExceptionDispatchInfo is useful for re-throwing, but can't expose enough information to be useful for logging, unfortunately.

ArieGato · 2024-09-26T05:30:26Z

Hi @nblumhardt,

Will it be possible to configure the Fallback sinks through configuration? Will it require a change in the Serilog.Settings.Configuration?

Cheers,
Arjan

nblumhardt · 2024-09-26T06:07:44Z

Thanks for taking a look @ArieGato. The API used in the PR should work with the current feature set of Serilog.Settings.Configuration 👍

As an intermediate step, it should also be possible to configure fallbacks internally within your own sink configuration method, if you want to preserve configuration/API compatibility.

nblumhardt · 2024-09-27T21:12:11Z

Planning to merge this after #2118 so we can try it out more broadly. It's a nice benefit of the switch to a built-in batching implementation that this should work with most of the existing ecosystem without any changes :-)

nblumhardt · 2024-09-28T21:51:18Z

+{
+    public void OnLoggingFailed(object sender, LoggingFailureKind kind, string message, IReadOnlyCollection<LogEvent>? events, Exception? exception)
+    {
+        if (events is not null && kind is LoggingFailureKind.Final or LoggingFailureKind.Permanent)


For batched sinks this will only occur after retries have been performed, causing a delay before the fallback sink will see the events.

By default I think this is okay, but we may wish to add a RetryCount or DisableRetries type of setting to BatchingOptions so that fallbacks are tried immediately when this behavior is required.

ArieGato · 2024-10-22T05:09:12Z

Thanks for taking a look @ArieGato. The API used in the PR should work with the current feature set of Serilog.Settings.Configuration 👍

As an intermediate step, it should also be possible to configure fallbacks internally within your own sink configuration method, if you want to preserve configuration/API compatibility.

hi @nblumhardt: I'm trying to configure the fallback/fallback chain sinks through appsettings.json, but I can't get it to work. Can you give an example how I can achieve this?

nblumhardt · 2024-10-25T05:16:27Z

Hi @ArieGato; I suspect I got this wrong, and the params usage in the API isn't yet supported by Serilog.Settings.Configuration. I'll open a ticket over there.

ArieGato · 2024-11-04T11:44:31Z

@nblumhardt I did some investigating and I think the problem with configuring the FallackChain is partly due to the fact that the WriteTo.FallbackChain is not an extension method. This can be resolved in two ways:

Change the method to an extension method so that it is more inline with configuring Sinks.
Alter Serilog.Settings.Configuration to support normal non-extension methods.

What are your thoughts on this?

nblumhardt · 2024-11-08T21:17:36Z

Thanks for taking a look @ArieGato. There's already a mechanism in Serilog.Settings.Configuration to deal with these:

https://github.com/serilog/serilog-settings-configuration/blob/dev/src/Serilog.Settings.Configuration/Settings/Configuration/SurrogateConfigurationMethods.cs

I'm a bit short on time for the next few weeks unfortunately, but it looks like it should be a fairly quick addition.

davidkarlsson · 2025-02-22T14:21:23Z

What would adding support for this to Serilog.Settings.Configuration look like ideally? Wouldn't it be weird to try to add FallbackChain to WriteTo somehow since that key just takes an array of sinks? Should a separate "write to with fallbacks" key be added for this instead even though it then wouldn't match the Serilog api for it?

nblumhardt · 2025-02-24T04:50:09Z

@davidkarlsson I think the current mapping of C# configuration onto JSON means that the "FallbackChain" would be under "WriteTo", and then under that would be another level of "WriteTo". It's a bit awkward, really needs some tinkering with :)

ArieGato · 2026-04-19T12:26:14Z

@nblumhardt I did some investigating and I think the problem with configuring the FallackChain is partly due to the fact that the WriteTo.FallbackChain is not an extension method. This can be resolved in two ways:

Change the method to an extension method so that it is more inline with configuring Sinks.

Alter Serilog.Settings.Configuration to support normal non-extension methods.

What are your thoughts on this?

@nblumhardt I finally got round adding support for FallbackChain and Fallable to Serilog.Settings.Configuration.

serilog/serilog-settings-configuration#474

Can you take a look at it?

nblumhardt added 5 commits August 29, 2024 15:51

Fallbacks spike

4708423

One test

adbec37

Disposal

48c2587

Project ISetLoggingFailureListener through wrapper sinks

25ddb84

Tidy up

f0e2c6a

nblumhardt mentioned this pull request Aug 30, 2024

Add fallback/file buffering when requests can't be sent serilog/serilog-sinks-opentelemetry#155

Open

alsi-lawr reviewed Aug 30, 2024

View reviewed changes

nblumhardt mentioned this pull request Sep 5, 2024

Fail-over to a second server if the first is unreachable datalust/serilog-sinks-seq#51

Closed

nblumhardt mentioned this pull request Sep 25, 2024

Upgrade Serilog to 4.0.x ArieGato/serilog-sinks-rabbitmq#212

Merged

nblumhardt commented Sep 28, 2024

View reviewed changes

nblumhardt merged commit 3da0979 into serilog:dev Oct 1, 2024

nblumhardt mentioned this pull request Oct 4, 2024

4.1.0 Release #2123

Merged

nblumhardt mentioned this pull request Oct 19, 2024

Support ILoggingFailureListener and fallback sinks serilog/serilog-sinks-async#98

Merged

nblumhardt mentioned this pull request Oct 25, 2024

params arguments should be regarded as optional serilog/serilog-settings-configuration#441

Closed

nblumhardt mentioned this pull request Nov 4, 2025

SelfLog with Log Level #1791

Closed

This was referenced May 1, 2026

RestrictedSink does not forward ISetLoggingFailureListener to inner sink #2231

Closed

Route optional interfaces through OptionalInterfaceForwardingSink for restricted sinks #2234

Merged

Uh oh!

Conversation

nblumhardt commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nblumhardt commented Aug 30, 2024

Uh oh!

alsi-lawr commented Aug 30, 2024

Uh oh!

alsi-lawr Aug 30, 2024

Choose a reason for hiding this comment

Uh oh!

nblumhardt Sep 2, 2024

Choose a reason for hiding this comment

Uh oh!

ArieGato commented Sep 26, 2024

Uh oh!

nblumhardt commented Sep 26, 2024

Uh oh!

nblumhardt commented Sep 27, 2024

Uh oh!

nblumhardt Sep 28, 2024

Choose a reason for hiding this comment

Uh oh!

ArieGato commented Oct 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nblumhardt commented Oct 25, 2024

Uh oh!

ArieGato commented Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nblumhardt commented Nov 8, 2024

Uh oh!

davidkarlsson commented Feb 22, 2025

Uh oh!

nblumhardt commented Feb 24, 2025

Uh oh!

ArieGato commented Apr 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nblumhardt commented Aug 30, 2024 •

edited

Loading

ArieGato commented Oct 22, 2024 •

edited

Loading

ArieGato commented Nov 4, 2024 •

edited

Loading