Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.RaiseChangeEvents throws uncatchable IOException on thread pool thread

@jozkee

Description

I use Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher in polling mode. If polling fails with an IOException, it takes the process down as the exception is thrown on a thread pool thread where it can not be catched.

Related: #71003 #72462
Reading the related issues, it seems like @jozkee is the right one to tag :-)
cc: @danmoseley

Stacktrace

Unhandled exception. System.IO.IOException: Host is down : '/mnt/smbshare'
   at System.IO.Enumeration.FileSystemEnumerator`1.Init()
   at System.IO.DirectoryInfo.InternalEnumerateInfos(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options)
   at Microsoft.Extensions.FileSystemGlobbing.Abstractions.DirectoryInfoWrapper.EnumerateFileSystemInfos()+MoveNext()
   at System.Collections.Generic.List`1.AddRange(IEnumerable`1 collection)
   at Microsoft.Extensions.FileSystemGlobbing.Internal.MatcherContext.Match(DirectoryInfoBase directory, String parentRelativePath)
   at Microsoft.Extensions.FileProviders.Physical.PollingWildCardChangeToken.CalculateChanges()
   at Microsoft.Extensions.FileProviders.Physical.PollingWildCardChangeToken.get_HasChanged()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.RaiseChangeEvents(Object state)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Details of my setup/environment

I don't think this issue is related to my setup/environment, just adding for completeness & reproducibility.

My code runs inside an ubuntu-chiseled-based docker container. On the unix container host, I have a CIFS mount defined in /etc/fstab like this:

//smbhost/smbshare$ /mnt/smbshare cifs credentials=/etc/smbcredentials_smbshare,iocharset=utf8,rw,file_mode=0777,dir_mode=0777 0 0

This mount is then further passed to the container:

worker:
  image: somecorp.azurecr.io/some-worker:latest
  read_only: true
  cap_drop:
    - ALL
  security_opt:
    - no-new-privileges:true
  environment:
    - DOTNET_USE_POLLING_FILE_WATCHER=1
  volumes:
    - /mnt/smbshare:/mnt/smbshare:Z

The mounted volume is only available at specific times though (say, 9 to 5 but not at night). Thus it will predictably go down and when it does it takes my worker process with it.

Reproduction Steps

Creation of my PhysicalFilesWatcher is quite straight forward:

_fileProvider = new PhysicalFileProvider(directoryToWatch);
_disposable = ChangeToken.OnChange(
    () => _fileProvider.Watch("**/*"),
    () =>
    {
        // Do something
    });

I have DOTNET_USE_POLLING_FILE_WATCHER=1set.

Point this PhysicalFilesWatcher to a path that becomes unavailable, in my case the host of a mounted network file system goes down. On the next poll, the process stops forcefully.

Expected behavior

PhysicalFilesWatcher does not throw any uncatchable exceptions.

One of:

IOExceptions are catched, like @danmoseley suggested but in all relevant places
There is an event or a similar interception possibility that can be used to handle these exceptions.

Actual behavior

PhysicalFilesWatcher takes processes down because it throws uncatchable exceptions.

Regression?

I don't think this is a regression.

Known Workarounds

Only: Don't use PhysicalFilesWatcher.

Configuration

docker container run --rm --entrypoint dotnet my-worker:latest --info

Host:
  Version:      8.0.0
  Architecture: x64
  Commit:       5535e31a71
  RID:          linux-x64

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Other information

No response

I don't have (or don't remember) context, but is there already an established pattern in file watcher for exceptions on its threadpool threads? If so any interest in offering a PR applying it in this path?

Sorry for the delay. I don't think there is a very clear established pattern.

FileSystemWatcher

If PhysicalFileWatcher uses the FileSystemWatcher backend, it handles the FileSystemWatcher.Error Event, which's docs read (emphasis mine):

This event is raised whenever something prevents the FileSystemWatcher object from monitoring changes. For example, if the object is monitoring changes in a remote directory and the connection to that directory is lost, the Error event is raised.

In that case, the listeners will be notified:

private void OnError(object sender, ErrorEventArgs e)
{
    // Notify all cache entries on error.
    foreach (string path in _filePathTokenLookup.Keys)
    {
        ReportChangeForMatchedEntries(path);
    }
}

Polling

The polling implementations PollingWildCardChangeToken.cs and PollingFileChangeToken.cs do however not contain any try-catch blocks at all.

How to proceed

From the top of my head there are multiple options:

Make PollingWildCardChangeToken and PollingFileChangeToken signal on error. They are public types though and if they are not used in the context of PhysicalFileWatcher, errors can be catched when geting the HasChanged property's value.
Add some OnError mechanism to PollingWildCardChangeToken and PollingFileChangeToken. This would be a new public API though.
Just have a try-catch-block in PhysicalFilesWatcher where it polls the tokens. This would get the polling behaviour in line with FileSystemWatcher-backed instances. It also wouldn't add/change any public API surface. It also wouldn't change any public API's behaviour - except from not taking down processes due to uncatchable exceptions on the thread pool - and signalling the listeners instead.

Thus the third option seems to be preferrable to me. It also seems like an almost non-invasive change (any chance to get this in a servicing release?) to me.

I'd be happy to create a PR when we agree on how to proceed 👍.

+1 to this. I have also encountered the same problem there is currently no way to recover when the connection to the remote directory is lost.

If PhysicalFileWatcher uses the FileSystemWatcher backend, it handles the FileSystemWatcher.Error Event

There are also uncaught exceptions on FSW when certain folders can't be enumerated, see callstack in #91879.

There's a 4th option, DirectoryInfoWrapper uses _directoryInfo.EnumerateFileSystemInfos("*", SearchOption.TopDirectoryOnly) that doesn't enable EnumerationOptions.IgnoreInaccessible. I would expect that using such option could fix this case.

However, it's been historically annoying to deal with unexpected exception in RaiseChangeEvents:

Wrapping the whole method body in a try-catch could also be best if its safe to do so.

Judging from it's docs alone I don't think using IgnoreInaccessible would fix cases where a network share goes down:

Gets or sets a value that indicates whether to skip files or directories when access is denied (for example, UnauthorizedAccessException or SecurityException).

Looking at it's source I'm unsure for which cases EBADF stands. The explicit "Host is down" message in my original stack trace could however indicate the internal error code is rather Error_EHOSTDOWN. Which means, if I understand it correctly, option 4 wouldn't fix this issue.

@georg-jung

I see, unless we modify IgnoreInaccessible on Linux, option 4 won't work. @georg-jung would you like to explore option 3 and send a PR?

Any updates on this issue? We are observing a few similar crashes and see no way of recovering from it while using the current active polling functionality.

Hi .NET team, could you share the planned timeline or target release for addressing this issue? Thanks!

Triage: We should provide some infrastructure to provide way to route the exception to developer.

Handle transient file system errors during active polling (#132617)

Navigation Menu

Repository navigation

Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.RaiseChangeEvents throws uncatchable IOException on thread pool thread #95631

Description

Stacktrace

Details of my setup/environment

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

FileSystemWatcher

Polling

How to proceed

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.RaiseChangeEvents throws uncatchable IOException on thread pool thread #95631

Description

Description

Stacktrace

Details of my setup/environment

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Activity

danmoseley commented on Dec 5, 2023

georg-jung commented on Dec 8, 2023

FileSystemWatcher

Polling

How to proceed

joel-jeremy commented on Dec 12, 2023

jozkee commented on Dec 14, 2023

jozkee commented on Dec 14, 2023

georg-jung commented on Dec 15, 2023

jozkee commented on Dec 15, 2023

sbs-dev1 commented on Jul 30, 2025

xwang-dev commented on Oct 6, 2025

cincuranet commented on Mar 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions