Describe the bug
I ran into this issue while using https://github.com/spf13/viper which uses this library as a dependency and I understand that this issue may not be related to fsnotify but rather how viper handles the error.
Suppose there's a directory that looks like this:

dir/
|- a.txt <- watched
|- b/ <- unix.Open returns EPERM 1 Operation not permitted
|- c.yaml <- file I'm interested in, not watched

And these 2 things happen:

1. c.yaml is never reached
2. The only event that's ever going to be fired is "b/ is created"

So in viper a watch for a directory is added here https://github.com/spf13/viper/blob/v1.10.1/viper.go#L450. During Add(), the code loops through all the files under dir/ here https://github.com/fsnotify/fsnotify/blob/v1.5.1/kqueue.go#L398, which include a.txt, b/, and c.yaml, in order.

	for _, fileInfo := range files {
		filePath := filepath.Join(dirPath, fileInfo.Name())
		filePath, err = w.internalWatch(filePath, fileInfo)
		if err != nil {
			return err
		}

		w.mu.Lock()
		w.fileExists[filePath] = true
		w.mu.Unlock()
	}

When w.internalWatch() is called for b/, an err is returned, which causes Add() to return with err. However, the watcher is already updated with w.fileExists["dir/a.txt"] = true. On the other end of the watcher, the w.readEvents() goroutine loops over all the files under dir/ again to discover new files. Relevant code here: https://github.com/fsnotify/fsnotify/blob/v1.5.1/kqueue.go#L444

func (w *Watcher) sendFileCreatedEventIfNew(filePath string, fileInfo os.FileInfo) (err error) {
	w.mu.Lock()
	_, doesExist := w.fileExists[filePath]
	w.mu.Unlock()
	if !doesExist {
		// Send create event
		select {
		case w.Events <- newCreateEvent(filePath):
		case <-w.done:
			return
		}
	}

	// like watchDirectoryFiles (but without doing another ReadDir)
	filePath, err = w.internalWatch(filePath, fileInfo)
	if err != nil {
		return err
	}

It first sees a.txt, which doesExist. It then sees b/. b/'s doesExist is false so a new event is fired. When the code runs to filePath, err = w.internalWatch(filePath, fileInfo), it errors again and returns early without going through the rest of the directory.

To Reproduce

Expected behavior
I think ideally for my use case, either make watchDirectoryFiles() and sendDirectoryChangeEvents() continue if a file fails to open, or have another map to track files that fail to open so it doesn't keep firing off events about them.

Which operating system and version are you using?

macOS: sw_vers

ProductName:	macOS
ProductVersion:	11.6.4
BuildVersion:	20G417

Additional context
If applicable, add screenshots or a code sample to help explain your problem.