Hi,
currently, there is a double pagecache issue:

1. FUSE daemon level: When stargz-snapshotter reads fscache files, data gets loaded into kernel pagecache
2. Container process level: When container processes read files through FUSE, the same data gets loaded into kernel pagecache again

The Direct mode only disables user-space memory cache but still uses regular os.Open() without O_DIRECT flags. This means backend fscache files are still cached in kernel pagecache, leading to double memory consumption.

There are two solutions available, and we can use a configuration option to decide whether to enable.

Option 1: Add posix_fadvise in closeFunc

// In (dc *directoryCache) Get method, modify closeFunc:
closeFunc: func() error {
    if err := posix_fadvise(file.Fd(), 0, 0, POSIX_FADV_DONTNEED); err != nil {
        log.Warnf("failed to drop pagecache: %v", err)
    }

    if opt.passThrough {
        return nil
    }
    // Add posix_fadvise to drop pagecache
    return file.Close()
}

Option 2: Use O_DIRECT for file operations

// In (dc *directoryCache) Get method, use O_DIRECT:
file, err := os.OpenFile(dc.cachePath(key), os.O_RDONLY|syscall.O_DIRECT, 0)

However, this approach may incur a performance overhead.

Are these two solutions acceptable? I can submit a PR to implement the one that makes the most sense.

Best regards,
Abush Wang