This is probably already well known, but it bit me while planning to do some comparative benchmarks between Linux and OSX. I started with OSX and direct=1, rw=randread, but noticed that OSX was reading at 1200MB/s from a 1GB file. This was unexpected as I only have one spinning rust HDD in the MBP.

So I looked up the ways to do direct I/O on OSX. On stack overflow and the Apple mailing lists, fcntl(fd, F_NOCACHE, 1) looked to be the canonical solution. This was also implemented in fio(1) in 2011 in commit 7e8ad19. It seems that F_NOCACHE disables the page cache from that point on, but the file in quesition was already in the page cache, it will not be purged and the pages will be used.

I also commented on stack overflow: clear buffer cache on OSX with my observations. I'll copy it here as well:

It's my impression that even when turning off the cache like this (with F_NOCACHE or F_GLOBAL_NOCACHE), if there are pages of the file already in the page cache, those will still be used. I tried to test this by using fio(1) with direct=1. It seems to confirm my suspicions (I get ~1200MB/s throughput on a random read, on a spindle HDD in my MBP, not an SSD). I've confirmed with dtruss that fio(1) actually calls fcntl correctly.

After running "sudo purge" and trying the same fio(1) invocation, it's much slower. So yes, it appears that F_NOCACHE is no direct equivalent of O_DIRECT on Linux.

I'm not advocating running sudo purge as part of fio, but perhaps it could be added to the documentation of direct that the behaviour is quite different from O_DIRECT but they're both fio's way of doing direct IO. Running sudo purge is both slow, has an adverse effect on the rest of the system while (and after) it runs for obvious reasons.

Another idea I had was to forcibly re-write the file each time for reading, while having the file open with F_NOCACHE, which makes the written pages not enter the page cache (UPC). That would also be slow (possibly) but hopefully it wouldn't evict other, unrelated files.

An interesting discussion involving an Apple dev on the Apple mailing list: http://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00010.html. Specifically the part mentioning that files opened with F_NOCACHE will still uses already loaded pages if they're present.

EDIT: Some train of thought rambling: but perhaps mmap + MAP_NOCACHE on OSX is a possible avenue (+ telling uses to use that for direct IO testing). The man page doesn't give me a lot of hope though, as it appears that there's not guarantess and the OS might keep things in memory if it feels like it anyway. A small reference mentioning it: http://www.qtcentre.org/threads/24733-High-performance-large-file-reading-on-OSX