The Issue

We found this during the audit phase of a new software project. LizardFS gives inconsistent results when walking the mounted filesystem. Running find ./ returns mostly correct files, but also a selection of duplicate files (the exact same path/file two or more times), as well a selection of files that don't actually exist. From what we can tell, all files stored on the filesystem do exist, but occasionally filenames are being returned with incorrect paths. The incorrect paths also exist, but is not where the file in question is stored.

This has been tested and recreated on 3.11.3 and 3.12.0 on Ubuntu Xenial in multiple datacenters (i.e. separate hardware).

Recreating

Setup a new 3.12.0 LizardFS environment with one or more chunk servers.

On the master server:

• Mount the LizardFS root (e.g. /mnt/lfs).
• Write ~3 million files to the filesystem distributed in paths 3 or more levels deep (e.g. ./1/2/3/file1 etc).
• When the files are in place, run concurrent find commands (or any other script etc that walks the filesystem), sorted out to a file, e.g.
  find /mnt/lfs | sort > /var/tmp/find.out.1
find /mnt/lfs | sort > /var/tmp/find.out.2
• Then compare the output:
  comm -3 /var/tmp/find.out.1 /var/tmp/find.out.2

In most cases, if the find commands are run concurrently, you will get a handful of differences. In a sample size of 3 million, we're seeing 50 - 200 incorrect items.