Pipeline Attempt on 598993155 for 01f44a1

https://gitlab.com/MatrixAI/open-source/js-encryptedfs/-/pipelines/598993155

Legend

Pipeline Attempt on 599061810 for ad2f42e

https://gitlab.com/MatrixAI/open-source/js-encryptedfs/-/pipelines/599061810

Ok there's another concurrency non-determinism. This time caused by a slower write while there is a write stream. It's quite confusing. But this is my theory.

We have a write stream that writes 10 chunks of AAAAA. Therefore if it was writing by itself, we would see something like:

AAAAA AAAAA AAAAA AAAAA ... etc

Now we also have an atomic call to a duplicated FD but that which starts at position 0. It writes 25 characters of B.

The test currently expects the 25 characters to be written atomically. Therefore the test thinks the end result is either one of these:

AAAAA AAAAA ... etc
OR
B*25 A*25

But the problem with this expectation is that, the former is true if the B write occurred first, and then A writes started. The latter can occur when A writes complete, then B write starts.

But if B write starts in the middle of A writes. Then we would get a mixed output.

This is because:

1. Let's suppose that after the first A chunk, the B writes were written. We would have B*25
2. On the second A chunk, it is still writing from the first chunks ending position.
3. This results in BBBBB as the first 5 bytes, but subsequent bytes will all be A.

Hence is why we have:

    Expected: "BBBBBBBBBBBBBBBBBBBBBBBBBAAAAAAAAAAAAAAAAAAAAAAAAA"
    Received: "BBBBBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"

Now this could also change if the B writes occurs after the 2nd chunk. In that case we should use B*10 and then all As.

The reason why this is quite non-deterministic is because there are 2 levels of asynchronicity here. The ones involving await, but also lazy drainage events, where just because we do writeStream.write(dataA), there's no guarantee that the writes have written to the disk. Of course in our in-memory FS, we could sequence it properly with a callback to wait for when the chunk is in fact written.

But the end result is that the B writes can occur at any point in time between the atomic A writes. This is the same problem as the previous problem, but it wasn't as clear, because B writes starting from position 0, whereas A writes are in sequence.

@tegefaulkes

So the solution could be to strictly control the concurrency, but that doesn't allow our test to be robust. A robust test here should instead have a predicate that allows any of the possible permutations of the output. That being of all As, or any number of chunks of B based on the chunk size of A, up to 5 chunks (the length of B divided by the chunk size of A).

This has been resolved with a more robust regex check. I suspect more of these bugs are lurking in the tests.

Pipeline Attempt on 599151439 for a70d4c4

https://gitlab.com/MatrixAI/open-source/js-encryptedfs/-/pipelines/599151439

Pipeline Attempt on 599152359 for 92efef0

https://gitlab.com/MatrixAI/open-source/js-encryptedfs/-/pipelines/599152359

Pipeline Succeeded on 599152359 for 92efef0

https://gitlab.com/MatrixAI/open-source/js-encryptedfs/-/pipelines/599152359