Repository navigation
fix(filetxn): read directory identity through per-OS variants - #386
barbatjuan wants to merge 6 commits into
Conversation
internal/filetxn type-asserted info.Sys().(*syscall.Stat_t) to read the device and inode of accepted directories, which does not compile on Windows. Move the read behind directoryIdentity: the unix variant keeps Stat_t, and the Windows variant opens the directory without following reparse points and reads the volume serial number and file index through GetFileInformationByHandle, refusing a handle that no longer names the caller's Lstat result. The restart_test.go helpers use the same function. The test helper that replaces a directory now pins the original identity before the rename, because Windows loads an Lstat identity lazily from the path. Fixes refactor-ia#370
Fleet review (glm)Lens: requirements + adversarial. The diff satisfies issue #370's expected direction: the Stat_t-based identity read is behind a build-tagged directoryIdentity function, the unix variant preserves the previous behavior exactly (same Dev/Ino read, same Ino==0 refusal), and the Windows variant opens the directory with FILE_FLAG_BACKUP_SEMANTICS|FILE_FLAG_OPEN_REPARSE_POINT and refuses non-directories, reparse points, zero file index, and any handle whose stat does not pass os.SameFile against the caller's Lstat result, so divergence between path and Lstat fails closed. Verified at head 62ad16b: the syscall import in directories.go is still used (EINVAL/ENOTSUP at line 414), so the import-hunk absence in the diff is not a build break. The expected-files count in tests/test_community_policy.py is updated 406->409, exactly the three new files. No blocking findings found: every Windows-only edge I attacked (replaced path, stale Lstat, reparse point, long-path open failure) fails closed to an 'identity unavailable'/'directory changed' error rather than wrong behavior. Three non-blocking items: the ReFS 128-bit file-ID limitation the author already disclosed (the persisted AcceptedDirectory shape is uint64, so a low-64-bit collision on ReFS could make a replacement directory look accepted), the scope narrowing versus the test-pass expectation recorded in #370's plan, and long paths (>260 chars) failing the Windows identity open without an extended-length prefix. Windows filesystem behavior is explicitly not verified yet, so none of these gate this build fix. Suggestion (2)
Question (1)
Verdict: ready to merge |
Fleet review (deepseek)Lens: evidence + test design. The diff does what issue #370 asks for: the directory-identity read moves behind build-tagged variants (identity_unix.go keeps the previous Stat_t device/inode read including the Ino == 0 refusal, identity_windows.go uses GetFileInformationByHandle), the four call sites stop referencing syscall.Stat_t, the persisted AcceptedDirectory shape and the unix behavior are unchanged, and docs/verification.md plus the publication inventory are updated consistently. The build claim on windows/amd64 is backed by a stated before/after (undefined: syscall.Stat_t at four sites, then go build ./... passing on win/linux/darwin, go vet ./internal/filetxn/, and the full Linux suite on a fresh clone). What the evidence does not cover: none of the new Windows refusal branches (non-directory, reparse point, zero index, os.SameFile mismatch) is exercised by any test, so those guards can be removed without a red test, and the stale-lstat subtest passes on ok == false, which cannot distinguish a correct drift refusal from an unrelated identity-read failure. No blocking finding: no acceptance criterion of #370 is contradicted and I found no concrete failure scenario on a supported platform. The PR's own scope question (build fix only vs. the remaining Windows fsync/mode work) is answered below as a question, not a gate. Suggestion (3)
Question (2)
Verdict: ready to merge |
|
Thanks for the review. Taking the three items in order: ReFS file IDs. Agreed. Fixing it here would need either a manifest identity format change or refusing ReFS (which would include Dev Drives), so I put it in the follow-up issue as a design point: #389. Long paths. I couldn't reproduce this one. On Windows 11 with go1.26.5 and Scope split. Recorded in #389 as evidence for the Phase B mode/backup semantics design (scope item 2 in |
Address the fleet review on refactor-ia#386: - identity_windows_test.go covers a regular file, a junction, a directory replaced by a junction or by another directory after Lstat, and a path longer than MAX_PATH. - The stale-Lstat subtest now states the expected result per OS: a refusal on Windows, where the identity is read from the path, and the original identity on unix, where it is part of the Lstat result. A fresh Lstat is pinned first so a refusal cannot come from an unreadable path. - identity_other.go refuses on platforms without an identity source, so js/wasm and wasip1 compile and fail closed. Refs refactor-ia#370
|
Thanks, these were worth fixing. I pushed 4e3e08c. Windows refusal branches.
The single-guard rows stay green because those checks overlap: removing one of them changes no behavior, since another check rejects the same input. Every removal that changes behavior turns a test red. Stale-Lstat subtest. It now pins a fresh Lstat first and states the expected result per OS: a refusal on Windows, where the identity is read from the path, and the original identity on unix, where it comes from the Lstat result itself (so it is no longer vacuous there). Other platforms. Long paths. Covered by the new Windows gate. For this PR: Verification: Windows 11 go1.26.5 (identity tests and the mutation runs above), and Linux ( |
Fleet review (glm)Lens: requirements + adversarial. The diff satisfies issue #370's expected direction: the syscall.Stat_t reads in acceptCreatedDirectories and restartDirectoryMatchesAccepted (plus the two restart_test.go call sites) now go through a build-tagged directoryIdentity function, with identity_unix.go preserving the exact previous behavior (same Stat_t read, same Ino == 0 refusal), identity_windows.go reading volume serial + file index through GetFileInformationByHandle, and identity_other.go failing closed on remaining platforms. Adversarially, the Windows variant is fail-closed where it matters: it refuses non-directories, reparse points (so junctions are rejected even though the handle is opened with FILE_FLAG_OPEN_REPARSE_POINT), a zero index, and any case where os.SameFile(info, opened) disagrees; in the restart paths a TOCTOU replacement between Lstat and the handle open still ends in a mismatch against the persisted accepted identity, so a replacement is refused rather than accepted. The stale-Lstat tests pin the lazily loaded Windows identity the same way production callers do via os.SameFile, and the 406-to-411 inventory update matches the five new files. The PR also honestly discloses that #370's plan promised go test ./internal/filetxn/... on Windows while this PR only delivers the build fix and the identity tests; the author asked the maintainer how to slice it. That scope gap is the one thing to pin down before merge so it does not silently disappear. No blocking findings. Suggestion (1)
Question (1)
Verdict: ready to merge |
Fleet review (deepseek)Lens: evidence + test design. The extraction is sound: identity_unix.go is a literal move of the previous Stat_t read (same Dev/Ino and the Ino == 0 refusal), identity_other.go now completes the platform set by failing closed, both call sites in directories.go and restart.go pass the same path they Lstat'ed, and the persisted AcceptedDirectory shape is unchanged. Two earlier gaps are closed at this head: identity_windows_test.go now exercises the regular-file, junction, replaced-after-lstat and long-path refusals, and the stale-lstat subtest pins a fresh Lstat before the stale call, so ok == false is attributable to the mismatch. Two problems remain on the Windows side. First, the new identity read converts the caller's path directly with syscall.UTF16PtrFromString and syscall.CreateFile, so it has none of the long-path handling the os package applies to every other operation in the transaction; the Blocking (1)
Suggestion (1)
Question (1)
Verdict: not ready |
Fleet review (qwen38)Lens: evidence. Issue #370's acceptance criterion is met with real evidence: the identity read is extracted behind build-tagged directoryIdentity variants, the unix variant preserves the previous Stat_t read exactly (including the Ino == 0 refusal), both production call sites (directories.go acceptCreatedDirectories, restart.go restartDirectoryMatchesAccepted) and both restart_test.go helpers route through it, identity_other.go fails closed elsewhere, and the PR reports go build/vet plus GOOS=linux|darwin|windows cross-builds passing and TestDirectoryIdentity passing on native Windows 11. The publication inventory bump 406->411 matches the five new files exactly, and the persisted AcceptedDirectory shape is unchanged. No blocking defect found: the windows variant refuses non-directories, reparse points, and a zero file index, and guards the handle against the caller's Lstat with os.SameFile. Remaining findings are evidence gaps: the Windows-only refusal suite TestDirectoryIdentityWindowsRefusals has no reported run result, its long-path subtest depends on an unstated host setting, and the disclosed ReFS 128-bit file-ID truncation is recorded only in the PR body. Suggestion (1)
Question (2)
Verdict: ready to merge |
Fleet review (glmflash)Lens: adversarial. The per-OS split is structurally sound: unix behavior is byte-for-byte the old Stat_t read, identity_other.go fails closed, and the Windows variant refuses non-directories, reparse points, zero indexes, and identity mismatches, so every adversarial path I attacked (junction swap after Lstat, directory replacement between Lstat and CreateFile, unpinned stale info in classifyRestartDirectories/rollbackRestartDirectories) ends in a refusal or in an identity that then fails the accepted-Device/Inode comparison. The accepted-identity property itself survives the TOCTOU windows because the identity always comes from the handle and is compared against the persisted accepted values. One blocking problem: the new Windows test suite asserts long-path success that a direct syscall.CreateFile cannot deliver on stock Windows, and the test evidence in the body is consistent with those Windows-only subtests never having passed. Blocking (1)
Question (2)
Verdict: not ready |
- identity_windows.go and docs/verification.md state that directory identity on ReFS volumes is not guaranteed unique, pointing to refactor-ia#389, so the limit lives in the repository and not only in the PR. - identity_windows_test.go explains why the MAX_PATH subtest holds on every supported host: the Go runtime opts into long-path support at startup, and the os package then adds no \?\ prefix either. - identity_test.go is constrained to unix || windows, the platforms with an identity source; elsewhere identity_other.go refuses by design, so the portable assertions do not apply. Refs refactor-ia#370
Fleet review (mimo)Lens: requirements. Issue #370's expected direction - extract the directory-identity read behind a build-tagged interface (unix Stat_t variant, Windows GetFileInformationByHandle variant) so ./cmd/cortex builds on windows/amd64 - is satisfied at head 4e3e08c: both production call sites (directories.go acceptCreatedDirectories, restart.go restartDirectoryMatchesAccepted) and both restart_test.go helpers route through directoryIdentity, a read of the head files shows only identity_unix.go still references syscall.Stat_t, exactly one variant compiles per GOOS, directories.go keeps its syscall import for the EINVAL/ENOTSUP sync check at line 414 (so the missing import hunk is not a build break), the unix read is byte-identical to the previous one (same Dev/Ino read and Ino==0 refusal, so unix behavior is unchanged as the PR claims), the persisted AcceptedDirectory shape is unchanged, and the publication inventory moves 406->411 for exactly the five new files. I do not share the first-line blocking verdict on the >MAX_PATH subtest: the blocking scenarios assume the os package adds long-path handling that syscall.CreateFile lacks, but on any Windows build >=15063 running a Go binary (including the named verification host, Windows 11 go1.26.5) the Go runtime sets the PEB IsLongPathAwareProcess bit at startup and os.fixLongPath then returns the path unmodified, so os.Lstat/os.MkdirAll and the new identity read pass raw long paths identically - a failure would hit the subtest's own os.MkdirAll first, not directoryIdentity, and the author reports the subtest passing with LongPathsEnabled=0. Four non-blocking findings follow: the still-undocumented host/OS precondition of that subtest, the Windows accept-baseline laziness question, a build-tag gap in identity_test.go, and the stale test evidence in the PR body for this head. Suggestion (1)
Question (3)
Verdict: ready to merge |
directoryIdentity passed the caller's path straight to CreateFile. On Windows builds before 10.0.15063 the Go runtime cannot opt the process into long-path support, and the os package then adds the \?\ prefix on its own calls, so a long target would reach every os operation of the transaction and fail only in the identity read. extendedLengthPath now adds the prefix to every long clean absolute path (and the \?\UNC\ form for UNC paths), as os does, so the identity read no longer depends on that opt-in or on the LongPathsEnabled registry value. TestExtendedLengthPath pins the conversion, and the MAX_PATH subtest now exercises the prefixed open on any host. Refs refactor-ia#370
|
Thanks for this round. I pushed 948448b and 1011b41, and updated the PR body with the current-head evidence. Long paths (deepseek and glmflash, blocking; qwen38 and mimo). Fixed in code, so the test no longer depends on the host. On the premise: mimo's reading matches the Go sources. On Windows 10 1703 and later, Test evidence and subtest counts (glmflash, mimo, qwen38). The Swap between creation and acceptance on Windows (glmflash, mimo). Agreed. ReFS limitation (glm, qwen38). Recorded at the index derivation in identity_test.go on refuse-by-design platforms (deepseek, mimo). Now Windows gate (deepseek). Verification: Windows 11 as above; Linux in a |
Fleet review (glm)Lens: requirements + adversarial. The PR satisfies issue #370's core requirement: internal/filetxn now builds on windows/amd64 by reading directory identity behind build-tagged variants (unix Stat_t read unchanged and bit-identical in behavior, Windows GetFileInformationByHandle, other platforms fail closed). The Windows variant is carefully hardened: non-directories refused, reparse points refused (so junctions are rejected even though the handle is opened with FILE_FLAG_OPEN_REPARSE_POINT), zero file index refused, and an os.SameFile cross-check between the caller's Lstat result and the opened handle; ?\ prefixing keeps long paths independent of the runtime long-path opt-in. New tests pin stability, sibling distinctness, replacement distinctness, stale-Lstat refusal, junction refusal, and >MAX_PATH behavior, and the guard-mutation evidence shows the tests actually bite. The docs now disclose the ReFS 64-bit-index limitation and the not-yet-verified Windows transaction behavior. No blocking defects found; three clarifications below, two of which the author already documents and defers. Suggestion (1)
Question (2)
Verdict: ready to merge |
Fleet review (deepseek)Lens: evidence + test design. I re-reviewed from scratch at head 1011b41 (my previous review of 4e3e08c blocked on the missing Windows long-path handling; Suggestion (2)
Verdict: ready to merge |
Address the fleet review on refactor-ia#386: - extendedLengthPath now passes device paths (\.\) through unchanged, as the os package does; prefixing them would change their meaning. - TestExtendedLengthPath pins the 247/248-character threshold and a long device path. - TestDirectoryIdentityOpensLongPathsExtended opens a long directory whose name ends with a dot, which only the \?\ form preserves, so the test fails if directoryIdentity stops using the prefix even on hosts where the runtime long-path opt-in makes plain long paths work. - TestRestartDirectoryMatchesAcceptedWindows drives the production restart check with the resolved target on Windows. Refs refactor-ia#370
|
Thanks. I pushed adaec91. extendedLengthPath boundary and device paths (deepseek). Fixed. Device paths ( Production wiring on Windows (deepseek). Added two tests:
Mutation check on Windows, each change applied alone with the same tests run:
Inventory duplication (glm). Agreed that the repeated asserts are fragile; that predates this PR. I can open a small cleanup issue if you want one. Ownership of the lazy accept-time identity and the #370 scope (glm). #389 owns the accept-time identity, recorded in its comment with the candidate fix. Whether #370 closes on this PR or stays open until the full Windows suite passes is your call. Verification at adaec91: Windows 11 (identity, prefix, and restart tests; mutations as above); |
Fleet review (deepseek)Lens: evidence + test design. Reviewed at head adaec91; my earlier review of 1011b41 asked for the extendedLengthPath boundary and device-path cases, and for coverage of the production wiring. Both are now answered: TestExtendedLengthPath carries the 247/248 threshold, a long UNC path, a long device path, an already-extended path, a long relative path and a long unclean path, and TestRestartDirectoryMatchesAcceptedWindows drives the restart check through a resolved target, including the root-relative mismatch and the replaced directory. The build fix itself is unchanged in shape and still correct: the syscall.Stat_t read sits behind directoryIdentity with unix, windows and fail-closed variants, the unix variant is the previous expression including the Ino == 0 refusal, the Windows variant opens with FILE_FLAG_BACKUP_SEMANTICS|FILE_FLAG_OPEN_REPARSE_POINT, refuses non-directories, reparse points and a zero index, and uses os.SameFile to turn a path-based read into a stale-Lstat refusal; the persisted AcceptedDirectory shape is untouched. The inventory arithmetic in tests/test_community_policy.py is honest: five new files against 406 -> 411, and the new paths are added to EXPECTED_FILES. No blocking defect. What remains is one production call site that the declared Windows gate still does not execute, and a pre-existing duplicated assertion block this PR renumbers but does not clean up. Suggestion (2)
Verdict: ready to merge |
Fleet review (glm)Lens: requirements + adversarial. The PR satisfies issue #370's core requirement: internal/filetxn now builds on windows/amd64 by reading directory identity behind build-tagged variants (unix Stat_t read unchanged and bit-identical in behavior, Windows GetFileInformationByHandle, other platforms fail closed). The Windows variant is carefully hardened: non-directories refused, reparse points refused (so junctions are rejected even though the handle is opened with FILE_FLAG_OPEN_REPARSE_POINT), zero file index refused, and an os.SameFile cross-check between the caller's Lstat result and the opened handle; ?\ prefixing keeps long paths independent of the runtime long-path opt-in. New tests pin stability, sibling distinctness, replacement distinctness, stale-Lstat refusal, junction refusal, and >MAX_PATH behavior, and the guard-mutation evidence shows the tests actually bite. The docs now disclose the ReFS 64-bit-index limitation and the not-yet-verified Windows transaction behavior. No blocking defects found; three clarifications below, two of which the author already documents and defers. Suggestion (1)
Question (2)
Verdict: ready to merge |
Pull Request
Linked Issue
Fixes #370
PR Type
type:featureFeaturetype:bugBug fixtype:docsDocumentationtype:refactorRefactoringtype:choreMaintenancetype:breaking-changeBreaking changeSummary
internal/filetxnread accepted-directory identity throughinfo.Sys().(*syscall.Stat_t), which does not exist on Windows, so./cmd/cortexdid not build on windows/amd64. The identity read now sits behind a build-taggeddirectoryIdentityfunction, and the unix behavior is unchanged.Changes
identity_unix.go(//go:build unix): sameStat_tdevice and inode read as before, including theIno == 0refusal.identity_windows.go: opens the directory withFILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINTand reads the volume serial number and the 64-bit file index throughGetFileInformationByHandle. It refuses non-directories, reparse points, a zero index, and a handle thatos.SameFilesays is not the caller's Lstat result. Long clean absolute paths are opened with the\\?\(or\\?\UNC\) prefix, as the os package does, so the read does not depend on the runtime long-path opt-in or onLongPathsEnabled.identity_other.go(!unix && !windows): refuses, so js/wasm and wasip1 compile and fail closed.acceptCreatedDirectoriesandrestartDirectoryMatchesAcceptedcalldirectoryIdentity(target, info). The persistedAcceptedDirectoryshape is unchanged.restart_test.gohelpers use the same function.replaceDirectoryWithDistinctIdentity(test helper) pins the original identity before the rename. On Windows,os.SameFileloads an Lstat identity lazily from the path, so the helper otherwise compared the replacement with itself. On unix the extra check is a no-op.identity_test.go(//go:build unix || windows): the identity is stable for the same directory, distinct for siblings, distinct after replacement, and, for a stale Lstat result, refused on Windows and the original identity on unix.identity_windows_test.go: refusal of a regular file, a junction, and a directory replaced by a junction or by another directory after Lstat; a path longer than MAX_PATH; andTestExtendedLengthPathfor the prefix conversion.docs/verification.md: Windows still has no support, but it now builds from source; directory identity on ReFS is not guaranteed unique (Feature: record Windows filetxn semantics decisions for the Phase B design #389).Focused Tests
Evidence for head adaec91. Windows host: Windows 11, go1.26.5 windows/amd64, default
LongPathsEnabled=0.go build ./...failed withundefined: syscall.Stat_tatdirectories.go:258andrestart.go:406; the test build also failed atrestart_test.go:418and:575.go build ./...andgo vet ./internal/filetxn/pass on windows/amd64, andgo vet ./internal/filetxn/passes for js/wasm, wasip1/wasm, linux/amd64, and darwin/arm64.go test -count=1 -v -run 'TestDirectoryIdentity|TestExtendedLengthPath|TestRestartDirectoryMatchesAcceptedWindows' ./internal/filetxn/on Windows passesTestDirectoryIdentity(4 subtests),TestDirectoryIdentityWindowsRefusals(5),TestDirectoryIdentityOpensLongPathsExtended,TestExtendedLengthPath(247/248 boundary, long, UNC, device, already extended, relative, unclean), andTestRestartDirectoryMatchesAcceptedWindows.<to<=, or 248 to 260), not applying the prefix indirectoryIdentity, and removing theos.SameFilecheck each turn at least one test red. Disabling only one ofisRealDirectory, the directory attribute, or the reparse-point attribute stays green, because those checks overlap; disabling them in pairs turns tests red.golang:1.26container, go1.26.10 linux/amd64, LF sources, non-root uid 1000): gofmt clean, andgo vet ./...andgo test -count=1 ./...pass with no failures.go build ./...plus the identity tests above.go vet ./...and the fullgo test ./internal/filetxn/on Windows are tracked in Feature: record Windows filetxn semantics decisions for the Phase B design #389.Runtime Harness
N/A. This changes how the build reads directory identity; install, update, and uninstall do not change on supported platforms.
Rollback
Reverting the branch's commits restores the previous
Stat_treads. The manifest format does not change.Chain Dependencies / Out of Scope
The rest of
go test ./internal/filetxn/does not pass on Windows yet. The failures are pre-existing Windows semantics that are not related to directory identity, and they belong to the #332 Phase B mode/backup design:os.Open(dir).Sync()fails on Windows, which causes most of the failures (sync directory parent failed).0600/0640, and Windows reports0666.internal/qapi/profile_test.goalso usessyscall.Mkfifo, sogo vet ./...on Windows still fails in that test package.A Windows ReFS volume can use 128-bit file IDs, and
GetFileInformationByHandlereturns the 64-bit index only. NTFS is fully covered.On Windows, the Lstat result captured when a directory is created loads its identity lazily, at the first
os.SameFile. A directory swapped between creation and acceptance is therefore compared by path, not by the object actually created.Scope. My plan in #370 said
go test ./internal/filetxn/...would pass on Windows. This PR delivers the build fix and the identity tests; everything listed in this section is tracked in #389 as input for the Phase B mode/backup semantics design.Checklist
status:approved.type:*label is applied.