What

best_effort_cursor_update (wal.rs:838-850) and ShardManifestStore::claim_epoch (manifest.rs:413-454) both perform read-modify-write against the same shard manifest version namespace using PUT-IF-NOT-EXISTS, but only claim_epoch is fail-fast. When a tailer with with_cursor_updates(true) wins the version race, claim_epoch returns the misleading error "another writer may have claimed it" and WalAppender::open fails despite no writer ever having claimed the shard.

Step-by-step proof

Initial manifest at version N, writer_epoch E, wal_entry_position_last_seen = X.

1. Tailer: read_entry(P) succeeds where P > X, spawns tokio::spawn(best_effort_cursor_update(&ms, P)).
2. Tailer task runs manifest_store.read_latest() -> returns manifest at version N.
3. Concurrently, WalAppender::open -> claim_epoch runs read_latest() -> also returns manifest at version N.
4. Tailer task prepares updated with version = N+1, same writer_epoch = E, new wal_entry_position_last_seen = P.
5. claim_epoch prepares new manifest with version = N+1, writer_epoch = E+1.
6. Both target manifest_filename(N+1) via PUT-IF-NOT-EXISTS in ShardManifestStore::write.
7. Tailer wins (line 873-879 of wal.rs: let _ = manifest_store.write(&updated).await; succeeds).
8. claim_epoch's self.write(&new_manifest).await returns AlreadyExists -> wrapped at manifest.rs:447-451 as "Failed to claim shard X (version N+1): another writer may have claimed it".
9. claim_epoch does not retry — manifest.rs:400-402 documents this is intentional ("prevents epoch wars"), but that intent assumes only fencing claimants compete for the version slot.
10. WalAppender::open returns the error to the caller.

Why existing safeguards don't prevent this

• The no-retry rule was designed to prevent epoch wars between competing writers. best_effort_cursor_update writes the same epoch — it is not fencing — but it shares the version namespace with claimants.
• with_cursor_updates(true) is a public, tested API surface (test_wal_tailer_uses_manifest_cursor_hint enables it). Each successful read_entry spawns one fire-and-forget tokio::spawn task, so even a single tailer reading N entries produces up to N concurrent updates contending against any claim_epoch window.
• Both code paths are new in this PR (WalAppender::open and WalTailer::with_cursor_updates), so this is a regression introduced by this PR's interaction with the pre-existing claim_epoch policy.

Impact

Whenever a tailer with cursor updates is active alongside a new writer, WalAppender::open becomes flaky and surfaces a misleading message that points operators at "another writer" when none exists. Not a corruption bug — callers can retry — but it produces silent, hard-to-diagnose failures in the documented happy-path configuration of these new primitives.

Suggested fix

Any of:

1. Have claim_epoch retry on AlreadyExists when the conflicting manifest's writer_epoch is unchanged (i.e. only a cursor update bumped the version), preserving the no-epoch-war guarantee for real claimants.
2. Have best_effort_cursor_update re-read after preparing and skip the write if a newer version landed in the meantime.
3. Store the cursor hint in a separate (non-versioned) namespace so it never contends with claim writes.

The mismatch. This PR adds a Shard Assignment bullet to the Shard Manifest section in docs/src/format/table/mem_wal.md:

shard_spec_id, shard_field_values, and shard_field_string_values record how this shard maps to its shard spec. shard_field_values stores int32-valued shard fields and shard_field_string_values stores UTF-8 shard fields.

But neither shard_field_string_values nor a separate string-valued map exists in the implementation introduced by this same PR.

What the implementation actually contains. The proto in protos/table.proto adds exactly one new field to ShardManifest:

// Computed shard field values as raw Arrow scalar bytes, keyed by shard
// field id. The byte encoding follows Arrow's little-endian convention:
// int32 is 4 LE bytes, utf8 is raw UTF-8 bytes, etc. The receiver looks
// up the result_type from the ShardSpec to interpret each value.
repeated ShardFieldEntry shard_field_entries = 14;

The Rust ShardManifest in rust/lance-index/src/mem_wal.rs has a single map:

pub shard_field_values: HashMap<String, Vec<u8>>,

Each entry holds raw Arrow scalar bytes (int32 LE bytes, raw UTF-8 bytes, etc.), and the receiver looks up the matching ShardField.result_type from the ShardSpec to interpret the bytes. There is no second map.

Proof — grep across the entire repo. shard_field_string_values appears only at docs/src/format/table/mem_wal.md:177. Zero hits in protos/, zero hits in rust/, zero hits in tests. The conversion code in rust/lance-index/src/mem_wal.rs (From<&ShardManifest> for pb::ShardManifest and TryFrom<pb::ShardManifest> for ShardManifest) only maps shard_field_entries ↔ shard_field_values, with no string-typed counterpart. The unit test test_initialize_shard_writes_v1_with_epoch_zero in manifest.rs writes an int32 value into shard_field_values (7i32.to_le_bytes()) and reads it back from the same single map.

Where the confusion likely came from. The PR also rewrites the Shard Snapshot Arrow Schema section, which legitimately splits string vs int columns at the Lance file level using two prefixes:

| shard_field_{field_id} | int32 | one column per int shard field |
| shard_string_field_{field_id} | utf8 | one column per string shard field |

But that is the snapshot Arrow file's column-naming convention, not a ShardManifest protobuf field. The doc bullet conflates the two by inventing a shard_field_string_values field on the manifest that doesn't exist.

Impact. Docs only, no runtime effect. But this section is presented as the authoritative description of the Shard Manifest, and an implementer reading it would either look for a missing protobuf field or implement two maps that do not match the wire format.

Suggested fix. Replace the line at docs/src/format/table/mem_wal.md:177 with something like:

Shard assignment: shard_spec_id and shard_field_values record how this shard maps to its shard spec. shard_field_values is a map from shard field id to the raw Arrow scalar bytes of the computed value; the matching ShardField.result_type in the ShardSpec determines how to interpret each entry (e.g., 4 little-endian bytes for int32, raw UTF-8 bytes for utf8).