Skip to content

test(volume): verify node payload and mock driver publishContext round-trip. - #2094

Merged
Michelle Au (msau42) merged 3 commits into
agent-substrate:mainfrom
lekhoavu2909:ip-acls-v2
Oct 16, 2026
Merged

Michelle Au (msau42) merged 3 commits into
agent-substrate:mainfrom
lekhoavu2909:ip-acls-v2

Conversation

@lekhoavu2909

@lekhoavu2909 Vu Le (lekhoavu2909) commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Some external storage backends (such as Cloud Filestore IP export rules) scope volume access to the Kubernetes worker node currently hosting an actor via CSI ControllerPublishVolume and ControllerUnpublishVolume.
This PR verifies end-to-end NodeId and PublishContext forwarding across the CSI plugin, control plane resume workflow, and functional actor lifecycle (including cross-node migration).

Changes

  • CSI plugin: TestPlugin_AttachVolume / TestPlugin_DetachVolume assert the
    NodeId and VolumeId sent on publish and unpublish, including the Unimplemented path.

  • Mock drivers: MockVolumePlugin and the functional test's attachFailVolumePlugin
    return a publish context naming the target node and volume, so existing lifecycle tests
    now carry a non-empty publishContext through Restore.

  • Lifecycle test: builds on the enhance tests to validate volume lifecycle for actor state changes #1581 pause lifecycle test, renamed
    TestPauseActor_VolumeLifecycle_DetachResumeAndMigrate. It checks the Restore request's
    publishContext after each resume, adds a migration leg (suspend → worker on node2 →
    resume → attach to node2), and verifies detach, volume deletion, and actor removal on delete.
    lastRestoreVolumePublishContext replaces the per-test lookup of the Restore request.

  • Unit test: TestEnsureVolumesAttached_ReturnsPublishContext also asserts that
    AttachVolume is called once, for the mounted volume, with the worker's node.

  • Tests pass

@lekhoavu2909
Vu Le (lekhoavu2909) marked this pull request as draft October 1, 2026 22:36
@lekhoavu2909
Vu Le (lekhoavu2909) marked this pull request as ready for review October 1, 2026 22:55
@lekhoavu2909
Vu Le (lekhoavu2909) marked this pull request as draft October 1, 2026 22:55
@lekhoavu2909
Vu Le (lekhoavu2909) marked this pull request as ready for review October 1, 2026 22:59
@lekhoavu2909
Vu Le (lekhoavu2909) marked this pull request as draft October 1, 2026 23:00
@lekhoavu2909

Copy link
Copy Markdown
Collaborator Author

hajiler stacking this on your #1578 branch as discussed, will drop out once #1578 merges. For now I will keep it in Draft. I don't have triage access on the repo, so I couldn't request you as reviewer — feel free to add yourself. Thanks!

@lekhoavu2909 Vu Le (lekhoavu2909) changed the title test(volume), doc(security): verify node payload and mock driver publishContext round-trip, documentation. test(volume): verify node payload and mock driver publishContext round-trip. Oct 6, 2026
@lekhoavu2909 Vu Le (lekhoavu2909) changed the title test(volume): verify node payload and mock driver publishContext round-trip. volume: verify node-scoped CSI publish and publishContext lifecycle Oct 6, 2026
@lekhoavu2909
Vu Le (lekhoavu2909) marked this pull request as ready for review October 6, 2026 20:51
@lekhoavu2909 Vu Le (lekhoavu2909) changed the title volume: verify node-scoped CSI publish and publishContext lifecycle volume: verify node-scoped CSI publish and publishContext lifecycle Oct 6, 2026
@hajiler

hajiler commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Thanks for adding publishContext coverage. Before this goes in, I'd suggest building on the lifecycle tests Sneha added in #1581 rather than adding parallel ones, since most of the new setup here already exists there.

Functional test

TestExternalVolume_PublishContextLifecycle replays steps 1–3 of TestPauseActor_VolumeLifecycle_DetachAndResumeAttach verbatim (same template, same worker on node1, same attachFailVolumePlugin, same resume/pause/detach assertions). The only new ground is the migration leg to node2 and the restore-request assertions. Because this PR changes attachFailVolumePlugin to return a publish context, the #1581 test is already driving publishContext through two resumes; it just never asserts it.

Proposal:

  • Add a small helper on top of the existing tc.fakeAtelet.lastRestoreRequest() that returns the publish context for a named volume. That replaces the two copies of the lock/walk/find block.
  • In the enhance tests to validate volume lifecycle for actor state changes #1581 pause lifecycle test, assert the publish context after each resume, then append the migration leg (suspend → node2 worker → resume → assert node2 attach and node2 in the restore request → suspend → delete).
  • Add a plugin.deleted assertion after DeleteActor. The fake already records it, and the current test comment says delete is covered but nothing checks it. (TestResumeActor_VolumeAttachFailure_DeleteActor from enhance tests to validate volume lifecycle for actor state changes #1581 does assert this, so it's a one-liner here, not a gap.)
  • Drop TestExternalVolume_PublishContextLifecycle and rename the extended test to reflect its scope.

That gives one lifecycle test with strictly more coverage than either and no duplicated setup.

Unit test

TestEnsureVolumesAttached_ReturnsPublishContext already exists on main. ensureVolumesAttached only reads worker.GetNodeName() and has no branch that depends on a prior assignment, so TestEnsureVolumesAttached_NodeMigration exercises the same path with a different node string. I'd fold the req.Node assertion into the existing test (have publishContextVolumePlugin record the request) and drop nodeTrackingPlugin and the second test. Real migration coverage belongs in the functional test above, where the scheduler actually picks a different node.

@lekhoavu2909 Vu Le (lekhoavu2909) changed the title volume: verify node-scoped CSI publish and publishContext lifecycle volume: volume: record attached node and detach from it after crash or worker loss Oct 7, 2026
@lekhoavu2909 Vu Le (lekhoavu2909) self-assigned this Oct 7, 2026
@lekhoavu2909

Copy link
Copy Markdown
Collaborator Author

Address hajiler 's comment above

@lekhoavu2909 Vu Le (lekhoavu2909) changed the title volume: volume: record attached node and detach from it after crash or worker loss volume: record attached node and detach from it after crash or worker loss Oct 7, 2026
@lekhoavu2909
Vu Le (lekhoavu2909) force-pushed the ip-acls-v2 branch 3 times, most recently from 15393f2 to 2c05799 Compare October 7, 2026 19:37
@lekhoavu2909
Vu Le (lekhoavu2909) force-pushed the ip-acls-v2 branch 2 times, most recently from e4ea563 to b9eba6b Compare October 7, 2026 22:45
…d-trip

Assert that ControllerPublishVolume and ControllerUnpublishVolume receive the target NodeId in CSI plugin unit tests. Update MockVolumePlugin.AttachVolume to return realistic publish context metadata reflecting the target node.

Add TestExternalVolume_PublishContextLifecycle to verify complete volume attachment, node migration, and clean detachment across resume, pause, suspend, and delete, asserting on fakeAtelet wire delivery.

Add TestEnsureVolumesAttached_NodeMigration verifying in-memory publish context delivery on worker node migration.
@lekhoavu2909 Vu Le (lekhoavu2909) changed the title volume: record attached node and detach from it after crash or worker loss test(volume): verify node payload and mock driver publishContext round-trip. Oct 13, 2026
@hajiler

hajiler commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

/LGTM

}

// 5. Clean up by suspending and deleting the actor.
// 5. Suspend to detach from node1 and clear local snapshot, then migrate to node2.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 should-fix 🟡 – The merge from main reverted the rename and doc comment at cmd/ateapi/internal/controlapi/functionaltest/actor_test.go:2692-2704. The function is still named TestPauseActor_VolumeLifecycle_DetachAndResumeAttach, and its header comment still describes the old 5-step cleanup instead of the 6-step migration and delete flow tested here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed

Comment on lines +333 to +341
driver.controllerUnpublishVolumeFunc = func(ctx context.Context, req *csi.ControllerUnpublishVolumeRequest) (*csi.ControllerUnpublishVolumeResponse, error) {
if req.GetNodeId() != "node-1" {
t.Errorf("ControllerUnpublishVolume received NodeId %q, want %q", req.GetNodeId(), "node-1")
}
if req.GetVolumeId() != "test-vol" {
t.Errorf("ControllerUnpublishVolume received VolumeId %q, want %q", req.GetVolumeId(), "test-vol")
}
return &csi.ControllerUnpublishVolumeResponse{}, nil
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 should-fix 🟡 – Unlike AttachVolume (where the test checks resp.PublishContext), DetachVolume returns no response payload. If DetachVolume skips the RPC, controllerUnpublishVolumeFunc never runs and both checks in this test pass silently. Record a called counter and assert it ran after each DetachVolume call.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed

@msau42
Michelle Au (msau42) added this pull request to the merge queue Oct 16, 2026
Merged via the queue into agent-substrate:main with commit 95b39f3 Oct 16, 2026
15 checks passed
Huy Pham (hdp617) pushed a commit to hdp617/substrate that referenced this pull request Oct 20, 2026
…etach on crash (agent-substrate#2611)

Some storage backends (e.g. Cloud Filestore IP export rules) restrict a
volume to the node it is published to via CSI `ControllerPublishVolume`
/ `ControllerUnpublishVolume`, so every publish must be matched by an
unpublish to the same node. Since agent-substrate#2222 , detach targets
`ActorStatus.assigned_node`, which survives crashes and worker loss.
This PR adds checks before attach and test coverage for the paths where
detach used to be skipped.
Follows agent-substrate#2094, which covers the publish side.
- *ateapi: check mounted volumes are attachable before publishing*
- *test(ateapi): cover volume detach via assigned_node after crash or
worker loss*

#### Changes
- Worker without a node name: `assignWorkerAttempt` rejects such a
worker with when the actor mounts external volumes. Previously the
resume failed in `ensureVolumesAttached`, after the worker was booked.
- Volume checks before attach: `ensureVolumesAttached` checks that every
mounted volume is `STATUS_CREATED` and has a `storage_volume_id` before
any attach call, so one bad volume can't leave earlier volumes
published.
- Unit tests, functional tests
- E2E: `assigned_node` equals the worker's `node_name` while RUNNING and
is cleared after suspend and revert. The OSS NFS driver doesn't
implement `ControllerPublishVolume`.

- [x] Tests pass
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants