Repository navigation
ateapi: one bad actor row fails the whole ListActors response, with no indication which actor caused it #1401
Description
Expected Behavior
A single actor whose serialized form cannot be decoded by a caller should cost that caller that
actor — an error naming it, a skipped row, a partial response with a warning. Losing the view of
every actor in the atespace is disproportionate to the fault.
Actual Behavior
ListActors returns a single message containing every actor, so any decode failure on any row fails
the entire response for every caller. Observed as:
Error: failed to list actors: rpc error: code = Internal desc = grpc:
failed to unmarshal the received message: string field contains invalid UTF-8
There is nothing in the error identifying the actor, so recovery starts with an unbounded search
across the atespace. In the case that produced this (see the schema issue above), the trigger was
one running actor among many, and the only way to find it was to decode the wire bytes by hand.
Steps to Reproduce the Problem
- Produce any actor whose row a given client cannot decode. The reproducible route today is the tag
reuse described in the linked issue: an old client plus one RUNNING actor with a
worker_assignmentof 128 bytes or more. - Call
ListActorsfor that atespace, e.g.kubectl-ate get actors -a <atespace>. - Observe that the response fails entirely, that no other actor is listed, and that nothing in the
error names the responsible actor.
Specifications
- Version:
mainatea3bdc32(2026-09-02);rpc ListActorsatateapi.proto:102, response
ListActorsResponseat:1588. - Platform: any.
Notes
Worth fixing on its own merits regardless of the schema issue: any future decode-affecting change,
on any field, inherits this blast radius. Pagination already exists on this RPC, so per-page partial
success plus a per-row error list is a natural shape.
Related to #1396
Activity
cloudonly commentedon Sep 29, 2026
Looked into where this actually breaks; there are two layers with different owners.
Layer 1 — the trigger in the report is client-side, and no server change fixes it. Actors are stored as proto bytes and the server decodes rows with
DiscardUnknown(atepg.go:275), which succeeds for anything a newer schema wrote — in the current descriptor the fields involved are messages, not strings, so the server never produces an invalid string. The failure happens in old clients: descriptors generated before #737 (2026-08-04) read tags 5/6/7 asstring(ateom_pod_*), and aworker_assignmentof 128+ bytes makes the length varint start with0x82— not a valid UTF-8 start byte — so proto3 unmarshal rejects the whole response. (Between #737 and #1025 only tag 6 was stillstring; those clients misread silently rather than failing.) Rebuilding the client is the only fix for this layer.Layer 2 — server-visible bad rows fail the page unnamed, and the name is in scope. When a stored row fails to decode under the current descriptor,
actor.go:208and:256fail the whole page withunmarshaling actorand no name — same shape inactor_template.go:189,atespace.go:102,tag.go:94/140, andworker_assignment.go:247.Two-step proposal, if there's interest:
ListActorsResponsegainsrepeated RowError { ObjectRef actor = 1; string error = 2; }; the store skips rows that fail to decode and reports them;kubectl-ateprints them. One subtlety worth deciding up front:has-morecurrently followslen(decoded) > page_size(actor.go:218/266) — with skipped rows it has to follow the number of rows scanned, or a page containing two bad rows can emit an empty token and silently truncate the list. And if the shape is agreeable: apply it to all six List RPCs, or start with ListActors?Maya Wang (@mayawang) — you mentioned in #1396 splitting the ListActors per-row isolation out; if you're still planning to take that, please read the above as input rather than a land grab. Otherwise I'm happy to send the step-1 PR (and step 2 once the shape is settled). Either way it should stay coordinated with #1204 (also touching atepg actor storage) and #1607 (a new field is a non-breaking change for the guard).
ericcurtin commentedon Sep 30, 2026
Sent step 1 (name the row) in #2000. Step 2 (per-row isolation) is untouched.
atepg: name the row in list decode errors (agent-substrate#2000)