Repository navigation
Egress gateway reports actor JWT cache hits and mint latency - #2317
Max Thompson (thompsonmax) wants to merge 1 commit into
Conversation
faf02d7 to
0e6396d
Compare
The egress gateway's actor JWT minter now records two instruments: ate.egress.actor_jwt.lookups counts each token lookup by outcome (hit, miss, error, cancelled, timeout), so an operator can see whether the token cache is helping, and ate.egress.actor_jwt.mint.duration times each MintActorJWT call, so a slow egress request can be traced to a slow ateapi. Failures carry the gRPC code in error.type. The router builds the instruments and passes them through egress.New, as it does for the ingress parking metrics. Both instruments and the new outcome attribute are in the metric registry.
0e6396d to
13f372c
Compare
Jeff Luo (JeffLuoo)
left a comment
Jeff Luo (JeffLuoo)
left a comment
There was a problem hiding this comment.
Do we need either of these instruments?
-
ate.egress.actor_jwt.mint.durationduplicatesrpc.client.call.duration.Minter.minttimes onlym.client.MintActorJwt(ctx, req). The router buildss.apiClientwithotelgrpc.NewClientHandler()inrouter.go, so each mint already recordsrpc.client.call.durationwithrpc.method="ateapi.Control/MintActorJwt"andrpc.response.status_code. ateapi recordsrpc.server.call.durationfor the same call. Both histograms measure the same RPC: the mint runs undercontext.WithoutCancel, and requests that join a singleflight make no call. The other egress RPCs (GetActor,GetEgressPolicy,FetchSecret) have no second histogram for each method. -
ate.egress.actor_jwt.lookupsmeasures a cache that has no size limit.Minterusescache.Expiringand reuses each token until one third ofexpiration_secondsremains. Thus a miss occurs only on the first request or after a token expires, and the miss count is almost equal to theMintActorJwtcall count. The hit ratio shows only how many egress requests an actor sends during one token lifetime, and no setting in the gateway changes it. This is not likeate.imagecache.requests: there, a miss pulls OCI layers and disk GC removes entries.policyCachein the same package has no lookup counter.
Can we use atenet.router.route.duration to find slow egress callouts, and rpc.client.call.duration by rpc.method to find the slow call? If you want to keep one instrument, please drop ate.egress.actor_jwt.mint.duration at a minimum. If lookups goes, the ate.egress.actor_jwt.outcome attribute and its ateattr constants go too.
I see, #1 makes sense to me. I think for #2, the lookup counter isn't too helpful on its own (and currently we're missing it on the policyCache). It may make sense to later instrument all the caches in egress gateway with hit/miss metrics, but I don't think we have a strong operational use case for this today. I think I'll close this one for now to free up some review bandwidth prior to GA. Thanks! |
) Part of agent-substrate#1660. - `docs/egress-credential-injection.md` gains an Actor JWTs section: the `actorJwt` policy fields and their limits, the claims a minted token carries, and how long the gateway reuses a token before minting another. - The outcome table gains rows for actor JWT entries: 403 when the actor no longer exists, 503 when ateapi is unreachable, and 500 (Envoy) or 403 (agentgateway) for any other mint failure. The intro and "How it works" mention the ateapi path. - An operational note says tokens follow the actor's name, so a recreated actor can get its predecessor's token until the refresh point. The metrics from agent-substrate#2317 aren't mentioned, since they aren't on main yet. Testing: docs only. The YAML example parses with `kubectl ate`'s policy parser, and each documented limit matches ateapi's validation, both checked with throwaway tests. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
Part of #1660.
ate.egress.actor_jwt.lookupscounts each token lookup in the egress gateway's actor JWT cache by outcome (hit,miss,error,cancelled,timeout), so the hit ratio shows whether the cache is saving mints.ate.egress.actor_jwt.mint.durationtimes eachMintActorJwtcall to ateapi, so a slow egress request can be traced to a slow mint. Its count is the mint rate.error.type. No label names an actor or an audience.egress.New, as it does for the ingress parking metrics. The registry,ateattr, and the observability table gain the two instruments and the outcome attribute.egress.Newgains a parameter; the tests that call it pass nil.Testing:
ManualReaderunit tests cover a miss then a hit, a failed mint, and callers that cancel or time out.hack/verify/metrics.shpasses with Weaver v0.25.1. Not run on a cluster, and no dashboard panel yet.