Skip to content

docs: how an actor exports telemetry under the egress lockdown - #2120

Merged
Jeff Luo (JeffLuoo) merged 2 commits into
agent-substrate:mainfrom
git286:docs-actor-otel-egress-policy
Oct 2, 2026
Merged

Jeff Luo (JeffLuoo) merged 2 commits into
agent-substrate:mainfrom
git286:docs-actor-otel-egress-policy

Conversation

@git286

Copy link
Copy Markdown
Collaborator

An actor's own OpenTelemetry SDK does not use the ateom relay; it dials the collector over the network, and the actor egress lockdown denies that connection unless the actor has an EgressPolicy rule for it. Verified on kind and on GKE (main 4e079e7): with no policy every export fails with "connection reset by peer" and the gateway logs "actor has no egress policy"; with an http rule naming the collector Service on 4317/4318 spans and metrics arrive.

Add a section to the observability guide with the rule for the kind and GKE managed collectors, why it has to be an http rule (OTLP/gRPC is h2c), that it is per actor, the reconnect lag a running actor sees after the policy appears, and that IP-based k8sattributes enrichment attributes the actor's telemetry to the egress gateway pod. Point to it from the egress traffic doc.

An actor's own OpenTelemetry SDK does not use the ateom relay; it dials
the collector over the network, and the actor egress lockdown denies that
connection unless the actor has an EgressPolicy rule for it. Verified on
kind and on GKE (main 4e079e7): with no policy every export fails with
"connection reset by peer" and the gateway logs "actor has no egress
policy"; with an http rule naming the collector Service on 4317/4318
spans and metrics arrive.

Add a section to the observability guide with the rule for the kind and
GKE managed collectors, why it has to be an http rule (OTLP/gRPC is h2c),
that it is per actor, the reconnect lag a running actor sees after the
policy appears, and that IP-based k8sattributes enrichment attributes the
actor's telemetry to the egress gateway pod. Point to it from the egress
traffic doc.

@haiyanmeng haiyanmeng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Will leave it to Bowei Du (@bowei) to approve.

Comment thread docs/observability.md Outdated

An actor's own OpenTelemetry SDK does not use the relay. It dials the collector over the network like any other outbound connection, and that connection is subject to the actor egress lockdown described in [Network Egress](network-egress.md): every actor TCP connection is redirected into `atunnel` and carried to the egress gateway, which refuses anything the actor's `EgressPolicy` does not allow. An actor with no policy, or a policy with no matching rule, cannot reach the collector at all. The SDK sees `Unavailable: connection reset by peer` on each export, and the gateway logs `egress denied: actor has no egress policy`.

To let an actor export, give it an `http` rule naming the collector's Service and the OTLP ports. It must be an `http` rule, not `https`: the OTLP/gRPC exporter speaks cleartext HTTP/2 (h2c) on `4317`, and the OTLP/HTTP exporter speaks cleartext HTTP/1.1 on `4318`, which is exactly what an `http` rule admits. On Kind the collector is `opentelemetry-collector.otel-system.svc`; on GKE with the managed collector it is `opentelemetry-collector.gke-managed-otel.svc.cluster.local`. Whatever the name, it has to match the host the actor's `OTEL_EXPORTER_OTLP_ENDPOINT` names, because the rule is matched on the request authority.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if actor wants to use gRPC? Does current egress mode support it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The OTLP/gRPC exporter on 4317 is exactly that case, gRPC over cleartext HTTP/2 (h2c), and it is what I verified on kind and GKE for this PR.

I reworded the sentence here to say the gRPC exporter goes through an http rule.

The sentence about the OTLP ports read as a note on transport. Make
the point explicit: the gRPC exporter is gRPC over h2c, and that is
what an http rule allows through.
@bowei
Bowei Du (bowei) added this pull request to the merge queue Oct 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 2, 2026
@JeffLuoo
Jeff Luo (JeffLuoo) added this pull request to the merge queue Oct 2, 2026
Merged via the queue into agent-substrate:main with commit 31e0f75 Oct 2, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants