Skip to content

Egresspolicy impl - #1545

Merged
Bowei Du (bowei) merged 7 commits into
agent-substrate:mainfrom
LiorLieberman:egresspolicy-impl
Sep 14, 2026
Merged

Bowei Du (bowei) merged 7 commits into
agent-substrate:mainfrom
LiorLieberman:egresspolicy-impl

Conversation

@LiorLieberman

@LiorLieberman Lior Lieberman (LiorLieberman) commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

Enforces an Actor's EgressPolicy at the egress gateway. Eeally encouraged to review commit by commit.

  • internal/egresspolicy: one parser and evaluator for hostname patterns and CIDRs, used by ateapi to validate a policy and by atenet to enforce it, so what the API accepts is what the gateway can match.
  • A per-actor policy cache in atenet (--egress-policy-cache-ttl, 10s default). The TTL is how stale a decision can be. A missing policy is cached as a deny; a failed fetch is not cached.
  • Enforcement in the egress ext_proc handler, on the leg that knows the destination, and the Envoy manifests that give it those legs. The inner listener is now also added to the non-sdsmint gateway (already existed in the sdsmint one). (yes we dont plan to have two gateways in the future but this is a separate workstream)
  • E2E coverage + README updates.

Request Flow

  • Every CONNECT is authenticated from the actor certificate. The tunnel opens only if some rule could match its destination, and the dialed IP:port is passed to the legs inside as filter state. An Actor with no policy gets no tunnel.
  • Every request the gateway can read (cleartext HTTP on both gateways, HTTPS the sdsmint gateway terminates) is evaluated once, as the API describes: rules in order, over the request's Host and the address the actor dialed. The first match authorizes it; no match is a 403 egress denied.
  • The request is sent to what the matching rule checked: a hostname match is resolved and dialed by name, an address or all match goes to the dialed address. On sdsmint both re-originate TLS with the SNI and certificate check taken from the Host. The bytes never go anywhere the rule did not look at.
  • Traffic the gateway cannot read (TLS on the plain gateway, opaque TCP) is decided at the CONNECT by address rules alone, and closed when none allows it.

Follow-ups

  • Hostname rules for TLS the plain gateway passes through. That needs the SNI (with a hostname) at the CONNECT.
  • Credential injection (inject_static_headers). A matched rule that declares one is refused with 501 rather than forwarded without it.
  • Metrics for policy decisions, which follow in their own PR.

It's a good idea to open an issue first for discussion.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

@LiorLieberman
Lior Lieberman (LiorLieberman) force-pushed the egresspolicy-impl branch 6 times, most recently from 6140417 to d89389a Compare September 9, 2026 00:39
Comment thread internal/egresspolicy/egresspolicy.go Resolved
Comment thread internal/egresspolicy/egresspolicy.go Outdated

// Compile parses every pattern and CIDR in policy. An entry that does not
// parse is dropped from its rule and reported; ateapi validated it on write,
// so this only happens on drift, and dropping fails closed. The returned

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"so this should not happen, but fail close in this case"?

Is this actually expected for version drift?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm,

only when ateapi and atenet at different versions mid-rollout or something. And to mitigate if Parse fails we dont include it.

Comment thread internal/egresspolicy/egresspolicy.go Resolved
Comment thread internal/egresspolicy/egresspolicy.go Resolved
Comment thread internal/egresspolicy/egresspolicy.go Resolved
Comment thread internal/egresspolicy/egresspolicy.go Outdated
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q is not a CIDR prefix", raw)
}
if prefix.Addr().Is4In6() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this matter?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

edge case but if someone added a rule on an IPv4 thats written in IPv6 form (with ::fffff). The gateway calls Unmap on every address before matching, so that rule would never match, so we reject it.

Comment thread internal/egresspolicy/egresspolicy.go Resolved
- name: envoy.filters.http.ext_proc
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
stat_prefix: egress_policy

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same stat_prefix is used for both ext_proc added in this PR. Let us use different prefixes.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@haiyanmeng haiyanmeng added the DO NOT MERGE This PR must not be merged yet. label Sep 9, 2026
Comment on lines +102 to +103
// HandleRequestHeaders dispatches on the filter chain the request arrived on.
// An empty chain name is a non-Envoy dataplane, which calls out for the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible that a non-Envoy dataplane lands here?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

depends if those non-envoy dataplanes call this extProc handler. HandleRequestHeaders is common and passing that chain name is something we only need to the envoy impl currently. I am leaving a way to support something that does not need this chain name.

@haiyanmeng

Copy link
Copy Markdown
Collaborator

For the egress gateway with MITM, if an actor's egress policy does not include any hostnameRule, the https traffic seems still getting to the inner listener. I think in this case, the CONNECT listener can make a decisive decision.

@haiyanmeng

Copy link
Copy Markdown
Collaborator

Can you update the PR description with more details on how traffic is handled in both gateways?

@LiorLieberman
Lior Lieberman (LiorLieberman) force-pushed the egresspolicy-impl branch 2 times, most recently from 0678a3a to 2948995 Compare September 13, 2026 20:45
@LiorLieberman
Lior Lieberman (LiorLieberman) marked this pull request as ready for review September 13, 2026 20:45
@LiorLieberman Lior Lieberman (LiorLieberman) removed the DO NOT MERGE This PR must not be merged yet. label Sep 13, 2026
The egress gateway is about to enforce EgressPolicy, which means two
components now interpret the same hostname patterns and CIDRs: ateapi
when it validates a policy, and atenet when it matches one. Putting the
parsers in internal/egresspolicy and having ateapi's custom validators
call them keeps "accepted by the API" and "enforceable by the gateway"
the same set by construction. The package also compiles a policy once
and evaluates it the way the API describes, in rule order over whatever
the caller knows of the destination, first match wins: a hostname rule
never matches a destination with no name, and an IP rule never matches
one whose address the caller does not know. The decision says which
kind of rule matched, because the gateway sends the bytes to what that
rule checked. HasHostnameRules answers the question that follows, which
is whether a decision point holding only an address should refuse
outright or leave the destination to one that will see a name.

Hostname patterns now reject any name whose last label is all digits,
per RFC 1123 section 2.1, instead of consulting an IP parser. That is
the same set as before for every real address and additionally refuses
dotted-quad spellings such as "01.2.3.4" that no parser accepts.

The actor SPIFFE ID moves to internal/resources with a parser next to
the builder: ateapi mints it into the certificate's URI SAN and the
egress gateway reads it back as the actor identity on the legs inside
the tunnel, so both ends need the one definition.
The egress ext_proc handler is about to authorize every request inside
every tunnel against the actor's EgressPolicy. Fetching that policy from
ateapi per request would put the control plane on the data path, so it
is read through a per-actor cache instead.

The TTL is exactly how stale a decision can be: a create, update or
delete is visible to new requests within one TTL, and a deletion becomes
a deny. "No policy" is cached like a policy is, because deny-by-default
is the answer a flood of refused requests hits. A failed fetch is never
cached, so an ateapi outage fails only the requests it overlaps.
Concurrent fetches of one actor's policy are collapsed into a single
call, and that call outlives the caller that started it, so the burst an
actor sends the moment it wakes up costs one round trip. Expired entries
are removed once the live set has doubled, so a gateway that has seen many
actors does not keep a policy for each of them forever.
Comment on lines -673 to -679
Scheme: "spiffe",
// TODO(identity): Must be configurable per-install, so that each install can set it to a unique value.
Host: "substrate-actor.local",
// TODO(identity): Prefix with "atunnel" to prevent
// confusion between atunnel and an actor pretending to be
// an atunnel.
Path: path.Join("atespace", dbActor.GetMetadata().GetAtespace(), "actor", dbActor.GetMetadata().GetName()),

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODOs were moved into internal/resources/spiffe.go

The egress handler so far authenticated the actor behind a CONNECT and
let everything through. It now authorizes the traffic against the actor's
EgressPolicy, on the leg where the destination is known. Which leg that
is comes from the filter chain name the dataplane asserts, the same
signal the mux already dispatches on.

  * egress, the outer CONNECT, is the only leg that sees a certificate,
    and the first to see the address the actor's kernel dialed. It keeps
    authenticating that certificate and decides the address rules,
    ip_blocks and all, against that address, once per connection. When
    one of them allows it the handler hands the address back as dynamic
    metadata, which is what lets the gateway dial it for traffic it
    cannot read and what tells the legs inside what was dialed. When
    none does but the policy names hostnames, the tunnel is opened and
    the requests inside are left to the legs that will see a name. When
    neither is true nothing inside the tunnel could ever be allowed, so
    the CONNECT is refused. A dataplane that calls out for the CONNECT
    alone sends no chain name and has no such legs behind it, so it gets
    the whole decision here and is refused rather than deferred.

  * egress_tls_mitm and egress_cleartext are the legs the gateway can
    read. Every request is decided the way the API describes: the rules
    in order, over the request's Host and the address the actor dialed,
    first match wins. It runs per request because the Host can change
    between requests on one connection. The answer also says where the
    request goes, so the bytes reach what the matching rule checked: a
    hostname match is resolved and dialed by name, an address or all
    match goes to the dialed address. Sending an address match by name
    would let an actor put any Host on an allowed address; sending a
    hostname match to the dialed address would turn the rule into a
    header check, since the actor picks both. A Host header that
    disagrees with :authority is refused rather than policed on one name
    and dialed on the other.

Identity on the request legs is the actor's SPIFFE ID, which the outer
chain sets as filter state from the certificate Envoy verified and
shares across the internal-listener hop; nothing inside the tunnel can
write it, and a callout without it is refused. The certificate's URI SAN
must name the same actor as its ActorIdentity extension, because the
CONNECT authenticates on the extension while the request legs attribute
traffic to the SAN.

Policies are read through the per-actor cache, whose TTL is a new
--egress-policy-cache-ttl flag defaulting to 10s, so a request inside a
tunnel is not a control-plane round trip.

Credential injection is not implemented yet. A matched rule that declares
one denies with 501 rather than forwarding without the secret the policy
promised.
…owed

The handler decides address rules at the CONNECT and every readable
request on its Host and the dialed address, so both gateways have to
give it those legs and act on what it decided.

The CONNECT chain's ext_proc may answer with dynamic metadata, and the
gateway admits exactly one namespace of it, dev.ate.egress. A
set_filter_state filter placed after the ext_proc copies
dev.ate.egress:passthrough_destination into Envoy's
envoy.network.transport_socket.original_dst_address and shares it with
the upstream connection. That is the whole mechanism carrying the
decision from the process that made it to the socket that acts on it,
with no header anywhere in between.

Inside the tunnel a connection lands on one of three chains, chosen by
tls_inspector and http_inspector under the 1s timeout that already
governs sdsmint:

  * egress_cleartext, on both gateways, matches raw_buffer with an HTTP
    ALPN and runs the policy ext_proc on every request. Its answer,
    dev.ate.egress:dial, picks the route: name goes to the
    dynamic_forward_proxy cluster, which resolves the Host, and address
    to an ORIGINAL_DST cluster fed by the same filter state as the
    passthrough chains, so the bytes go to what the matching rule
    checked. There is no route without a dial, and the answer clears the
    route cache, because Envoy picks a route before the filter runs. The
    chain accepts HTTP/1.0, which http_inspector steers here and the
    codec used to refuse before any filter ran.
  * egress_tls_mitm, sdsmint only, is that same per-request check and
    routing on the requests the gateway decrypted. Both of its dials
    re-originate TLS with the SNI and certificate check taken from the
    Host, so a Host the dialed origin cannot prove fails the handshake.
    Its filter sits below the existing #ATE_MITM_EXTPROC_FILTER markers,
    which keep working for an external processor spliced in by the
    installer.
  * the passthrough chains take what is left: on the plain gateway TLS it
    will not terminate, and on both gateways anything the inspectors
    could not classify, a client that said nothing before the timeout
    included. They run no filter. Their tcp_proxy points at an
    ORIGINAL_DST cluster with no address of its own, so it dials whatever
    the CONNECT decision put in filter state and closes the connection
    when there is nothing there. A destination no address rule allowed is
    therefore unreachable for traffic the gateway cannot read, which is
    what makes deferring a decision to the request legs safe.

    There is one such chain per transport protocol rather than a single
    chain matching nothing, because Envoy buckets filter chains by
    transport protocol and never falls out of a bucket that exists. With
    egress_cleartext holding the raw_buffer bucket and filling it with
    HTTP application protocols alone, a chain matching nothing would be
    unreachable and every opaque or unclassified connection would be
    closed as no_filter_chain_match, address rule or not. A manifest test
    now pins the invariant that each transport protocol the inspectors
    set has a chain, and that each has one catching what they could not
    name.

The plain gateway gains the inner listener it never had; until now it
spliced the tunnel straight to the IP:port and could police nothing
inside it. Its TLS stays end to end encrypted, since there is no MITM CA
there, so what it can enforce for HTTPS is the address, which the
manifest says in as many words. Server-speaks-first protocols pay the 1s
sniff timeout on this gateway now, as they already did on sdsmint.

The outer hop's connection pool is keyed per actor through
envoy.network.upstream_server_name, set from the same verified
certificate as the identity. Envoy captures shared filter state when a
pool is created and a string object takes no part in the pool's key, so
without that key a second actor's tunnel could be handed a pool carrying
the first actor's identity. The clusters that target an internal
listener also cap themselves at one request per connection.

The manifest tests pin the chain names, the attributes each ext_proc
requests, the metadata namespace every leg may answer in, the filter
that turns the CONNECT leg's answer into the passthrough address and the
absence of any other writer of that key, the two routes each request leg
has and the clusters they select, and a distinct stat_prefix per
ext_proc, because a mismatch fails closed at runtime and nothing else
would notice.
The egress gateway now refuses every tunnel for an actor without an
EgressPolicy, so every e2e actor that egresses needs one. A new
e2e.EnsureEgressPolicy creates or replaces an actor's policy, and the
networking and egressmitm suites give their actors an allow-all rule at
creation, which keeps every existing egress test meaning what it meant.

The cases cover the policy itself. An actor whose policy names one
hostname is refused with 403 and "egress denied" when it asks for
another over cleartext HTTP, and for the same origin by address, since
an IP-literal Host names no host. An actor with no policy gets no tunnel
at all, which the demo app reports as a 502 from a failed connect. A
policy naming the origin's address admits a plain HTTP request to it
both by address and by name, because the request is decided on the
address the actor dialed, and it gets no tunnel for a name that resolves
anywhere else. Over HTTPS an address policy admits example.com on both
gateways: the plain gateway by address at the CONNECT, sdsmint on the
decrypted request, which it then sends to the dialed address with the
certificate checked against the name.

The one case where the gateways differ is a hostname policy over HTTPS,
and it is two tests, one per gateway, each skipping on the other.
sdsmint reads the decrypted request, so it admits the named host and
answers 403 for another; the plain gateway cannot read TLS and no
address rule allows either destination, so it closes both connections
before a byte reaches an origin. E2E_EGRESS_MITM tells the suite which
gateway it is running against. The egressmitm suite adds the same 403
to its probe, next to the trust-bundle assertions it already makes.
…y leg

The router README gets the gateway-side account, a leg at a time,
including the metadata that carries the CONNECT leg's decision to the
chain that dials. The egress demo README stops calling destination
authorization a follow-up.
@haiyanmeng

Copy link
Copy Markdown
Collaborator
  • Enforcement in the egress ext_proc handler, on the leg that knows the destination, and the Envoy manifests that give it those legs. The inner listener is not also added to the non-sdsmint gateway (already existed in the sdsmint one). (yes we dont plan to have two gateways in the future but this is a separate workstream)

The inner listener is not -> The inner listener is now

@haiyanmeng

Copy link
Copy Markdown
Collaborator

For the egress gateway with MITM, if an actor's egress policy does not include any hostnameRule, the https traffic seems still getting to the inner listener. I think in this case, the CONNECT listener can make a decisive decision.

Bowei Du (@bowei) , Lior Lieberman (@LiorLieberman) , we can address this in a followup PR, potentially bundled with the other changes to the EgressPolicy API. However, this specific enhancement is not blocked by any API change.

@bowei Bowei Du (bowei) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments are for follow ups

)

// DefaultPolicyCacheTTL is how long a fetched policy stays usable, and so how
// stale a decision can be.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we cache decisions or the policy itself?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we really need the API to tell us what validity is ok. Add a TODO here.

// Handler authenticates the actor behind each egress CONNECT.
// deniedBody is the body of every policy denial. The reason goes to the log,
// not to the actor.
const deniedBody = "egress denied"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably want to put this in a bunch of functions that create the egress denied error(s) rather than have the bare const here.

// Only the Envoy gateway has request legs behind this one. A dataplane
// that calls out for the CONNECT alone sends no chain name and is
// refused below.
slog.InfoContext(ctx, "egress tunnel opened: no address rule allows the destination, requests inside it are decided one by one", attrs...)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"requests inside it are decided one by one" do we need to clarify this with the fact that we check the :authority: that it matches the connection-based allow?

@bowei
Bowei Du (bowei) merged commit c1d5b29 into agent-substrate:main Sep 14, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants