Repository navigation
Egresspolicy impl - #1545
Egresspolicy impl#1545
Conversation
6140417 to
d89389a
Compare
|
|
||
| // Compile parses every pattern and CIDR in policy. An entry that does not | ||
| // parse is dropped from its rule and reported; ateapi validated it on write, | ||
| // so this only happens on drift, and dropping fails closed. The returned |
There was a problem hiding this comment.
"so this should not happen, but fail close in this case"?
Is this actually expected for version drift?
There was a problem hiding this comment.
hmm,
only when ateapi and atenet at different versions mid-rollout or something. And to mitigate if Parse fails we dont include it.
| if err != nil { | ||
| return netip.Prefix{}, fmt.Errorf("%q is not a CIDR prefix", raw) | ||
| } | ||
| if prefix.Addr().Is4In6() { |
There was a problem hiding this comment.
why does this matter?
There was a problem hiding this comment.
edge case but if someone added a rule on an IPv4 thats written in IPv6 form (with ::fffff). The gateway calls Unmap on every address before matching, so that rule would never match, so we reject it.
| - name: envoy.filters.http.ext_proc | ||
| typed_config: | ||
| "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor | ||
| stat_prefix: egress_policy |
There was a problem hiding this comment.
The same stat_prefix is used for both ext_proc added in this PR. Let us use different prefixes.
There was a problem hiding this comment.
done
| // HandleRequestHeaders dispatches on the filter chain the request arrived on. | ||
| // An empty chain name is a non-Envoy dataplane, which calls out for the |
There was a problem hiding this comment.
Is it possible that a non-Envoy dataplane lands here?
There was a problem hiding this comment.
depends if those non-envoy dataplanes call this extProc handler. HandleRequestHeaders is common and passing that chain name is something we only need to the envoy impl currently. I am leaving a way to support something that does not need this chain name.
|
For the egress gateway with MITM, if an actor's egress policy does not include any |
|
Can you update the PR description with more details on how traffic is handled in both gateways? |
0678a3a to
2948995
Compare
The egress gateway is about to enforce EgressPolicy, which means two components now interpret the same hostname patterns and CIDRs: ateapi when it validates a policy, and atenet when it matches one. Putting the parsers in internal/egresspolicy and having ateapi's custom validators call them keeps "accepted by the API" and "enforceable by the gateway" the same set by construction. The package also compiles a policy once and evaluates it the way the API describes, in rule order over whatever the caller knows of the destination, first match wins: a hostname rule never matches a destination with no name, and an IP rule never matches one whose address the caller does not know. The decision says which kind of rule matched, because the gateway sends the bytes to what that rule checked. HasHostnameRules answers the question that follows, which is whether a decision point holding only an address should refuse outright or leave the destination to one that will see a name. Hostname patterns now reject any name whose last label is all digits, per RFC 1123 section 2.1, instead of consulting an IP parser. That is the same set as before for every real address and additionally refuses dotted-quad spellings such as "01.2.3.4" that no parser accepts. The actor SPIFFE ID moves to internal/resources with a parser next to the builder: ateapi mints it into the certificate's URI SAN and the egress gateway reads it back as the actor identity on the legs inside the tunnel, so both ends need the one definition.
The egress ext_proc handler is about to authorize every request inside every tunnel against the actor's EgressPolicy. Fetching that policy from ateapi per request would put the control plane on the data path, so it is read through a per-actor cache instead. The TTL is exactly how stale a decision can be: a create, update or delete is visible to new requests within one TTL, and a deletion becomes a deny. "No policy" is cached like a policy is, because deny-by-default is the answer a flood of refused requests hits. A failed fetch is never cached, so an ateapi outage fails only the requests it overlaps. Concurrent fetches of one actor's policy are collapsed into a single call, and that call outlives the caller that started it, so the burst an actor sends the moment it wakes up costs one round trip. Expired entries are removed once the live set has doubled, so a gateway that has seen many actors does not keep a policy for each of them forever.
2948995 to
870a03e
Compare
870a03e to
955df7f
Compare
| Scheme: "spiffe", | ||
| // TODO(identity): Must be configurable per-install, so that each install can set it to a unique value. | ||
| Host: "substrate-actor.local", | ||
| // TODO(identity): Prefix with "atunnel" to prevent | ||
| // confusion between atunnel and an actor pretending to be | ||
| // an atunnel. | ||
| Path: path.Join("atespace", dbActor.GetMetadata().GetAtespace(), "actor", dbActor.GetMetadata().GetName()), |
There was a problem hiding this comment.
TODOs were moved into internal/resources/spiffe.go
The egress handler so far authenticated the actor behind a CONNECT and
let everything through. It now authorizes the traffic against the actor's
EgressPolicy, on the leg where the destination is known. Which leg that
is comes from the filter chain name the dataplane asserts, the same
signal the mux already dispatches on.
* egress, the outer CONNECT, is the only leg that sees a certificate,
and the first to see the address the actor's kernel dialed. It keeps
authenticating that certificate and decides the address rules,
ip_blocks and all, against that address, once per connection. When
one of them allows it the handler hands the address back as dynamic
metadata, which is what lets the gateway dial it for traffic it
cannot read and what tells the legs inside what was dialed. When
none does but the policy names hostnames, the tunnel is opened and
the requests inside are left to the legs that will see a name. When
neither is true nothing inside the tunnel could ever be allowed, so
the CONNECT is refused. A dataplane that calls out for the CONNECT
alone sends no chain name and has no such legs behind it, so it gets
the whole decision here and is refused rather than deferred.
* egress_tls_mitm and egress_cleartext are the legs the gateway can
read. Every request is decided the way the API describes: the rules
in order, over the request's Host and the address the actor dialed,
first match wins. It runs per request because the Host can change
between requests on one connection. The answer also says where the
request goes, so the bytes reach what the matching rule checked: a
hostname match is resolved and dialed by name, an address or all
match goes to the dialed address. Sending an address match by name
would let an actor put any Host on an allowed address; sending a
hostname match to the dialed address would turn the rule into a
header check, since the actor picks both. A Host header that
disagrees with :authority is refused rather than policed on one name
and dialed on the other.
Identity on the request legs is the actor's SPIFFE ID, which the outer
chain sets as filter state from the certificate Envoy verified and
shares across the internal-listener hop; nothing inside the tunnel can
write it, and a callout without it is refused. The certificate's URI SAN
must name the same actor as its ActorIdentity extension, because the
CONNECT authenticates on the extension while the request legs attribute
traffic to the SAN.
Policies are read through the per-actor cache, whose TTL is a new
--egress-policy-cache-ttl flag defaulting to 10s, so a request inside a
tunnel is not a control-plane round trip.
Credential injection is not implemented yet. A matched rule that declares
one denies with 501 rather than forwarding without the secret the policy
promised.
…owed
The handler decides address rules at the CONNECT and every readable
request on its Host and the dialed address, so both gateways have to
give it those legs and act on what it decided.
The CONNECT chain's ext_proc may answer with dynamic metadata, and the
gateway admits exactly one namespace of it, dev.ate.egress. A
set_filter_state filter placed after the ext_proc copies
dev.ate.egress:passthrough_destination into Envoy's
envoy.network.transport_socket.original_dst_address and shares it with
the upstream connection. That is the whole mechanism carrying the
decision from the process that made it to the socket that acts on it,
with no header anywhere in between.
Inside the tunnel a connection lands on one of three chains, chosen by
tls_inspector and http_inspector under the 1s timeout that already
governs sdsmint:
* egress_cleartext, on both gateways, matches raw_buffer with an HTTP
ALPN and runs the policy ext_proc on every request. Its answer,
dev.ate.egress:dial, picks the route: name goes to the
dynamic_forward_proxy cluster, which resolves the Host, and address
to an ORIGINAL_DST cluster fed by the same filter state as the
passthrough chains, so the bytes go to what the matching rule
checked. There is no route without a dial, and the answer clears the
route cache, because Envoy picks a route before the filter runs. The
chain accepts HTTP/1.0, which http_inspector steers here and the
codec used to refuse before any filter ran.
* egress_tls_mitm, sdsmint only, is that same per-request check and
routing on the requests the gateway decrypted. Both of its dials
re-originate TLS with the SNI and certificate check taken from the
Host, so a Host the dialed origin cannot prove fails the handshake.
Its filter sits below the existing #ATE_MITM_EXTPROC_FILTER markers,
which keep working for an external processor spliced in by the
installer.
* the passthrough chains take what is left: on the plain gateway TLS it
will not terminate, and on both gateways anything the inspectors
could not classify, a client that said nothing before the timeout
included. They run no filter. Their tcp_proxy points at an
ORIGINAL_DST cluster with no address of its own, so it dials whatever
the CONNECT decision put in filter state and closes the connection
when there is nothing there. A destination no address rule allowed is
therefore unreachable for traffic the gateway cannot read, which is
what makes deferring a decision to the request legs safe.
There is one such chain per transport protocol rather than a single
chain matching nothing, because Envoy buckets filter chains by
transport protocol and never falls out of a bucket that exists. With
egress_cleartext holding the raw_buffer bucket and filling it with
HTTP application protocols alone, a chain matching nothing would be
unreachable and every opaque or unclassified connection would be
closed as no_filter_chain_match, address rule or not. A manifest test
now pins the invariant that each transport protocol the inspectors
set has a chain, and that each has one catching what they could not
name.
The plain gateway gains the inner listener it never had; until now it
spliced the tunnel straight to the IP:port and could police nothing
inside it. Its TLS stays end to end encrypted, since there is no MITM CA
there, so what it can enforce for HTTPS is the address, which the
manifest says in as many words. Server-speaks-first protocols pay the 1s
sniff timeout on this gateway now, as they already did on sdsmint.
The outer hop's connection pool is keyed per actor through
envoy.network.upstream_server_name, set from the same verified
certificate as the identity. Envoy captures shared filter state when a
pool is created and a string object takes no part in the pool's key, so
without that key a second actor's tunnel could be handed a pool carrying
the first actor's identity. The clusters that target an internal
listener also cap themselves at one request per connection.
The manifest tests pin the chain names, the attributes each ext_proc
requests, the metadata namespace every leg may answer in, the filter
that turns the CONNECT leg's answer into the passthrough address and the
absence of any other writer of that key, the two routes each request leg
has and the clusters they select, and a distinct stat_prefix per
ext_proc, because a mismatch fails closed at runtime and nothing else
would notice.
The egress gateway now refuses every tunnel for an actor without an EgressPolicy, so every e2e actor that egresses needs one. A new e2e.EnsureEgressPolicy creates or replaces an actor's policy, and the networking and egressmitm suites give their actors an allow-all rule at creation, which keeps every existing egress test meaning what it meant. The cases cover the policy itself. An actor whose policy names one hostname is refused with 403 and "egress denied" when it asks for another over cleartext HTTP, and for the same origin by address, since an IP-literal Host names no host. An actor with no policy gets no tunnel at all, which the demo app reports as a 502 from a failed connect. A policy naming the origin's address admits a plain HTTP request to it both by address and by name, because the request is decided on the address the actor dialed, and it gets no tunnel for a name that resolves anywhere else. Over HTTPS an address policy admits example.com on both gateways: the plain gateway by address at the CONNECT, sdsmint on the decrypted request, which it then sends to the dialed address with the certificate checked against the name. The one case where the gateways differ is a hostname policy over HTTPS, and it is two tests, one per gateway, each skipping on the other. sdsmint reads the decrypted request, so it admits the named host and answers 403 for another; the plain gateway cannot read TLS and no address rule allows either destination, so it closes both connections before a byte reaches an origin. E2E_EGRESS_MITM tells the suite which gateway it is running against. The egressmitm suite adds the same 403 to its probe, next to the trust-bundle assertions it already makes.
…y leg The router README gets the gateway-side account, a leg at a time, including the metadata that carries the CONNECT leg's decision to the chain that dials. The egress demo README stops calling destination authorization a follow-up.
955df7f to
6436d3e
Compare
|
Bowei Du (@bowei) , Lior Lieberman (@LiorLieberman) , we can address this in a followup PR, potentially bundled with the other changes to the EgressPolicy API. However, this specific enhancement is not blocked by any API change. |
Bowei Du (bowei)
left a comment
There was a problem hiding this comment.
Comments are for follow ups
| ) | ||
|
|
||
| // DefaultPolicyCacheTTL is how long a fetched policy stays usable, and so how | ||
| // stale a decision can be. |
There was a problem hiding this comment.
do we cache decisions or the policy itself?
There was a problem hiding this comment.
we really need the API to tell us what validity is ok. Add a TODO here.
| // Handler authenticates the actor behind each egress CONNECT. | ||
| // deniedBody is the body of every policy denial. The reason goes to the log, | ||
| // not to the actor. | ||
| const deniedBody = "egress denied" |
There was a problem hiding this comment.
Probably want to put this in a bunch of functions that create the egress denied error(s) rather than have the bare const here.
| // Only the Envoy gateway has request legs behind this one. A dataplane | ||
| // that calls out for the CONNECT alone sends no chain name and is | ||
| // refused below. | ||
| slog.InfoContext(ctx, "egress tunnel opened: no address rule allows the destination, requests inside it are decided one by one", attrs...) |
There was a problem hiding this comment.
"requests inside it are decided one by one" do we need to clarify this with the fact that we check the :authority: that it matches the connection-based allow?
c1d5b29
into
agent-substrate:main
Enforces an Actor's
EgressPolicyat the egress gateway. Eeally encouraged to review commit by commit.internal/egresspolicy: one parser and evaluator for hostname patterns and CIDRs, used by ateapi to validate a policy and by atenet to enforce it, so what the API accepts is what the gateway can match.--egress-policy-cache-ttl, 10s default). The TTL is how stale a decision can be. A missing policy is cached as a deny; a failed fetch is not cached.Request Flow
IP:portis passed to the legs inside as filter state. An Actor with no policy gets no tunnel.Hostand the address the actor dialed. The first match authorizes it; no match is a 403egress denied.allmatch goes to the dialed address. On sdsmint both re-originate TLS with the SNI and certificate check taken from theHost. The bytes never go anywhere the rule did not look at.Follow-ups
inject_static_headers). A matched rule that declares one is refused with 501 rather than forwarded without it.