Reduce memory utilization in Envoy egress gateway by deduplicating policy object by yanavlasov · Pull Request #2651 · agent-substrate/substrate

yanavlasov · 2026-10-14T15:17:39Z

This PR reduces memory usage by SNI based egress policies, surface by scalability tests with 1M actors and policies and 1K SNI rules per policy.

The cause of memory consumption was two fold:

policy proto sent from ext_proc to Envoy was kept in memory, even though it was no longer used after it was converted into internal policy representation.
Each CONNECT request kept its own copy of the egress policy, duplicating policy in memory by factor of Actors*Connections_per_actor.

This PR makes two high level changes:

policy proto sent from the ext_proc is erased after it is parsed into the Envoy internal policy object.
policy object is hashed using high quality Rapidhash and stored in thread local store using computed hash key. The policy hash is then communicated to the inner filter chain that fetches policy from the store and enforces SNI rules.

With this change, if all actors have exactly the same policy, Envoy will only have 1 copy of the policy object per worker thread.

Tests pass
Appropriate changes to documentation are included in the PR

Signed-off-by: Yan Avlasov <[email protected]>

yanavlasov · 2026-10-14T16:03:58Z

Benchmark results of single core dataplane with 1000QPS of H/1 requests did not change from baseline. As expected.

Lior Lieberman (LiorLieberman) · 2026-10-14T22:05:36Z

is this ready for review?

Yufan Su (yufan-su) · 2026-10-14T23:57:53Z

I ran a small egress test with this change and see some big improvements on memory per tunnel.

Setup: 100 actors × 1,000 endpoints = 100,000 HTTPS tunnels. Every actor has the same 1,000-host policy. One gateway replica (16 Envoy threads).

Metric	Before PR	With PR 2651	Change
Envoy memory, 100k HTTPS tunnels	51.5 GiB	6.9 GiB	−87%
Memory per HTTPS tunnel	520 KiB	56 KiB	−89% (9.3× less)
Memory per HTTP tunnel (control)	49 KiB	50 KiB	unchanged

yanavlasov · 2026-10-15T00:44:36Z

is this ready for review?

Yes it is. Test failed due to unrelated flake. I will try to re-start it.

Lior Lieberman (LiorLieberman)

nice improvement in memory!! thanks

Lior Lieberman (LiorLieberman) · 2026-10-15T01:02:08Z

+    };
+    envoy_log_trace!("egress policy store: policy={:?}", policy);
+    let mut local_store = GlobalEgressPolicyStore::get_local_store();
+    if let Some(existing_policy) = local_store.lookup(policy.hash) {


is there a chance of collision here? I read that its small, but how expensive is it for us to compare the full policy on a store hit and fail closed if it doesn't match?

I'm not sure. Could be a lot of string comparisons for policies with a lot of SNIs. I have added a sanity check for now. We could see if it shows up in CPU profiles for load tests.

Lior Lieberman (LiorLieberman) · 2026-10-15T01:03:24Z


-## What it decides
+To minimize memory usage egress policies are hashed using [Rapidhash](https://docs.rs/rapidhash/latest/rapidhash/)
+which produces collision resistant 64 bit hashes. Policies are then stored in the worker local


nit: this is not really collision resistant since its non cryto.

Lior Lieberman (LiorLieberman) · 2026-10-15T01:10:24Z

@@ -63,6 +63,12 @@ const (
 	// the egress-policy module: {"rules": [{"pattern": ..., "mode": ...}]},
 	// most specific first.


nit: I think with your change, now only the hash is copied, right? added a suggestion but need to update

the same for the mitm_internal comments (1 and 2) in atenet-egress.yaml and router README.md

suggesting this for line 61-68

// EgressPolicyMetadataNamespace holds the SNI rules returned on CONNECT, as // JSON under EgressPolicyJSONKey: {"rules": [{"pattern": ..., "mode": ...}]}, // most specific first. EgressPolicyMetadataNamespace = "dev.ate.policy.egress" // EgressPolicyHashFilterStateKey holds the hash the inner listener uses to // find those rules in the egress-policy module's per-thread store. EgressPolicyHashFilterStateKey = "dev.ate.policy.egress.hash"

I have changed comments and fixed the doc.

Signed-off-by: Yan Avlasov <[email protected]>

yanavlasov added 23 commits October 13, 2026 00:15

Serialize policy to JSON in ext-proc

d09acec

Signed-off-by: Yan Avlasov <[email protected]>

Empty egress policy store filter

1d72124

Signed-off-by: Yan Avlasov <[email protected]>

Refactor common code

8e653ba

Signed-off-by: Yan Avlasov <[email protected]>

WiP

70db3a0

Signed-off-by: Yan Avlasov <[email protected]>

refactor

ce0380a

Signed-off-by: Yan Avlasov <[email protected]>

consolidate filters

ab4f982

Signed-off-by: Yan Avlasov <[email protected]>

Move common code

2961bec

Signed-off-by: Yan Avlasov <[email protected]>

WiP

ba0ddef

Signed-off-by: Yan Avlasov <[email protected]>

Implement hasher

ba9d98b

Signed-off-by: Yan Avlasov <[email protected]>

WiP

9b02060

Signed-off-by: Yan Avlasov <[email protected]>

Plumb hash into filter state

e1b519e

Signed-off-by: Yan Avlasov <[email protected]>

Pass policy hash via filter state

51e172f

Signed-off-by: Yan Avlasov <[email protected]>

cleanup

2f5cdd4

Signed-off-by: Yan Avlasov <[email protected]>

Add store cleanup

7a2968c

Signed-off-by: Yan Avlasov <[email protected]>

Add store size gauge

44f5da3

Signed-off-by: Yan Avlasov <[email protected]>

Add stats

8d28694

Signed-off-by: Yan Avlasov <[email protected]>

Update doc

d74f2bf

Signed-off-by: Yan Avlasov <[email protected]>

Cleanup dynamic metadata

dceffc8

Signed-off-by: Yan Avlasov <[email protected]>

Merge branch 'main' into mem-opt-1

6b74d74

doc

51f943c

Signed-off-by: Yan Avlasov <[email protected]>

Merge branch 'main' into mem-opt-1

48e9191

comments

105c1ee

Signed-off-by: Yan Avlasov <[email protected]>

fix tests

e1ebb6b

Signed-off-by: Yan Avlasov <[email protected]>

Bowei Du (bowei) added area/network kind/bug Something isn't working / bugfixes labels Oct 14, 2026

Pancho Trujillo (panchoruy) mentioned this pull request Oct 14, 2026

atenet-router: add safe actor assignments caching and retry logic #2420

Open

2 tasks done

Merge branch 'main' into mem-opt-1

ca82bfd

Lior Lieberman (LiorLieberman) reviewed Oct 15, 2026

View reviewed changes

yanavlasov added 2 commits October 15, 2026 01:48

update docs

1e9e216

Signed-off-by: Yan Avlasov <[email protected]>

Add sanity check for hash collisions

c9c06db

Signed-off-by: Yan Avlasov <[email protected]>

Lior Lieberman (LiorLieberman) approved these changes Oct 15, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) added this pull request to the merge queue Oct 15, 2026

Merged via the queue into agent-substrate:main with commit 1738112 Oct 15, 2026
15 checks passed

Yufan Su (yufan-su) mentioned this pull request Oct 22, 2026

Egress gateway: HTTPS tunnels each hold a copy of the actor's SNI rules, causing increased memory usage at scale #2324

Closed

Navigation Menu

Repository navigation

Reduce memory utilization in Envoy egress gateway by deduplicating policy object - #2651

Reduce memory utilization in Envoy egress gateway by deduplicating policy object#2651
Lior Lieberman (LiorLieberman) merged 26 commits into
agent-substrate:mainfrom
yanavlasov:mem-opt-1

yanavlasov commented Oct 14, 2026

Uh oh!

yanavlasov commented Oct 14, 2026

Uh oh!

Lior Lieberman (LiorLieberman) commented Oct 14, 2026

Uh oh!

Yufan Su (yufan-su) commented Oct 14, 2026

Uh oh!

yanavlasov commented Oct 15, 2026

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Uh oh!

Lior Lieberman (LiorLieberman) Oct 15, 2026

Uh oh!

yanavlasov Oct 15, 2026

Uh oh!

Lior Lieberman (LiorLieberman) Oct 15, 2026

Uh oh!

yanavlasov Oct 15, 2026

Uh oh!

Lior Lieberman (LiorLieberman) Oct 15, 2026

Uh oh!

Lior Lieberman (LiorLieberman) Oct 15, 2026

Uh oh!

yanavlasov Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -63,6 +63,12 @@ const (
		// the egress-policy module: {"rules": [{"pattern": ..., "mode": ...}]},
		// most specific first.

Conversation

yanavlasov commented Oct 14, 2026

Uh oh!

yanavlasov commented Oct 14, 2026

Uh oh!

Lior Lieberman (LiorLieberman) commented Oct 14, 2026

Uh oh!

Yufan Su (yufan-su) commented Oct 14, 2026

Uh oh!

yanavlasov commented Oct 15, 2026

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

yanavlasov Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

yanavlasov Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

yanavlasov Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants