Site-to-site VPN: authorized-forwarder role + per-VPC route table (vs. kube-ovn port-security disable) #6

Summary

Assess a first-class site-to-site VPN feature for cozyplane (connect a tenant
VPC to a remote network over IPsec/WireGuard). Not scheduled — this is a
"keep in mind" design note for a capability clients already depend on today. The
goal is to capture how much better cozyplane's model can do it than the common
kube-ovn approach, and which existing primitives it should reuse.

The status quo (kube-ovn + strongSwan in a VM)

The typical pattern: stand up a VM, run strongSwan inside it, and configure the
CNI to route all traffic destined for <remote-cidr> through
<strongswan-vm-ip>, namespace-wide. It works, but it forces two compromises:

• Port security must be disabled on the VPN VM. A VPN endpoint emits packets
  sourced from behind it (the remote peers) and receives packets destined
  beyond it (into remote-cidr). Anti-spoofing/port-security blocks exactly
  that, so the whole protection is switched off for that workload — which then
  can spoof any address and route anything. A huge blast radius for what should
  be a narrow capability.
• Static-IP, namespace-wide routing. "Route remote-cidr via
  <strongswan-vm-ip>" is coarse (whole namespace), brittle (breaks if the VM's
  IP changes or it reschedules), and hand-maintained.

Both are symptoms of one gap: the platform has no concept of an authorized
router, so "this workload forwards transit traffic" degrades to "turn its port
security off," and "reach a remote network" degrades to a static-IP route-table
entry.

Why cozyplane's model changes the shape

The datapath delivers by Port identity, not by trusting a packet's claimed
source, and it already has a first-class notion of a Port that legitimately
carries traffic that is not its own: the gateway Port (ports bit 31, the
gateway mark, the TUN_F_GATEWAY VNI bit). Off-VPC traffic is already steered to
a gateway by the VPC's net_id, never by a static IP. That turns the two
compromises into existing primitives:

1. Router role = a scoped grant, not a safety off-switch. A VPN endpoint is
   an authorized forwarder for a declared remote-cidr — the same category the
   egress gateway already occupies — instead of a VM with all anti-spoofing
   disabled. Port security stays on everywhere else; the exception is narrow and
   declared ("this Port may source/sink 10.50.0.0/16").
2. Identity-based routing, not static-IP. The route "remote-cidr → the VPN
   endpoint" resolves to a Port identity (or a managed gateway keyed by VNI),
   so the endpoint can reschedule or change IP and the control plane re-resolves
   it. The route never names an IP.
3. Per-VPC scoping for free. The route lives in one VPC's net_id scope, so
   it is tenant-scoped by construction — no global routing table, and overlapping
   tenant CIDRs still coexist.

The deeper insight: a VPC route table

Today a VPC has exactly two off-net dispositions: the default gateway (internet)
or drop. A site-to-site VPN asks for a third — and once there are three, there is
a per-VPC route table: remote-cidr → vpn-gw, 0.0.0.0/0 → nat-gw, else
drop. The existing NAT gateway is retroactively just one entry in it. The clean
generalization is not to bolt on a VPN but to grow the VPC a small route table
(the AWS VPC-router model), where both egress and VPN are routes to named
next-hops.

Shapes it could take

• Managed VPN gateway (cloud-like). A VPNConnection { remoteCIDR, peer, IKE/ESP or WireGuard params, secretRef } resource; the controller runs the
  tunnel termination in a gateway-class pod, installs the route-table entry, and
  manages HA. The tenant never touches strongSwan. The inbound direction
  (remote-cidr → tenant IP) delivers by identity — structurally the same
  primitive as floating-IP ingress (Floating IPs: 1:1 public-address NAT anchored on the per-VPC gateway (internet-gateway equivalent) #5), with the tunnel as the "external" side.
• BYO endpoint with a scoped grant. For clients who must run their own
  strongSwan (their crypto policy, their gear interop): keep it a tenant VM, but
  grant its Port the narrow "authorized forwarder for remote-cidr" role and
  point the VPC route at its identity. Fixes both status-quo compromises without
  cozyplane owning the crypto.

Leaning managed for the product story, with the scoped-grant path as the escape
hatch for enterprise interop.

What cozyplane does not make disappear (risks)

• The crypto still lives somewhere — IKE/ESP (strongSwan) or WireGuard, in a
  pod or on gateway nodes. cozyplane cleans up routing and authorization, not the
  tunnel itself.
• IPsec vs. the eBPF datapath. IPsec termination is kernel XFRM; reconciling
  XFRM with the tcx / bpf_redirect datapath is a real integration question.
  WireGuard (in-kernel, key-based) is a cleaner fit for a managed tunnel — but
  enterprises frequently require IPsec/IKEv2 to interop with existing gear (the
  very reason strongSwan is common today), so a managed offering likely needs
  IPsec anyway. That integration is the main risk to scope early.
• HA. A single tunnel pod is a SPOF — the same limitation as today's VPN VM
  and as the current single-pod NAT gateway; the distributed/HA gateway story
  applies here too.

Bottom line

cozyplane can do this materially better on authorization (a scoped
authorized-forwarder role vs. disabling port security) and routing
(identity-based, per-VPC route table vs. static-IP namespace routes), and the
inbound path reuses the floating-IP / north-south bridge machinery. The crypto
and the IPsec-vs-datapath integration are the parts that stay genuinely hard.