Skip to content

Commit 2eb5374

Browse files
committed
fix(kubeovn-webhook): make the observation unforgeable and stop advertising an enable path that does not exist
Review found the observation could be silenced and the feature could not be turned on. Both are fixed here; enforcement itself stays out of scope for this step. Rate-limit the observation instead of remembering issuers. Keying on the issuer was the mistake: with RequestClientCert that string comes from the caller, so a bounded set of first-seen values can be filled with 32 fabricated issuers before the API server's first call, permanently suppressing the signal the feature exists to produce, while an unbounded one grows without limit. A rate limit cannot be starved -- a burst costs the current interval and the next real handshake is logged in the one after. The two outcomes are limited separately so a flood of one cannot hide the other, and both lines now carry the peer address, since 'none presented' fires for any certless caller and was not attributable. Say 'verified' rather than 'UNVERIFIED' once enforcement is on, where the handshake really has checked the chain. Stop telling operators to point --client-ca-file at an in-pod path. No such path exists: the pod mounts only its own serving secret, whose ca.crt is this webhook's serving CA, so every value either aborts startup or rejects the API server -- and with failurePolicy: Fail that stops pod creation cluster-wide. Say what is actually true and defer enabling to the step that adds the volume. Correct the claim in cert.go that this webhook configures no ClientCAs. It can now, that pool is read once at startup and never reloaded, and the comment justified skipping reload on the old premise. Record that reloading has to land together with the mount, since neither is usable alone. Cover the observation with tests: rate limiting, the burst it exists to survive, concurrency, both branches, the enforcing wording, and that admission is never blocked by logging. Signed-off-by: Andrei Kvapil <[email protected]>
1 parent 07fd4d9 commit 2eb5374

4 files changed

Lines changed: 263 additions & 43 deletions

File tree

‎packages/system/kubeovn-webhook/images/kubeovn-webhook/cert.go‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,16 @@ import "crypto/tls"
2222
// LoadX509KeyPair therefore cannot observe a partial renewal.
2323
// - The per-handshake cost (a few KB of file I/O plus a PEM/key parse) is
2424
// negligible next to the asymmetric crypto the handshake already performs.
25-
// - The webhook configures no mTLS (no ClientCAs), so GetCertificate's
26-
// limitation of not refreshing client CA pools does not apply.
25+
// - GetCertificate refreshes the serving key pair only. It does not refresh
26+
// tls.Config.ClientCAs, which the handshake reads directly. When
27+
// --client-ca-file is set, main reads that bundle once at startup and the
28+
// pool stays fixed for the process lifetime: rotating the API server's
29+
// client CA then requires restarting these pods, or the webhook keeps
30+
// validating against the retired CA and rejects the API server. With
31+
// failurePolicy: Fail that stops pod creation cluster-wide, so reloading
32+
// the pool has to land together with the mount that makes enforcement
33+
// reachable at all -- neither is usable without the other, and today
34+
// --client-ca-file has no supported in-pod path to point at.
2735
//
2836
// If the mounted files genuinely become unreadable (Secret deleted, permissions
2937
// broken: an operator error, not a normal renewal) the handshake fails loudly

‎packages/system/kubeovn-webhook/images/kubeovn-webhook/main.go‎

Lines changed: 46 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -16,39 +16,46 @@ var (
1616
RoutesGlobal string
1717
)
1818

19-
// Observation state for logClientCert. The set is capped on purpose: with
20-
// tls.RequestClientCert the server accepts whatever certificate a caller
21-
// offers, so the issuer string is attacker-supplied. An unbounded set would
22-
// grow with every fabricated issuer and emit a log line for each — a memory
23-
// and log-volume amplifier reachable by anyone who can open a TCP connection.
24-
// Past the cap the webhook keeps serving and simply stops recording.
25-
const maxSeenClientIssuers = 32
26-
27-
var (
28-
clientIssuerMu sync.Mutex
29-
seenClientIssuers = make(map[string]struct{}, maxSeenClientIssuers)
30-
clientIssuerCapped bool
31-
)
19+
// Observation state for logClientCert.
20+
//
21+
// Deliberately NOT a set keyed on the issuer. With tls.RequestClientCert the
22+
// issuer string is supplied by the caller, so anything keyed on it is both
23+
// unbounded and forgeable: remembering every value grows without limit, and
24+
// remembering only the first N lets anyone who can reach the port fill those N
25+
// with fabricated issuers before the API server's first call and permanently
26+
// suppress the signal this exists to produce. Rate limiting cannot be starved
27+
// that way -- a burst costs the attacker at most the current interval, and the
28+
// next real handshake is logged in the one after it.
29+
//
30+
// The two outcomes are limited separately so a flood of one cannot hide the
31+
// other: "no certificate" and "certificate presented" answer different
32+
// questions, and it is the pair that decides whether enforcement is safe.
33+
const observationInterval = 30 * time.Second
34+
35+
type observationLimiter struct {
36+
mu sync.Mutex
37+
last time.Time
38+
seen bool
39+
}
3240

33-
// noteClientIssuer reports whether this issuer should be logged: true only when
34-
// it is new and there is still room to remember it.
35-
func noteClientIssuer(key string) bool {
36-
clientIssuerMu.Lock()
37-
defer clientIssuerMu.Unlock()
38-
if _, seen := seenClientIssuers[key]; seen {
41+
// allow reports whether this observation should be logged: always the first
42+
// one, then at most one per observationInterval.
43+
func (l *observationLimiter) allow(now time.Time) bool {
44+
l.mu.Lock()
45+
defer l.mu.Unlock()
46+
if l.seen && now.Sub(l.last) < observationInterval {
3947
return false
4048
}
41-
if len(seenClientIssuers) >= maxSeenClientIssuers {
42-
if !clientIssuerCapped {
43-
clientIssuerCapped = true
44-
log.Printf("client certificate: %d distinct issuers observed, no longer recording new ones; if this cap was reached on a quiet cluster the issuers are probably fabricated", maxSeenClientIssuers)
45-
}
46-
return false
47-
}
48-
seenClientIssuers[key] = struct{}{}
49+
l.seen = true
50+
l.last = now
4951
return true
5052
}
5153

54+
var (
55+
noCertObservations observationLimiter
56+
withCertObservations observationLimiter
57+
)
58+
5259
// logClientCert reports, once per distinct issuer, whether the caller presented
5360
// a client certificate and who signed it.
5461
//
@@ -67,17 +74,21 @@ func noteClientIssuer(key string) bool {
6774
// RequireAndVerifyClientCert on a cluster that sends nothing would fail every
6875
// admission call, and this webhook is registered failurePolicy: Fail — so pod
6976
// creation would stop cluster-wide. Observe first, enforce second.
70-
func logClientCert(next http.Handler) http.Handler {
77+
func logClientCert(enforcing bool, next http.Handler) http.Handler {
7178
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
72-
key := "<none>"
79+
issuer := ""
7380
if r.TLS != nil && len(r.TLS.PeerCertificates) > 0 {
74-
key = r.TLS.PeerCertificates[0].Issuer.String()
81+
issuer = r.TLS.PeerCertificates[0].Issuer.String()
7582
}
76-
if noteClientIssuer(key) {
77-
if key == "<none>" {
78-
log.Printf("client certificate: none presented — client-ca-file enforcement is NOT yet safe on this cluster")
83+
now := time.Now()
84+
switch {
85+
case issuer == "" && noCertObservations.allow(now):
86+
log.Printf("client certificate: none presented by %s -- client-ca-file enforcement is NOT yet safe on this cluster", r.RemoteAddr)
87+
case issuer != "" && withCertObservations.allow(now):
88+
if enforcing {
89+
log.Printf("client certificate: presented by %s, issuer %q -- verified against --client-ca-file during the handshake", r.RemoteAddr, issuer)
7990
} else {
80-
log.Printf("client certificate: presented, issuer %q (UNVERIFIED — nothing has checked this certificate, and any caller can claim any issuer). Confirm out of band that this is your API server's CA before pinning it with --client-ca-file", key)
91+
log.Printf("client certificate: presented by %s, issuer %q (UNVERIFIED -- nothing has checked this certificate, and any caller can claim any issuer). Confirm out of band that this is your API server's CA before pinning it with --client-ca-file", r.RemoteAddr, issuer)
8192
}
8293
}
8394
next.ServeHTTP(w, r)
@@ -133,7 +144,7 @@ func main() {
133144
server := &http.Server{
134145
Addr: ":8443",
135146
TLSConfig: tlsConfig,
136-
Handler: logClientCert(mux),
147+
Handler: logClientCert(clientCAFile != "", mux),
137148
ReadHeaderTimeout: 10 * time.Second,
138149
ReadTimeout: 30 * time.Second,
139150
WriteTimeout: 30 * time.Second,
Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
package main
2+
3+
import (
4+
"crypto/tls"
5+
"crypto/x509"
6+
"crypto/x509/pkix"
7+
"log"
8+
"net/http"
9+
"net/http/httptest"
10+
"strings"
11+
"sync"
12+
"testing"
13+
"time"
14+
)
15+
16+
// captureLog redirects the standard logger for the duration of fn and returns
17+
// what was written.
18+
func captureLog(t *testing.T, fn func()) string {
19+
t.Helper()
20+
var buf strings.Builder
21+
flags := log.Flags()
22+
out := log.Writer()
23+
log.SetOutput(&buf)
24+
log.SetFlags(0)
25+
defer func() {
26+
log.SetOutput(out)
27+
log.SetFlags(flags)
28+
}()
29+
fn()
30+
return buf.String()
31+
}
32+
33+
// resetObservations clears the package-level limiters between cases.
34+
func resetObservations() {
35+
noCertObservations = observationLimiter{}
36+
withCertObservations = observationLimiter{}
37+
}
38+
39+
func TestObservationLimiterAllowsFirstThenRateLimits(t *testing.T) {
40+
var l observationLimiter
41+
base := time.Now()
42+
43+
if !l.allow(base) {
44+
t.Fatal("first observation must always be logged")
45+
}
46+
if l.allow(base.Add(observationInterval - time.Nanosecond)) {
47+
t.Error("a second observation inside the interval must be suppressed")
48+
}
49+
if !l.allow(base.Add(observationInterval)) {
50+
t.Error("an observation at the interval boundary must be logged again")
51+
}
52+
}
53+
54+
// A burst of fabricated issuers is the attack the rate limiter exists to
55+
// survive: it may cost the current interval, but it must not suppress
56+
// observation permanently the way a first-come set of issuers would.
57+
func TestObservationSurvivesBurstOfFabricatedIssuers(t *testing.T) {
58+
var l observationLimiter
59+
base := time.Now()
60+
61+
for i := 0; i < 1000; i++ {
62+
l.allow(base)
63+
}
64+
if !l.allow(base.Add(observationInterval)) {
65+
t.Error("observation must resume after the interval, whatever the burst")
66+
}
67+
}
68+
69+
func TestObservationLimiterIsConcurrencySafe(t *testing.T) {
70+
var l observationLimiter
71+
now := time.Now()
72+
73+
var wg sync.WaitGroup
74+
var mu sync.Mutex
75+
allowed := 0
76+
for i := 0; i < 50; i++ {
77+
wg.Add(1)
78+
go func() {
79+
defer wg.Done()
80+
if l.allow(now) {
81+
mu.Lock()
82+
allowed++
83+
mu.Unlock()
84+
}
85+
}()
86+
}
87+
wg.Wait()
88+
89+
if allowed != 1 {
90+
t.Errorf("exactly one concurrent observation may pass at a fixed instant, got %d", allowed)
91+
}
92+
}
93+
94+
// request builds a request carrying either no peer certificate or one issued by
95+
// the named issuer.
96+
func request(issuer string) *http.Request {
97+
r := httptest.NewRequest(http.MethodPost, "/mutate-pods", nil)
98+
r.RemoteAddr = "10.0.0.7:34512"
99+
if issuer != "" {
100+
r.TLS = &tls.ConnectionState{PeerCertificates: []*x509.Certificate{
101+
{Issuer: pkix.Name{CommonName: issuer}},
102+
}}
103+
}
104+
return r
105+
}
106+
107+
func TestLogClientCertNoCertificate(t *testing.T) {
108+
resetObservations()
109+
next := http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})
110+
111+
out := captureLog(t, func() {
112+
logClientCert(false, next).ServeHTTP(httptest.NewRecorder(), request(""))
113+
})
114+
115+
if !strings.Contains(out, "none presented") {
116+
t.Errorf("a certless caller must be reported, got %q", out)
117+
}
118+
if !strings.Contains(out, "10.0.0.7:34512") {
119+
t.Errorf("the peer address must be logged so the caller can be attributed, got %q", out)
120+
}
121+
}
122+
123+
func TestLogClientCertUnverifiedWhenNotEnforcing(t *testing.T) {
124+
resetObservations()
125+
next := http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})
126+
127+
out := captureLog(t, func() {
128+
logClientCert(false, next).ServeHTTP(httptest.NewRecorder(), request("some-ca"))
129+
})
130+
131+
if !strings.Contains(out, "UNVERIFIED") {
132+
t.Errorf("without enforcement the issuer is a claim and must be marked unverified, got %q", out)
133+
}
134+
if !strings.Contains(out, "some-ca") {
135+
t.Errorf("the issuer must appear in the log, got %q", out)
136+
}
137+
}
138+
139+
func TestLogClientCertVerifiedWhenEnforcing(t *testing.T) {
140+
resetObservations()
141+
next := http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})
142+
143+
out := captureLog(t, func() {
144+
logClientCert(true, next).ServeHTTP(httptest.NewRecorder(), request("some-ca"))
145+
})
146+
147+
if strings.Contains(out, "UNVERIFIED") {
148+
t.Errorf("with enforcement on the handshake has verified the chain; the warning is wrong, got %q", out)
149+
}
150+
if !strings.Contains(out, "verified against --client-ca-file") {
151+
t.Errorf("the log should say the certificate was verified, got %q", out)
152+
}
153+
}
154+
155+
// The two outcomes are limited separately, so a caller flooding one branch
156+
// cannot hide the other. That pairing is what decides whether enforcement is
157+
// safe, so losing either half would defeat the observation.
158+
func TestLogClientCertBranchesAreLimitedIndependently(t *testing.T) {
159+
resetObservations()
160+
next := http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})
161+
h := logClientCert(false, next)
162+
163+
out := captureLog(t, func() {
164+
for i := 0; i < 20; i++ {
165+
h.ServeHTTP(httptest.NewRecorder(), request("flooding-ca"))
166+
}
167+
h.ServeHTTP(httptest.NewRecorder(), request(""))
168+
})
169+
170+
// "none presented by ..." also contains "presented by", so key on the
171+
// issuer field, which only the certificate branch emits.
172+
if strings.Count(out, ", issuer ") != 1 {
173+
t.Errorf("the flooded branch must be rate limited to one line, got %q", out)
174+
}
175+
if !strings.Contains(out, "none presented") {
176+
t.Errorf("the other branch must still be reported despite the flood, got %q", out)
177+
}
178+
}
179+
180+
// The handler is observation only: it must never interfere with admission.
181+
func TestLogClientCertAlwaysCallsNext(t *testing.T) {
182+
resetObservations()
183+
called := 0
184+
next := http.HandlerFunc(func(http.ResponseWriter, *http.Request) { called++ })
185+
h := logClientCert(false, next)
186+
187+
captureLog(t, func() {
188+
h.ServeHTTP(httptest.NewRecorder(), request(""))
189+
h.ServeHTTP(httptest.NewRecorder(), request("some-ca"))
190+
h.ServeHTTP(httptest.NewRecorder(), request("some-ca"))
191+
})
192+
193+
if called != 3 {
194+
t.Errorf("every request must reach the handler regardless of logging, got %d of 3", called)
195+
}
196+
}

‎packages/system/kubeovn-webhook/values.yaml‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,16 @@ routes: ""
1414
# "client certificate: presented, issuer ..." → enforcement is possible
1515
# "client certificate: none presented" → it is not, yet
1616
#
17-
# Set it to the in-pod path of a CA bundle to switch to
18-
# RequireAndVerifyClientCert. Do this ONLY after the log confirms the API
19-
# server presents a certificate: this webhook is registered with
20-
# failurePolicy: Fail, so rejecting the API server stops pod creation
21-
# cluster-wide. The API server sends a certificate only when the cluster runs
22-
# with an AdmissionConfiguration that names a kubeConfigFile for this webhook.
17+
# Leave it empty. There is nothing to point it at yet: the pod mounts only its
18+
# own serving secret at /etc/webhook/certs, and the ca.crt in there is this
19+
# webhook's serving CA, not the API server's client CA. A path that does not
20+
# exist aborts startup; that ca.crt starts cleanly and then rejects every
21+
# API-server handshake. Either way this webhook is registered with
22+
# failurePolicy: Fail, so pod creation stops cluster-wide.
23+
#
24+
# Enabling enforcement is the second step, and it lands as one change: a volume
25+
# carrying the API server's client CA, a pool that reloads when that CA rotates,
26+
# and this key pointed at the mounted path. Until then the flag exists so the
27+
# observation above can run, not so it can be switched on.
2328
clientCAFile: ""
2429
image: ghcr.io/cozystack/cozystack/kubeovn-webhook:v1.6.0@sha256:593c324a38db59495497c5b68b90cad25dfe6753c73b7a287b3280e26f70a15f

0 commit comments

Comments
 (0)