Skip to content

Commit 96a84de

Browse files
authored
fix(backend): handle IPv6 single-stack in outbound URL and TLS cert validation (#10418)
* fix(backend): handle IPv6 single-stack in outbound URL + TLS cert validation On IPv6-only clusters the dashboard backend failed TLS hostname validation against the Kubernetes API server (ERR_TLS_CERT_ALTNAME_INVALID), returning HTTP 500 for /api/config, storageclasses, etc. Two distinct failure modes: - Pattern 1 (truncated host): raw IPv6 literals were concatenated into host:port URLs without brackets, so the first colon truncated the host to its first hextet (e.g. fc00:30::1 -> host=fc00). - Pattern 2 (full host, still rejected): outbound calls set no custom checkServerIdentity, so Node's default IP identity check rejected a correctly-formed IPv6 host whose cert SAN was present but in a different canonical form (compressed vs expanded, or listed as a DNS SAN). Adds backend/src/utils/networkUtils.ts with: - bracketHostIfIPv6 to safely wrap IPv6 literals during URL construction - normalizeIPv6 to canonicalize addresses for comparison - ipv6AwareCheckServerIdentity, which defers to Node's default check and only falls back to normalized IPv6 SAN comparison for IPv6 hosts Routes outbound URL builders (proxy, feature store ClusterIP) through the bracket helper, and wires the IPv6-aware identity check into proxyCall and the feature store HTTPS request. Non-IPv6 hosts are completely unaffected. * fix(backend): address review - structured IPv6 cert check, fix Feast host Addresses PR review findings: - Cert identity (CWE-295): replace the hand-rolled SAN string parsing and the DNS-SAN fallback with Node's structured X509Certificate.checkIP(), which matches an IPv6 host only against real iPAddress SANs with proper canonicalization. This is fail-closed: no comma-splitting of subjectaltname (so a comma inside a quoted SAN can't masquerade as an IP entry), and a DNS SAN never authorizes an IP host. - Feast IPv6 request: makeAuthenticatedHttpRequest now passes the URL string to http(s).request so Node strips the IPv6 brackets when deriving the connect host (URL.hostname keeps brackets, which the hostname option would reject). Keeps bracketHostIfIPv6 in URL construction. - Drop the no-op checkServerIdentity from the Feast path (it uses rejectUnauthorized: false, so a custom identity check has no effect). Tests rewritten to use a real certificate fixture for the structured check, including a regression that a DNS-only SAN is rejected for an IP host. * test(backend): update Feast https.request mock for url-first signature makeAuthenticatedHttpRequest now calls request(url, options, callback); the mock previously assumed request(options, callback), so the response callback was never invoked. Read the callback from the last argument. * test(backend): cover proxyCall IPv6 checkServerIdentity wiring Addresses review follow-up: adds integration tests that proxyCall installs ipv6AwareCheckServerIdentity on https requests (including when a proxy agent is present, since the agent tunnels TLS to the origin) and omits it for http targets. * fix(backend): scope IPv6 cert check to IPv6 hosts; real-cert tests Addresses remaining review findings: - Scope checkServerIdentity to IPv6 destinations: proxyCall now only installs ipv6AwareCheckServerIdentity when the target host is an IPv6 literal (new isIPv6Url helper). DNS and IPv4 https targets keep Node's default identity check untouched (CodeRabbit). - Add representative tests that exercise the REAL Node identity checker (no tls mock) against real certificate fixtures: an IPv6 host is accepted against an IP Address SAN, and rejected when the address is only a DNS SAN or matches no SAN. Documents that current Node already canonicalizes IP SANs by default, so the structured fallback is defensive hardening (Fullsend test-inadequate). - Cover the IPv6-only wiring in proxyCall (IPv6 https sets the check; DNS https and http do not). * fix(backend): skip IPv6 cert check on proxied (E2E) connections Addresses review: only install ipv6AwareCheckServerIdentity for direct IPv6 HTTPS connections, not when a proxy agent is in use. getProxyAgent only returns an agent under E2E_USE_PROXY_FROM_ENV (E2E-only), and a tunneled TLS connection may not surface the origin host to the identity callback, so we leave Node's default check in place there rather than risk validating against the proxy host. The production IPv6-cluster path (no proxy) is unchanged. * fix(backend): apply IPv6 cert check to makeApiClient k8s clients too The IPv6-aware checkServerIdentity was only wired into proxyCall. The typed Kubernetes clients from kc.makeApiClient(...) use @kubernetes/client-node's own 'request' transport, which never received it — so startup reads (plugins/kube.ts) and the dashboard-config watcher (resourceUtils.ts) could still fail with ERR_TLS_CERT_ALTNAME_INVALID on IPv6 single-stack clusters. Root cause confirmed by reviewers: Node 22.23.1/.2 regression nodejs/node#64144 mis-normalizes IPv6 literals in the default identity check (fixed in later 22.x), so the fallback is genuinely load-bearing on the shipped UBI9 Node. Adds applyIpv6AwareIdentityCheck(kc): wraps KubeConfig.applyToRequest to attach an https.Agent carrying the kube TLS material plus ipv6AwareCheckServerIdentity. A pre-built agent is required because 'request' drops a bare checkServerIdentity option and pools agents by a key that ignores it. Applied to the shared kc and the per-user Feast userKc. Verified end-to-end against real client-node + makeApiClient; added unit coverage for the wiring. * fix(backend): only attach IPv6 TLS agent for HTTPS API servers Addresses review: applyIpv6AwareIdentityCheck attached an https.Agent unconditionally. When KubeConfig.loadFromDefault() falls back to http://localhost:8080, assigning an HTTPS agent to an HTTP request makes Node reject it with ERR_INVALID_PROTOCOL. Now no-op unless the current cluster server is https. Added unit coverage for the http-fallback and no-cluster cases.
1 parent 243f25f commit 96a84de

8 files changed

Lines changed: 541 additions & 13 deletions

File tree

‎backend/src/__tests__/featureStoreUtils.spec.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,9 @@ const spyKubeConfigForMockApi = (getMockApi: () => ReturnType<typeof createMockA
120120
({
121121
loadFromClusterAndUser,
122122
makeApiClient: jest.fn().mockReturnValue(getMockApi()),
123+
// applyIpv6AwareIdentityCheck wraps applyToRequest and reads getCurrentCluster().server.
124+
applyToRequest: jest.fn(),
125+
getCurrentCluster: () => ({ server: 'https://test' }),
123126
} as any),
124127
);
125128
return { loadFromClusterAndUser };
@@ -136,7 +139,10 @@ const mockHttpsJsonResponse = (
136139
body: unknown,
137140
options: { requestError?: Error } = {},
138141
) => {
139-
mockHttpsRequest.mockImplementation((_options, callback) => {
142+
// `makeAuthenticatedHttpRequest` calls `request(url, options, callback)`, so the response callback
143+
// is always the last argument regardless of how many leading args are passed.
144+
mockHttpsRequest.mockImplementation((...args) => {
145+
const callback = args[args.length - 1] as ((res: unknown) => void) | undefined;
140146
const req = new EventEmitter() as any;
141147
req.end = jest.fn();
142148
req.setTimeout = jest.fn();

‎backend/src/plugins/kube.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,15 @@ import * as k8s from '@kubernetes/client-node';
66
import { errorHandler, isKubeFastifyInstance } from '../utils';
77
import { DEV_MODE } from '../utils/constants';
88
import { cleanupKserveRoleBindings, initializeWatchedResources } from '../utils/resourceUtils';
9+
import { applyIpv6AwareIdentityCheck } from '../utils/networkUtils';
910

1011
const CONSOLE_CONFIG_YAML_FIELD = 'console-config.yaml';
1112

1213
const kc = new k8s.KubeConfig();
1314
kc.loadFromDefault();
15+
// Ensure the typed API clients created below validate IPv6 API-server certs correctly (the fix
16+
// wired into proxyCall does not cover @kubernetes/client-node's own request transport).
17+
applyIpv6AwareIdentityCheck(kc);
1418

1519
const currentContext = kc.getCurrentContext();
1620
const customObjectsApi = kc.makeApiClient(k8s.CustomObjectsApi);

‎backend/src/routes/api/featurestores/featureStoreUtils.ts‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { KubeFastifyInstance } from '../../../types';
55
import { DEV_MODE } from '../../../utils/constants';
66
import { isImpersonating, getImpersonateAccessToken } from '../../../devFlags';
77
import { getAccessToken } from '../../../utils/directCallUtils';
8+
import { applyIpv6AwareIdentityCheck, bracketHostIfIPv6 } from '../../../utils/networkUtils';
89

910
const REGISTRY_CONDITION_TYPE = 'Registry';
1011

@@ -109,6 +110,7 @@ function getUserScopedCustomObjectsApi(
109110

110111
const userKc = new k8s.KubeConfig();
111112
userKc.loadFromClusterAndUser(cluster, { name: 'current-user', token });
113+
applyIpv6AwareIdentityCheck(userKc);
112114

113115
return {
114116
api: userKc.makeApiClient(k8s.CustomObjectsApi),
@@ -498,7 +500,9 @@ export function constructRegistryProxyUrl(
498500
process.env.FEAST_REGISTRY_SERVICE_HOST ||
499501
`${serviceName}.${serviceNamespace}.svc.cluster.local`;
500502
const envPort = process.env.FEAST_REGISTRY_SERVICE_PORT || port;
501-
return `${protocol}://${host}:${envPort}/${path}`;
503+
// FEAST_REGISTRY_SERVICE_HOST is the service ClusterIP, which is a raw IPv6 literal on IPv6
504+
// single-stack clusters and must be bracketed before concatenating `:port`.
505+
return `${protocol}://${bracketHostIfIPv6(host)}:${envPort}/${path}`;
502506
}
503507

504508
return `${protocol}://${serviceName}.${serviceNamespace}.svc.cluster.local:${port}/${path}`;
@@ -554,10 +558,7 @@ export async function makeAuthenticatedHttpRequest<T = unknown>(
554558
): Promise<{ data: T; statusCode: number }> {
555559
const { timeout = 60000, rejectUnauthorized = false, agent } = options;
556560

557-
const urlObj = new URL(url);
558-
559-
const isHttps = urlObj.protocol === 'https:';
560-
const defaultPort = isHttps ? 443 : 80;
561+
const isHttps = new URL(url).protocol === 'https:';
561562

562563
const headers: Record<string, string> = {
563564
'Content-Type': 'application/json',
@@ -568,9 +569,6 @@ export async function makeAuthenticatedHttpRequest<T = unknown>(
568569
}
569570

570571
const requestOptions = {
571-
hostname: urlObj.hostname,
572-
port: urlObj.port || defaultPort,
573-
path: urlObj.pathname + urlObj.search,
574572
method: 'GET',
575573
headers,
576574
rejectUnauthorized,
@@ -579,7 +577,10 @@ export async function makeAuthenticatedHttpRequest<T = unknown>(
579577

580578
return new Promise((resolve, reject) => {
581579
const requestModule = isHttps ? https : http;
582-
const req = requestModule.request(requestOptions, (res) => {
580+
// Pass the URL string (not a hostname option) so Node strips the surrounding brackets from an
581+
// IPv6 literal when deriving the connect host — `URL.hostname` keeps the brackets, which the
582+
// `hostname` request option would reject.
583+
const req = requestModule.request(url, requestOptions, (res) => {
583584
let data = '';
584585
res.on('data', (chunk) => {
585586
data += chunk;

‎backend/src/utils/__tests__/httpUtils.spec.ts‎

Lines changed: 103 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,19 @@
1+
import { EventEmitter } from 'events';
2+
import https from 'https';
3+
import http from 'http';
14
import { HttpsProxyAgent } from 'https-proxy-agent';
2-
import { getProxyAgent } from '../httpUtils';
5+
import { getProxyAgent, proxyCall } from '../httpUtils';
6+
import { ipv6AwareCheckServerIdentity } from '../networkUtils';
7+
import { getDirectCallOptions } from '../directCallUtils';
8+
import { KubeFastifyInstance, OauthFastifyRequest } from '../../types';
9+
10+
jest.mock('https');
11+
jest.mock('http');
12+
jest.mock('../directCallUtils');
13+
jest.mock('../constants', () => ({
14+
...jest.requireActual('../constants'),
15+
DEV_MODE: true, // allow the http-target branch of proxyCall to run
16+
}));
317

418
describe('getProxyAgent', () => {
519
const originalEnv = process.env;
@@ -44,3 +58,91 @@ describe('getProxyAgent', () => {
4458
expect(getProxyAgent('http://api.example.test')).toBeUndefined();
4559
});
4660
});
61+
62+
describe('proxyCall', () => {
63+
const mockHttpsRequest = https.request as jest.Mock;
64+
const mockHttpRequest = http.request as jest.Mock;
65+
const mockGetDirectCallOptions = jest.mocked(getDirectCallOptions);
66+
67+
const fastify = {
68+
log: { info: jest.fn(), error: jest.fn() },
69+
} as unknown as KubeFastifyInstance;
70+
const request = { headers: {} } as unknown as OauthFastifyRequest;
71+
72+
const originalEnv = process.env;
73+
74+
// Capture the options the outbound request was built with.
75+
let capturedOptions: https.RequestOptions | undefined;
76+
77+
const installRequestMock = (mock: jest.Mock) => {
78+
mock.mockImplementation((_url, options, callback) => {
79+
capturedOptions = options;
80+
const res = new EventEmitter() as unknown as http.IncomingMessage & {
81+
setEncoding: () => void;
82+
};
83+
res.setEncoding = () => res;
84+
process.nextTick(() => {
85+
callback(res);
86+
res.emit('end');
87+
});
88+
const req = new EventEmitter() as unknown as EventEmitter & {
89+
write: () => void;
90+
end: () => void;
91+
};
92+
req.write = jest.fn();
93+
req.end = jest.fn();
94+
return req;
95+
});
96+
};
97+
98+
beforeEach(() => {
99+
jest.clearAllMocks();
100+
capturedOptions = undefined;
101+
process.env = { ...originalEnv };
102+
delete process.env.E2E_USE_PROXY_FROM_ENV;
103+
mockGetDirectCallOptions.mockResolvedValue({ headers: {} });
104+
installRequestMock(mockHttpsRequest);
105+
installRequestMock(mockHttpRequest);
106+
});
107+
108+
afterEach(() => {
109+
process.env = originalEnv;
110+
});
111+
112+
it('should wire the IPv6-aware checkServerIdentity for IPv6 https targets', async () => {
113+
await proxyCall(fastify, request, { method: 'GET', url: 'https://[fc00:30::1]/api/config' });
114+
115+
expect(mockHttpsRequest).toHaveBeenCalledTimes(1);
116+
expect(capturedOptions?.checkServerIdentity).toBe(ipv6AwareCheckServerIdentity);
117+
});
118+
119+
it('should not set checkServerIdentity for non-IPv6 (DNS) https targets', async () => {
120+
await proxyCall(fastify, request, {
121+
method: 'GET',
122+
url: 'https://kubernetes.default.svc/api/config',
123+
});
124+
125+
expect(mockHttpsRequest).toHaveBeenCalledTimes(1);
126+
expect(capturedOptions?.checkServerIdentity).toBeUndefined();
127+
});
128+
129+
it('should not install the IPv6-aware check when a proxy agent is used', async () => {
130+
process.env.E2E_USE_PROXY_FROM_ENV = 'true';
131+
process.env.HTTPS_PROXY = 'http://proxy.example.invalid:3128';
132+
133+
await proxyCall(fastify, request, { method: 'GET', url: 'https://[fc00:30::1]/api/config' });
134+
135+
// Through the (E2E-only) proxy agent the tunneled TLS connection may not surface the origin host
136+
// to the identity callback, so we leave Node's default check in place to avoid validating against
137+
// the proxy host.
138+
expect(capturedOptions?.agent).toBeInstanceOf(HttpsProxyAgent);
139+
expect(capturedOptions?.checkServerIdentity).toBeUndefined();
140+
});
141+
142+
it('should not set checkServerIdentity for http targets', async () => {
143+
await proxyCall(fastify, request, { method: 'GET', url: 'http://localhost:8080/api/config' });
144+
145+
expect(mockHttpRequest).toHaveBeenCalledTimes(1);
146+
expect(capturedOptions?.checkServerIdentity).toBeUndefined();
147+
});
148+
});

0 commit comments

Comments
 (0)