Skip to content

Commit 8d2773c

Browse files
dbbvitorclaude
dbbvitor
and
claude
authored
fix: Bind metrics, REST registry, ui, and lineage servers dual-stack (#6886)
* fix: Bind metrics and REST registry servers dual-stack The Prometheus metrics server and the REST registry server both hardcode IPv4-only binds ("" and "0.0.0.0"), so neither is reachable on IPv6-only or dual-stack clusters, unlike the gRPC registry server which already binds "[::]" unconditionally. Add an IPv6 probe (feast.utils._ipv6_available) and bind both servers to the IPv6 wildcard when available, falling back to IPv4-only on hosts without IPv6 support (e.g. kernels booted with ipv6.disable=1, where socket(AF_INET6) raises EAFNOSUPPORT). Co-Authored-By: Claude Sonnet 5 <[email protected]> Signed-off-by: dbbvitor <[email protected]> * style: Add type hints to _make_metrics_httpd, restore stray formatting Add type hints to _make_metrics_httpd per AGENTS.md, and hoist its stdlib wsgiref/socket imports to module level since there's no import-cycle risk. Also restore a blank line in test_metrics.py that a mis-resolved ruff (stale 0.6.9 from PATH, not the pinned 0.16.0) stripped during an earlier commit's pre-commit hook run. Signed-off-by: dbbvitor <[email protected]> * fix: Bind the REST registry server via a pre-built dual-stack socket uvicorn's host="::" binds IPv6-only: asyncio's loop.create_server() sets IPV6_V6ONLY=1 on any socket it creates itself from a host string, which would silently drop IPv4 clients on every host, not just IPv6-less ones. Bind the socket ourselves (IPV6_V6ONLY=0, falling back to IPv4-only when the host has no IPv6) and hand it to uvicorn via Server.run(sockets=...), the same pattern used for socket-activated Gunicorn workers. Signed-off-by: dbbvitor <[email protected]> * fix: Bind the ui and lineage servers dual-stack too start_server (ui_server.py) and start_lineage_server (lineage_server.py) both call uvicorn.run(app, host=host, port=port) directly. A plain host="::" there binds IPv6-only -- asyncio's loop.create_server sets IPV6_V6ONLY=1 on any socket it creates itself from a host string -- which silently drops IPv4 clients on every host, not just IPv6-less ones. This is the same bug the metrics and REST registry servers had, fixed separately on fix/sdk-dual-stack-binds. Add feast.utils._make_dual_stack_socket(port), a shared version of _make_rest_socket's pre-bound-socket approach, and route both servers through it when host == "::": build the socket with IPV6_V6ONLY cleared and hand it to uvicorn.Server(config).run(sockets=[sock]) instead of uvicorn.run(host=...). Any other host keeps today's plain uvicorn.run behavior unchanged. This closes the gap the Operator's dualStack option (feat/operator-dual-stack) depends on: without this fix, enabling dualStack for the ui/lineage services would have silently regressed their IPv4 reachability instead of adding IPv6. Signed-off-by: dbbvitor <[email protected]> * refactor: Share one dual-stack socket builder across REST, ui, lineage _make_rest_socket and the new _make_dual_stack_socket had identical bodies. Delete _make_rest_socket and have rest_registry_server.py call the shared feast.utils helper instead, so there is one implementation for all three uvicorn-based dual-stack servers. Moves the real (port-0) end-to-end socket reachability test from test_api_rest_registry_server.py to test_utils.py, targeting the shared helper directly -- the ui and lineage code paths now get real IPv4/IPv6 reachability coverage from the same test, not just mocks. Signed-off-by: dbbvitor <[email protected]> * fix: Close the dual-stack socket on bind failure, cache IPv6 probe An unguarded bind()/listen() leaked the socket fd on failure; wrap the setup in try/except so it closes before re-raising. Also cache the IPv6 support probe with lru_cache since it can't change at runtime. Signed-off-by: dbbvitor <[email protected]> * feat: Expose a host override on the REST registry server RestRegistryServer.start_server() gains a host param, defaulting to "::" (dual-stack, unchanged behavior). Any other value falls back to plain uvicorn.run(host=host), letting deployments opt out of dual-stack binding. Threaded through FeatureStore.serve_registry() and a new --host/-h CLI flag on serve_registry. Signed-off-by: dbbvitor <[email protected]> * fix: Add a blocking parameter to _make_dual_stack_socket setblocking(False) baked in an asyncio assumption invisible from the function name, so a caller expecting a blocking socket would fail silently. Default stays False for the existing asyncio-based callers. Signed-off-by: dbbvitor <[email protected]> --------- Signed-off-by: dbbvitor <[email protected]> Co-authored-by: Claude Sonnet 5 <[email protected]>
1 parent c5efecd commit 8d2773c

14 files changed

Lines changed: 788 additions & 35 deletions

‎sdk/python/feast/api/registry/rest/rest_registry_server.py‎

Lines changed: 25 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
str_to_auth_manager_type,
2424
)
2525
from feast.registry_server import RegistryServer
26-
from feast.utils import _utc_now
26+
from feast.utils import _make_dual_stack_socket, _utc_now
2727

2828
logger = logging.getLogger(__name__)
2929
logger.setLevel(logging.INFO)
@@ -322,30 +322,40 @@ def _init_auth(self):
322322
def start_server(
323323
self,
324324
port: int,
325+
host: str = "::",
325326
tls_key_path: str = "",
326327
tls_cert_path: str = "",
327328
):
329+
"""Start the REST registry server. host="::" (default) binds dual-stack
330+
via a pre-built socket; any other host falls back to plain uvicorn.run."""
328331
import uvicorn
329332

330333
from feast.registry_server import _sync_protected_project_tag
331334

332335
_sync_protected_project_tag(self.store)
333336

337+
ssl_kwargs: dict = {}
334338
if tls_key_path and tls_cert_path:
335-
logger.info("Starting REST registry server in TLS(SSL) mode")
336-
logger.info(f"REST registry server listening on https://localhost:{port}")
337-
uvicorn.run(
338-
self.app,
339-
host="0.0.0.0",
340-
port=port,
341-
ssl_keyfile=tls_key_path,
342-
ssl_certfile=tls_cert_path,
339+
ssl_kwargs = {"ssl_keyfile": tls_key_path, "ssl_certfile": tls_cert_path}
340+
scheme = "https"
341+
else:
342+
scheme = "http"
343+
344+
if host == "::":
345+
logger.info(
346+
f"Starting REST registry server in "
347+
f"{'TLS(SSL)' if ssl_kwargs else 'non-TLS(SSL)'} mode"
348+
)
349+
logger.info(
350+
f"REST registry server listening on {scheme}://localhost:{port}"
343351
)
352+
sock = _make_dual_stack_socket(port)
353+
config = uvicorn.Config(self.app, **ssl_kwargs)
354+
uvicorn.Server(config).run(sockets=[sock])
344355
else:
345-
logger.info("Starting REST registry server in non-TLS(SSL) mode")
346-
logger.info(f"REST registry server listening on http://localhost:{port}")
347-
uvicorn.run(
348-
self.app,
349-
host="0.0.0.0",
350-
port=port,
356+
logger.info(
357+
f"Starting REST registry server in "
358+
f"{'TLS(SSL)' if ssl_kwargs else 'non-TLS(SSL)'} mode"
351359
)
360+
logger.info(f"REST registry server listening on {scheme}://{host}:{port}")
361+
uvicorn.run(self.app, host=host, port=port, **ssl_kwargs)

‎sdk/python/feast/cli/serve.py‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,14 @@ def serve_transformations_command(ctx: click.Context, port: int):
225225
show_default=True,
226226
help="Start a REST API Registry Server",
227227
)
228+
@click.option(
229+
"--host",
230+
"-h",
231+
type=click.STRING,
232+
default="::",
233+
show_default=True,
234+
help="Bind address for the REST API Registry Server (ignored for gRPC).",
235+
)
228236
@click.pass_context
229237
def serve_registry_command(
230238
ctx: click.Context,
@@ -234,6 +242,7 @@ def serve_registry_command(
234242
grpc: bool,
235243
rest_api: bool,
236244
rest_port: int,
245+
host: str,
237246
):
238247
"""Start Feast Registry server (gRPC by default, REST opt-in)."""
239248
if (tls_key_path and not tls_cert_path) or (not tls_key_path and tls_cert_path):
@@ -253,7 +262,7 @@ def serve_registry_command(
253262
),
254263
multiprocessing.Process(
255264
target=_serve_rest_registry,
256-
args=(repo_path, rest_port, tls_key_path, tls_cert_path),
265+
args=(repo_path, rest_port, tls_key_path, tls_cert_path, host),
257266
name="rest_registry_server",
258267
),
259268
]
@@ -272,6 +281,7 @@ def serve_registry_command(
272281
tls_key_path=tls_key_path,
273282
tls_cert_path=tls_cert_path,
274283
rest_api=rest_api,
284+
host=host,
275285
)
276286

277287

@@ -289,7 +299,11 @@ def _serve_grpc_registry(
289299

290300

291301
def _serve_rest_registry(
292-
repo_path: str, port: int, tls_key_path: str, tls_cert_path: str
302+
repo_path: str,
303+
port: int,
304+
tls_key_path: str,
305+
tls_cert_path: str,
306+
host: str,
293307
):
294308
from feast import FeatureStore
295309

@@ -299,6 +313,7 @@ def _serve_rest_registry(
299313
tls_key_path=tls_key_path,
300314
tls_cert_path=tls_cert_path,
301315
rest_api=True,
316+
host=host,
302317
)
303318

304319

‎sdk/python/feast/feature_store.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4765,15 +4765,20 @@ def serve_registry(
47654765
tls_key_path: str = "",
47664766
tls_cert_path: str = "",
47674767
rest_api: bool = False,
4768+
host: str = "::",
47684769
) -> None:
4769-
"""Start registry server locally on a given port."""
4770+
"""Start registry server locally on a given port. `host` only applies
4771+
to the REST server (rest_api=True); defaults to dual-stack "::"."""
47704772
if rest_api:
47714773
from feast.api.registry.rest import rest_registry_server
47724774

47734775
server = rest_registry_server.RestRegistryServer(self)
47744776

47754777
server.start_server(
4776-
port=port, tls_key_path=tls_key_path, tls_cert_path=tls_cert_path
4778+
port=port,
4779+
host=host,
4780+
tls_key_path=tls_key_path,
4781+
tls_cert_path=tls_cert_path,
47774782
)
47784783
else:
47794784
from feast import registry_server

‎sdk/python/feast/lineage_server.py‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -256,4 +256,15 @@ def start_lineage_server(
256256
import uvicorn
257257

258258
logger.info(f"Starting Feast OpenLineage server on {scheme}://{host}:{port}")
259-
uvicorn.run(app, host=host, port=port, **ssl_kwargs)
259+
260+
if host == "::":
261+
# Plain uvicorn.run(host="::") binds IPv6-only (see
262+
# feast.utils._make_dual_stack_socket), which would drop IPv4
263+
# clients on every host, not just IPv6-less ones.
264+
from feast.utils import _make_dual_stack_socket
265+
266+
sock = _make_dual_stack_socket(port)
267+
config = uvicorn.Config(app, **ssl_kwargs)
268+
uvicorn.Server(config).run(sockets=[sock])
269+
else:
270+
uvicorn.run(app, host=host, port=port, **ssl_kwargs)

‎sdk/python/feast/metrics.py‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,15 @@
4646
import logging
4747
import os
4848
import shutil
49+
import socket
4950
import tempfile
5051
import threading
5152
import time
5253
from contextlib import contextmanager
5354
from dataclasses import dataclass
5455
from datetime import datetime, timezone
55-
from typing import TYPE_CHECKING, List, Optional
56+
from typing import TYPE_CHECKING, Callable, Iterable, List, Optional
57+
from wsgiref.simple_server import WSGIServer, make_server
5658

5759
import psutil
5860

@@ -583,6 +585,26 @@ def init_worker_freshness_monitoring(store: "FeatureStore"):
583585
t.start()
584586

585587

588+
def _make_metrics_httpd(port: int, app: Callable[..., Iterable[bytes]]) -> WSGIServer:
589+
"""Build the metrics HTTP server, dual-stack ("::") when the host
590+
supports IPv6, or IPv4-only (falls back to `make_server`'s default
591+
behavior) otherwise.
592+
"""
593+
from feast import utils
594+
595+
if not utils._ipv6_available():
596+
return make_server("", port, app)
597+
598+
class DualStackWSGIServer(WSGIServer):
599+
address_family = socket.AF_INET6
600+
601+
def server_bind(self) -> None:
602+
self.socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
603+
super().server_bind()
604+
605+
return make_server("::", port, app, server_class=DualStackWSGIServer)
606+
607+
586608
def start_metrics_server(
587609
store: "FeatureStore",
588610
port: int = 8000,
@@ -633,9 +655,7 @@ def start_metrics_server(
633655
registry = CollectorRegistry()
634656
MultiProcessCollector(registry)
635657

636-
from wsgiref.simple_server import make_server
637-
638-
httpd = make_server("", port, make_wsgi_app(registry))
658+
httpd = _make_metrics_httpd(port, make_wsgi_app(registry))
639659
metrics_thread = threading.Thread(target=httpd.serve_forever, daemon=True)
640660
metrics_thread.start()
641661
logger.info(

‎sdk/python/feast/ui_server.py‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1216,13 +1216,19 @@ def start_server(
12161216

12171217
logger.info(f"Starting Feast UI server on {host}:{port}")
12181218

1219+
ssl_kwargs: dict = {}
12191220
if tls_key_path and tls_cert_path:
1220-
uvicorn.run(
1221-
app,
1222-
host=host,
1223-
port=port,
1224-
ssl_keyfile=tls_key_path,
1225-
ssl_certfile=tls_cert_path,
1226-
)
1221+
ssl_kwargs["ssl_keyfile"] = tls_key_path
1222+
ssl_kwargs["ssl_certfile"] = tls_cert_path
1223+
1224+
if host == "::":
1225+
# Plain uvicorn.run(host="::") binds IPv6-only (see
1226+
# feast.utils._make_dual_stack_socket), which would drop IPv4
1227+
# clients on every host, not just IPv6-less ones.
1228+
from feast.utils import _make_dual_stack_socket
1229+
1230+
sock = _make_dual_stack_socket(port)
1231+
config = uvicorn.Config(app, **ssl_kwargs)
1232+
uvicorn.Server(config).run(sockets=[sock])
12271233
else:
1228-
uvicorn.run(app, host=host, port=port)
1234+
uvicorn.run(app, host=host, port=port, **ssl_kwargs)

‎sdk/python/feast/utils.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
import copy
2+
import functools
23
import itertools
34
import logging
45
import os
6+
import socket
57
import threading
68
import typing
79
import warnings
@@ -63,6 +65,50 @@
6365
USER_AGENT = "{}/{}".format(APPLICATION_NAME, get_version())
6466

6567

68+
@functools.lru_cache()
69+
def _ipv6_available() -> bool:
70+
"""Whether this process can bind AF_INET6 (cached: can't change at runtime)."""
71+
try:
72+
with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as sock:
73+
sock.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
74+
sock.bind(("::", 0))
75+
return True
76+
except OSError:
77+
return False
78+
79+
80+
def _make_dual_stack_socket(port: int, blocking: bool = False) -> socket.socket:
81+
"""Pre-bind a dual-stack socket for a server to adopt.
82+
83+
A framework handed host="::" directly (uvicorn.run, loop.create_server)
84+
sets IPV6_V6ONLY=1 on its own socket, silently dropping IPv4 clients.
85+
Binding here with V6ONLY cleared avoids that; falls back to IPv4-only
86+
when the host can't bind AF_INET6.
87+
88+
Non-blocking by default since every current caller hands the socket to
89+
an asyncio-based server (uvicorn.Server(...).run(sockets=[sock])), which
90+
requires that; pass blocking=True for a synchronous server instead.
91+
"""
92+
dual_stack = _ipv6_available()
93+
if dual_stack:
94+
sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
95+
address: tuple = ("::", port)
96+
else:
97+
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
98+
address = ("0.0.0.0", port)
99+
try:
100+
if dual_stack:
101+
sock.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
102+
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
103+
sock.bind(address)
104+
sock.listen(socket.SOMAXCONN)
105+
sock.setblocking(blocking)
106+
except Exception:
107+
sock.close()
108+
raise
109+
return sock
110+
111+
66112
def _parse_feature_or_view_ref(ref: str) -> Tuple[str, Optional[int], Optional[str]]:
67113
"""Parse 'fv_name[@version][:feature]' into (fv_name, version_number, feature_name).
68114

0 commit comments

Comments
 (0)