Skip to content

Commit afff726

Browse files
jyejarentkatholecursoragentcasaar97Carlos Sánchez
authored
chore: Manually Sync upstream master to midstream as auto-sync failed (#166)
* feat: Multi-arch publish for feast operator image Signed-off-by: ntkathole <[email protected]> Co-authored-by: Cursor <[email protected]> * feat: Allow users to have protected project on shared registry Signed-off-by: ntkathole <[email protected]> * fix: Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock (feast-dev#6648) feast_metrics.start_metrics_server() starts a feature-freshness thread in the Gunicorn master process, before Gunicorn forks its worker(s). That thread's first action, with no delay, is update_feature_freshness() -> store.list_feature_views(), which lazily builds the registry for the first time. For registry backends needing a lazy DBAPI import (e.g. the SQL registry importing pymysql via SQLAlchemy's create_engine()), this means a thread in the master can be mid-import, holding CPython's per-module import lock, at the exact moment Gunicorn forks a worker. POSIX fork() only duplicates the calling thread into the child process; every other thread in the parent, including this one, simply ceases to exist in the worker. If the fork lands while that thread holds a module's import lock, the lock stays permanently held in the new worker, since there is no longer any thread that can finish the import and release it. The worker's own later attempt to build its registry then deadlocks forever with no error - the process just hangs at "Waiting for application startup." This is intermittent by nature: it only manifests if the fork lands inside that narrow timing window. Resource monitoring already avoids this correctly (start_resource_monitoring= not uses_gunicorn plus the post_worker_init hook calling init_worker_monitoring()), but the freshness thread was not given the same treatment. This applies the identical pattern: start_metrics_server gains a start_freshness_monitoring flag (deferred exactly like resource monitoring's), and FeastServeApplication's post_worker_init hook now also calls the new init_worker_freshness_monitoring(store) after the fork, instead of feast_metrics.py starting it unconditionally beforehand. Fixes feast-dev#6647 Signed-off-by: Carlos Sánchez <[email protected]> Co-authored-by: Carlos Sánchez <[email protected]> --------- Signed-off-by: ntkathole <[email protected]> Signed-off-by: Carlos Sánchez <[email protected]> Co-authored-by: ntkathole <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Carlos Sánchez <[email protected]> Co-authored-by: Carlos Sánchez <[email protected]>
1 parent 876f6ea commit afff726

15 files changed

Lines changed: 287 additions & 29 deletions

File tree

‎.github/workflows/master_only.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -102,8 +102,8 @@ jobs:
102102
push_mode: all_tags
103103
- component: feast-operator
104104
target: feast-operator
105-
build_args: ""
106-
push_mode: all_tags
105+
build_args: DOCKER_PUSH=true DOCKER_PLATFORMS=linux/amd64,linux/arm64
106+
push_mode: imagetools
107107
env:
108108
REGISTRY: quay.io/feastdev-ci
109109
steps:
@@ -135,7 +135,7 @@ jobs:
135135
- name: Push image
136136
run: |
137137
if [[ "${{ matrix.push_mode }}" == "imagetools" ]]; then
138-
docker buildx imagetools create -t ${REGISTRY}/feature-server:develop ${REGISTRY}/feature-server:${GITHUB_SHA}
138+
docker buildx imagetools create -t ${REGISTRY}/${{ matrix.target }}:develop ${REGISTRY}/${{ matrix.target }}:${GITHUB_SHA}
139139
else
140140
docker tag ${REGISTRY}/${{ matrix.target }}:${GITHUB_SHA} ${REGISTRY}/${{ matrix.target }}:develop && docker push ${REGISTRY}/${{ matrix.target }} --all-tags
141141
fi

‎.github/workflows/publish_images.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ jobs:
7474
env:
7575
VERSION_WITHOUT_PREFIX: ${{ steps.get-version.outputs.version_without_prefix }}
7676
run: |
77-
if [ "${{ matrix.component }}" = "feature-server" ]; then
77+
if [[ "${{ matrix.component }}" == "feature-server" || "${{ matrix.component }}" == "feast-operator" ]]; then
7878
make build-${{ matrix.component }}-docker REGISTRY=${REGISTRY} VERSION=${VERSION_WITHOUT_PREFIX} DOCKER_PUSH=true DOCKER_PLATFORMS=linux/amd64,linux/arm64
7979
else
8080
make build-${{ matrix.component }}-docker REGISTRY=${REGISTRY} VERSION=${VERSION_WITHOUT_PREFIX}
@@ -84,17 +84,17 @@ jobs:
8484
VERSION_WITHOUT_PREFIX: ${{ steps.get-version.outputs.version_without_prefix }}
8585
HIGHEST_SEMVER_TAG: ${{ steps.get-version.outputs.highest_semver_tag }}
8686
run: |
87-
if [ "${{ matrix.component }}" = "feature-server" ]; then
88-
echo "feature-server image pushed via buildx during build step"
87+
if [[ "${{ matrix.component }}" == "feature-server" || "${{ matrix.component }}" == "feast-operator" ]]; then
88+
echo "${{ matrix.component }} image pushed via buildx during build step"
8989
else
9090
make push-${{ matrix.component }}-docker REGISTRY=${REGISTRY} VERSION=${VERSION_WITHOUT_PREFIX}
9191
fi
9292
9393
echo "Only push to latest tag if tag is the highest semver version $HIGHEST_SEMVER_TAG"
9494
if [ "${VERSION_WITHOUT_PREFIX}" = "${HIGHEST_SEMVER_TAG:1}" ]
9595
then
96-
if [ "${{ matrix.component }}" = "feature-server" ]; then
97-
docker buildx imagetools create -t ${REGISTRY}/feature-server:latest ${REGISTRY}/feature-server:${VERSION_WITHOUT_PREFIX}
96+
if [[ "${{ matrix.component }}" == "feature-server" || "${{ matrix.component }}" == "feast-operator" ]]; then
97+
docker buildx imagetools create -t ${REGISTRY}/${{ matrix.component }}:latest ${REGISTRY}/${{ matrix.component }}:${VERSION_WITHOUT_PREFIX}
9898
else
9999
docker tag ${REGISTRY}/${{ matrix.component }}:${VERSION_WITHOUT_PREFIX} ${REGISTRY}/${{ matrix.component }}:latest
100100
docker push ${REGISTRY}/${{ matrix.component }}:latest

‎.secrets.baseline‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Makefile‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -730,10 +730,16 @@ push-feast-operator-docker: ## Push Feast Operator Docker image
730730
$(MAKE) docker-push
731731

732732
build-feast-operator-docker: ## Build Feast Operator Docker image
733-
cd infra/feast-operator && \
734-
IMAGE_TAG_BASE=$(REGISTRY)/feast-operator \
735-
VERSION=$(VERSION) \
736-
$(MAKE) docker-build
733+
@if [ -n "$(DOCKER_PLATFORMS)" ]; then \
734+
cd infra/feast-operator && \
735+
docker buildx build --push --platform=$(DOCKER_PLATFORMS) \
736+
--tag $(REGISTRY)/feast-operator:$(VERSION) -f Dockerfile .; \
737+
else \
738+
cd infra/feast-operator && \
739+
IMAGE_TAG_BASE=$(REGISTRY)/feast-operator \
740+
VERSION=$(VERSION) \
741+
$(MAKE) docker-build; \
742+
fi
737743

738744
build-feast-operator-docker-on-mac: ## Build Feast Operator Docker image on Mac
739745
cd infra/feast-operator && \

‎infra/feast-operator/internal/controller/services/namespace_registry.go‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,22 @@ type NamespaceRegistryData struct {
3636
Namespaces map[string][]string `json:"namespaces"`
3737
}
3838

39+
// isProtectedProject checks if this CR is annotated as a protected project
40+
func (feast *FeastServices) isProtectedProject() bool {
41+
annotations := feast.Handler.FeatureStore.GetAnnotations()
42+
return annotations[ProtectedProjectAnnotation] == "true"
43+
}
44+
3945
// deployNamespaceRegistry creates and manages the namespace registry ConfigMap
4046
func (feast *FeastServices) deployNamespaceRegistry() error {
47+
// Skip namespace registry for protected projects.
48+
// Protected projects are managed externally and should not be visible to other instances.
49+
if feast.isProtectedProject() {
50+
logger := log.FromContext(feast.Handler.Context)
51+
logger.V(1).Info("Skipping namespace registry for protected project", "project", feast.Handler.FeatureStore.Spec.FeastProject)
52+
return nil
53+
}
54+
4155
// Check if we can determine the target namespace before creating any resources
4256
targetNamespace, err := feast.getNamespaceRegistryNamespace()
4357
if err != nil {
@@ -230,6 +244,11 @@ func (feast *FeastServices) getNamespaceRegistryNamespace() (string, error) {
230244

231245
// AddToNamespaceRegistry adds a feature store instance to the namespace registry
232246
func (feast *FeastServices) AddToNamespaceRegistry() error {
247+
// Skip for protected projects — they should not appear in the namespace registry.
248+
if feast.isProtectedProject() {
249+
return nil
250+
}
251+
233252
logger := log.FromContext(feast.Handler.Context)
234253
targetNamespace, err := feast.getNamespaceRegistryNamespace()
235254
if err != nil {

‎infra/feast-operator/internal/controller/services/services.go‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -509,6 +509,20 @@ func (feast *FeastServices) setContainers(podSpec *corev1.PodSpec) error {
509509
if feast.isUiServer() {
510510
feast.setContainer(&podSpec.Containers, UIFeastType, fsYamlB64)
511511
}
512+
513+
// When the CR is annotated as a protected project, set FEAST_PROTECTED_PROJECT=true
514+
// so the registry server tags its own project in the shared registry.
515+
// Other FeatureStore instances then exclude this project automatically.
516+
if feast.isProtectedProject() {
517+
protectedEnv := corev1.EnvVar{
518+
Name: "FEAST_PROTECTED_PROJECT",
519+
Value: "true",
520+
}
521+
for i := range podSpec.Containers {
522+
podSpec.Containers[i].Env = append(podSpec.Containers[i].Env, protectedEnv)
523+
}
524+
}
525+
512526
return nil
513527
}
514528

‎infra/feast-operator/internal/controller/services/services_types.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,14 @@ const (
4242
NamespaceRegistryDataKey = "namespaces"
4343
DefaultKubernetesNamespace = "feast-operator-system"
4444

45+
// ProtectedProjectAnnotation is the annotation key on a FeatureStore CR
46+
// that marks its project as protected. Protected projects are excluded
47+
// from project listings and shielded from teardown by other instances.
48+
// When this annotation is "true", the operator sets FEAST_PROTECTED_PROJECT=true
49+
// on the server pods, which causes the server to tag the project in the
50+
// shared registry on startup.
51+
ProtectedProjectAnnotation = "feast.dev/protected-project"
52+
4553
HttpPort = 80
4654
HttpsPort = 443
4755
HttpScheme = "http"

‎sdk/python/feast/api/registry/rest/rest_registry_server.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,10 @@ def start_server(
327327
):
328328
import uvicorn
329329

330+
from feast.registry_server import _sync_protected_project_tag
331+
332+
_sync_protected_project_tag(self.store)
333+
330334
if tls_key_path and tls_cert_path:
331335
logger.info("Starting REST registry server in TLS(SSL) mode")
332336
logger.info(f"REST registry server listening on https://localhost:{port}")

‎sdk/python/feast/constants.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,12 @@
4646

4747
# Default feature server registry ttl (seconds)
4848
DEFAULT_FEATURE_SERVER_REGISTRY_TTL = 5
49+
50+
# Tag key set on Feast projects that are protected.
51+
# Protected projects are excluded from project listings,
52+
# shielded from teardown, and hidden from delete operations.
53+
PROTECTED_PROJECT_TAG = "feast.dev/protected-project"
54+
55+
# Environment variable set by the operator on protected project pods.
56+
# When "true", the Feast server tags its own project as protected.
57+
FEAST_PROTECTED_PROJECT_ENV = "FEAST_PROTECTED_PROJECT"

‎sdk/python/feast/feature_server.py‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
# limitations under the License.
1414

1515
import asyncio
16+
import functools
1617
import os
1718
import sys
1819
import threading
@@ -1004,6 +1005,7 @@ def __init__(
10041005
store=store,
10051006
registry_ttl_sec=options["registry_ttl_sec"],
10061007
)
1008+
self._store = store
10071009
self._options = options
10081010
self._metrics_enabled = metrics_enabled
10091011
super().__init__()
@@ -1015,15 +1017,19 @@ def load_config(self):
10151017

10161018
self.cfg.set("worker_class", "uvicorn_worker.UvicornWorker")
10171019
if self._metrics_enabled:
1018-
self.cfg.set("post_worker_init", _gunicorn_post_worker_init)
1020+
self.cfg.set(
1021+
"post_worker_init",
1022+
functools.partial(_gunicorn_post_worker_init, self._store),
1023+
)
10191024
self.cfg.set("child_exit", _gunicorn_child_exit)
10201025

10211026
def load(self):
10221027
return self._app
10231028

1024-
def _gunicorn_post_worker_init(worker):
1025-
"""Start per-worker resource monitoring after Gunicorn forks."""
1029+
def _gunicorn_post_worker_init(store: "feast.FeatureStore", worker):
1030+
"""Start per-worker resource and freshness monitoring after Gunicorn forks."""
10261031
feast_metrics.init_worker_monitoring()
1032+
feast_metrics.init_worker_freshness_monitoring(store)
10271033

10281034
def _gunicorn_child_exit(server, worker):
10291035
"""Clean up Prometheus metric files for a dead worker."""
@@ -1061,6 +1067,7 @@ def start_server(
10611067
store,
10621068
metrics_config=flags,
10631069
start_resource_monitoring=not uses_gunicorn,
1070+
start_freshness_monitoring=not uses_gunicorn,
10641071
)
10651072

10661073
logger.debug("start_server called")

0 commit comments

Comments
 (0)