Add local k8s cluster setup#309
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a local Kubernetes development environment based on Kind to run AgentRegistry with a bundled PostgreSQL/pgvector setup, and documents a Helm-based install path for Kubernetes users.
Changes:
- Introduces new Makefile targets to create/use/teardown a Kind cluster, install PostgreSQL/pgvector, and deploy AgentRegistry via Helm.
- Adds a standalone
examples/postgres-pgvector.yamlmanifest for dev/test clusters and new Kind setup documentation underscripts/kind/. - Updates top-level docs (
README.md,DEVELOPMENT.md) with Kubernetes/Helm and local Kind workflow guidance.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/kind/setup-kind.sh | Makes Kind binary resolution more flexible and uses it for cluster/node operations. |
| scripts/kind/README.md | New guide for local Kind workflow, DB details, and troubleshooting. |
| examples/postgres-pgvector.yaml | New dev/test PostgreSQL+pgvector Kubernetes manifest. |
| README.md | Adds Kubernetes deployment section and references the example PG manifest and Kind docs. |
| Makefile | Adds Kind lifecycle + install targets and tool install helper. |
| DEVELOPMENT.md | Adds a Kind-based local Kubernetes workflow section ahead of architecture notes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
scripts/kind/README.md
Outdated
| | Variable | Default | Description | | ||
| |-------------------|---------------------------------|------------------------------------| | ||
| | `KIND_CLUSTER_NAME` | `agentregistry` | Kind cluster name | | ||
| | `NAMESPACE` | `agentregistry` | Kubernetes namespace | | ||
| | `DOCKER_REGISTRY` | `localhost:5001` | Local registry address | | ||
| | `DOCKER_REPO` | `agentregistry-dev/agentregistry` | Image repository prefix | | ||
| | `VERSION` | `git describe --tags --always` | Image tag to deploy | | ||
| | `JWT_KEY` | Random 32-byte hex | JWT private key for AgentRegistry | |
There was a problem hiding this comment.
The configuration table lists DOCKER_REPO as an override that affects the images deployed to Kind, but the current make install-agentregistry only sets image.registry and image.tag (not image.repository). Either update the Makefile to set image.repository from DOCKER_REPO, or adjust this documentation so it matches the actual behavior.
| | Variable | Default | Description | | |
| |-------------------|---------------------------------|------------------------------------| | |
| | `KIND_CLUSTER_NAME` | `agentregistry` | Kind cluster name | | |
| | `NAMESPACE` | `agentregistry` | Kubernetes namespace | | |
| | `DOCKER_REGISTRY` | `localhost:5001` | Local registry address | | |
| | `DOCKER_REPO` | `agentregistry-dev/agentregistry` | Image repository prefix | | |
| | `VERSION` | `git describe --tags --always` | Image tag to deploy | | |
| | `JWT_KEY` | Random 32-byte hex | JWT private key for AgentRegistry | | |
| | Variable | Default | Description | | |
| |-------------------|---------------------------------|----------------------------------------------------| | |
| | `KIND_CLUSTER_NAME` | `agentregistry` | Kind cluster name | | |
| | `NAMESPACE` | `agentregistry` | Kubernetes namespace | | |
| | `DOCKER_REGISTRY` | `localhost:5001` | Local registry address | | |
| | `DOCKER_REPO` | `agentregistry-dev/agentregistry` | Image repository prefix for local image builds | | |
| | `VERSION` | `git describe --tags --always` | Image tag to deploy | | |
| | `JWT_KEY` | Random 32-byte hex | JWT private key for AgentRegistry | |
Makefile
Outdated
| $(KIND_BIN) get kubeconfig --name $(KIND_CLUSTER_NAME) > /tmp/kind-config | ||
| KUBECONFIG=~/.kube/config:/tmp/kind-config kubectl config view --merge --flatten > ~/.kube/config.tmp && mv ~/.kube/config.tmp ~/.kube/config && chmod $(KUBECONFIG_PERM) ~/.kube/config |
There was a problem hiding this comment.
use-kind-cluster writes a kubeconfig containing client certs/keys to a fixed path in /tmp without setting restrictive permissions or cleaning it up. Use a mktemp file (chmod 600) and remove it after merging to avoid leaking kubeconfig material on multi-user systems.
| $(KIND_BIN) get kubeconfig --name $(KIND_CLUSTER_NAME) > /tmp/kind-config | |
| KUBECONFIG=~/.kube/config:/tmp/kind-config kubectl config view --merge --flatten > ~/.kube/config.tmp && mv ~/.kube/config.tmp ~/.kube/config && chmod $(KUBECONFIG_PERM) ~/.kube/config | |
| @TMP_KUBECONFIG=$$(mktemp) && chmod 600 $$TMP_KUBECONFIG && \ | |
| $(KIND_BIN) get kubeconfig --name $(KIND_CLUSTER_NAME) > $$TMP_KUBECONFIG && \ | |
| KUBECONFIG=~/.kube/config:$$TMP_KUBECONFIG kubectl config view --merge --flatten > ~/.kube/config.tmp && \ | |
| mv ~/.kube/config.tmp ~/.kube/config && chmod $(KUBECONFIG_PERM) ~/.kube/config && \ | |
| rm -f $$TMP_KUBECONFIG |
| helm upgrade --install agentregistry charts/agentregistry \ | ||
| --kube-context $(KIND_CLUSTER_CONTEXT) \ | ||
| --namespace $(KIND_NAMESPACE) \ | ||
| --create-namespace \ | ||
| --set image.pullPolicy=Always \ | ||
| --set image.registry=$(DOCKER_REGISTRY) \ | ||
| --set image.tag=$(VERSION) \ | ||
| --set database.host=postgres-pgvector.$(KIND_NAMESPACE).svc.cluster.local \ | ||
| --set database.password=agentregistry \ | ||
| --set database.sslMode=disable \ | ||
| --set config.jwtPrivateKey="$$JWT_KEY" \ | ||
| --set config.enableAnonymousAuth="true" \ |
There was a problem hiding this comment.
install-agentregistry hard-codes helm instead of using the Makefile’s HELM variable, and it never sets image.repository. If a developer overrides DOCKER_REPO (as documented in scripts/kind/README.md), the images built/pushed by docker-server won’t match what Helm deploys. Use $(HELM) and pass --set image.repository=$(DOCKER_REPO)/server (or otherwise derive the repository from the same variables used for building).
|
|
||
| - Docker Desktop with Docker Compose v2+ | ||
| - Go 1.25+ (for building from source) | ||
| - PostgreSQL with the [pgvector](https://github.com/pgvector/pgvector) extension |
There was a problem hiding this comment.
The Quick Start prerequisites now require a local PostgreSQL+pgvector install, but the default local dev flow (arctl starting the daemon via docker-compose) already runs PostgreSQL in a container (internal/daemon/docker-compose.yml). Consider rewording this prerequisite to clarify that PostgreSQL+pgvector is required for Kubernetes/Helm deployments, not for the default Docker Compose quick start.
| - PostgreSQL with the [pgvector](https://github.com/pgvector/pgvector) extension | |
| - PostgreSQL with the [pgvector](https://github.com/pgvector/pgvector) extension **for Kubernetes/Helm or external database deployments** (not required for the default Docker Compose quick start, which runs PostgreSQL in a container) |
Makefile
Outdated
| .PHONY: install-postgresql | ||
| install-postgresql: ## Deploy standalone PostgreSQL/pgvector into the Kind cluster | ||
| kubectl --context $(KIND_CLUSTER_CONTEXT) apply -f examples/postgres-pgvector.yaml | ||
| kubectl --context $(KIND_CLUSTER_CONTEXT) -n $(KIND_NAMESPACE) wait --for=condition=ready pod -l app=postgres-pgvector --timeout=120s |
There was a problem hiding this comment.
KIND_NAMESPACE is presented as configurable, but examples/postgres-pgvector.yaml hard-codes namespace: agentregistry and install-postgresql applies that manifest then waits in -n $(KIND_NAMESPACE). If KIND_NAMESPACE is overridden, the wait will target the wrong namespace and the install will appear to hang/fail. Either make the namespace fixed (remove the override) or template/patch the manifest so it deploys into $(KIND_NAMESPACE) consistently.
| kubectl --context $(KIND_CLUSTER_CONTEXT) -n $(KIND_NAMESPACE) wait --for=condition=ready pod -l app=postgres-pgvector --timeout=120s | |
| kubectl --context $(KIND_CLUSTER_CONTEXT) -n agentregistry wait --for=condition=ready pod -l app=postgres-pgvector --timeout=120s |
scripts/kind/setup-kind.sh
Outdated
| KIND="$(command -v kind)" | ||
| fi | ||
|
|
There was a problem hiding this comment.
If bin/kind is missing and kind is not on PATH, KIND="$(command -v kind)" can resolve to an empty string and the script will later fail with a confusing error. Consider explicitly validating that KIND resolves to an executable and printing a clear install hint before continuing.
| KIND="$(command -v kind)" | |
| fi | |
| KIND="$(command -v kind || true)" | |
| fi | |
| if [ -z "${KIND}" ] || [ ! -x "${KIND}" ]; then | |
| echo "Error: 'kind' binary not found." >&2 | |
| echo "Please install kind (https://kind.sigs.k8s.io/) and ensure it is either available at '${SCRIPT_DIR}/../../bin/kind' or on your PATH." >&2 | |
| exit 1 | |
| fi |
scripts/kind/setup-kind.sh
Outdated
| if ${KIND} get clusters | grep -qx "${KIND_CLUSTER_NAME}"; then | ||
| echo "Kind cluster '${KIND_CLUSTER_NAME}' already exists; skipping create." | ||
| else | ||
| kind create cluster --name "${KIND_CLUSTER_NAME}" \ | ||
| ${KIND} create cluster --name "${KIND_CLUSTER_NAME}" \ | ||
| --config scripts/kind/kind-config.yaml \ | ||
| --image="kindest/node:v${KIND_IMAGE_VERSION}" |
There was a problem hiding this comment.
${KIND} is executed unquoted here and later in the script. Quoting the command path (and other interpolated variables) avoids word-splitting surprises and makes the script safer if KIND ever contains spaces or additional args.
Makefile
Outdated
| @echo " prune-kind-cluster - Prune dangling images from Kind control-plane" | ||
| @echo " kind-debug - Shell into Kind control-plane with btop" | ||
| @echo " install-postgresql - Deploy PostgreSQL/pgvector into Kind" | ||
| @echo " install-agentregistry- Build images and Helm-install AgentRegistry into Kind" |
There was a problem hiding this comment.
The help text has a missing space: install-agentregistry- Build... will render as a single word in make help. Update the string to include a space after the target name for readability.
| @echo " install-agentregistry- Build images and Helm-install AgentRegistry into Kind" | |
| @echo " install-agentregistry - Build images and Helm-install AgentRegistry into Kind" |
Makefile
Outdated
| docker exec $(KIND_CLUSTER_NAME)-control-plane crictl images --no-trunc --quiet | \ | ||
| grep '<none>' | awk '{print $$3}' | xargs -r -n1 docker exec $(KIND_CLUSTER_NAME)-control-plane crictl rmi || : |
There was a problem hiding this comment.
prune-kind-cluster is unlikely to work as written: crictl images --quiet outputs only image IDs (so grep '<none>' | awk '{print $3}' will never match), and xargs -r is not portable (fails on macOS/BSD xargs). Consider using the non-quiet crictl images output to filter dangling images, and avoid -r by guarding the xargs call when the input is empty.
| docker exec $(KIND_CLUSTER_NAME)-control-plane crictl images --no-trunc --quiet | \ | |
| grep '<none>' | awk '{print $$3}' | xargs -r -n1 docker exec $(KIND_CLUSTER_NAME)-control-plane crictl rmi || : | |
| docker exec $(KIND_CLUSTER_NAME)-control-plane crictl images --no-trunc | \ | |
| awk '$$1=="<none>" && $$2=="<none>" {print $$3}' | \ | |
| while read -r img; do \ | |
| if [ -n "$$img" ]; then \ | |
| docker exec $(KIND_CLUSTER_NAME)-control-plane crictl rmi "$$img"; \ | |
| fi; \ | |
| done || : |
| @JWT_KEY=$$(kubectl --context $(KIND_CLUSTER_CONTEXT) -n $(KIND_NAMESPACE) \ | ||
| get secret agentregistry \ | ||
| -o jsonpath='{.data.AGENT_REGISTRY_JWT_PRIVATE_KEY}' 2>/dev/null | base64 -d); \ | ||
| if [ -z "$$JWT_KEY" ]; then JWT_KEY=$$(openssl rand -hex 32); fi; \ | ||
| helm upgrade --install agentregistry charts/agentregistry \ | ||
| --kube-context $(KIND_CLUSTER_CONTEXT) \ | ||
| --namespace $(KIND_NAMESPACE) \ | ||
| --create-namespace \ | ||
| --set image.pullPolicy=Always \ | ||
| --set image.registry=$(DOCKER_REGISTRY) \ | ||
| --set image.tag=$(VERSION) \ | ||
| --set database.host=postgres-pgvector.$(KIND_NAMESPACE).svc.cluster.local \ | ||
| --set database.password=agentregistry \ | ||
| --set database.sslMode=disable \ | ||
| --set config.jwtPrivateKey="$$JWT_KEY" \ | ||
| --set config.enableAnonymousAuth="true" \ |
There was a problem hiding this comment.
The install-agentregistry target passes the JWT signing key in the helm upgrade command via --set config.jwtPrivateKey="$$JWT_KEY", which exposes this secret in process arguments (e.g., ps output) to other local users while the command runs. A local attacker on the same host could capture this key and forge valid JWTs against the AgentRegistry instance, undermining authentication even after redeploys that reuse the same key. To mitigate this, avoid passing the key on the command line and instead load it from a Kubernetes secret or values file that is not exposed via process arguments.
|
some copilot comments to fix; but otherwise looks good to me! |
Description
Adds a local Kubernetes development environment using Kind. A single make setup-kind-cluster target creates a Kind cluster, deploys
PostgreSQL/pgvector, builds the server image, and installs AgentRegistry via Helm.
Change Type
/kind cleanup
/kind documentation
Changelog