Skip to content

Add local k8s cluster setup#309

Merged
peterj merged 7 commits intoagentregistry-dev:mainfrom
nikolasmatt:local-k8s-env
Mar 11, 2026
Merged

Add local k8s cluster setup#309
peterj merged 7 commits intoagentregistry-dev:mainfrom
nikolasmatt:local-k8s-env

Conversation

@nikolasmatt
Copy link
Copy Markdown
Collaborator

Description

Adds a local Kubernetes development environment using Kind. A single make setup-kind-cluster target creates a Kind cluster, deploys
PostgreSQL/pgvector, builds the server image, and installs AgentRegistry via Helm.

  • New Makefile targets: setup-kind-cluster, create-kind-cluster, install-postgresql, install-agentregistry, delete-kind-cluster, use-kind-cluster, prune-kind-cluster, kind-debug, install-tools
  • New examples/postgres-pgvector.yaml — standalone PG/pgvector manifest for dev/test clusters
  • New scripts/kind/README.md — setup guide, DB details, troubleshooting
  • Updated DEVELOPMENT.md and README.md with local K8s and Helm install sections

Change Type

/kind cleanup
/kind documentation

Changelog

NONE

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a local Kubernetes development environment based on Kind to run AgentRegistry with a bundled PostgreSQL/pgvector setup, and documents a Helm-based install path for Kubernetes users.

Changes:

  • Introduces new Makefile targets to create/use/teardown a Kind cluster, install PostgreSQL/pgvector, and deploy AgentRegistry via Helm.
  • Adds a standalone examples/postgres-pgvector.yaml manifest for dev/test clusters and new Kind setup documentation under scripts/kind/.
  • Updates top-level docs (README.md, DEVELOPMENT.md) with Kubernetes/Helm and local Kind workflow guidance.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
scripts/kind/setup-kind.sh Makes Kind binary resolution more flexible and uses it for cluster/node operations.
scripts/kind/README.md New guide for local Kind workflow, DB details, and troubleshooting.
examples/postgres-pgvector.yaml New dev/test PostgreSQL+pgvector Kubernetes manifest.
README.md Adds Kubernetes deployment section and references the example PG manifest and Kind docs.
Makefile Adds Kind lifecycle + install targets and tool install helper.
DEVELOPMENT.md Adds a Kind-based local Kubernetes workflow section ahead of architecture notes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +100 to +107
| Variable | Default | Description |
|-------------------|---------------------------------|------------------------------------|
| `KIND_CLUSTER_NAME` | `agentregistry` | Kind cluster name |
| `NAMESPACE` | `agentregistry` | Kubernetes namespace |
| `DOCKER_REGISTRY` | `localhost:5001` | Local registry address |
| `DOCKER_REPO` | `agentregistry-dev/agentregistry` | Image repository prefix |
| `VERSION` | `git describe --tags --always` | Image tag to deploy |
| `JWT_KEY` | Random 32-byte hex | JWT private key for AgentRegistry |
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The configuration table lists DOCKER_REPO as an override that affects the images deployed to Kind, but the current make install-agentregistry only sets image.registry and image.tag (not image.repository). Either update the Makefile to set image.repository from DOCKER_REPO, or adjust this documentation so it matches the actual behavior.

Suggested change
| Variable | Default | Description |
|-------------------|---------------------------------|------------------------------------|
| `KIND_CLUSTER_NAME` | `agentregistry` | Kind cluster name |
| `NAMESPACE` | `agentregistry` | Kubernetes namespace |
| `DOCKER_REGISTRY` | `localhost:5001` | Local registry address |
| `DOCKER_REPO` | `agentregistry-dev/agentregistry` | Image repository prefix |
| `VERSION` | `git describe --tags --always` | Image tag to deploy |
| `JWT_KEY` | Random 32-byte hex | JWT private key for AgentRegistry |
| Variable | Default | Description |
|-------------------|---------------------------------|----------------------------------------------------|
| `KIND_CLUSTER_NAME` | `agentregistry` | Kind cluster name |
| `NAMESPACE` | `agentregistry` | Kubernetes namespace |
| `DOCKER_REGISTRY` | `localhost:5001` | Local registry address |
| `DOCKER_REPO` | `agentregistry-dev/agentregistry` | Image repository prefix for local image builds |
| `VERSION` | `git describe --tags --always` | Image tag to deploy |
| `JWT_KEY` | Random 32-byte hex | JWT private key for AgentRegistry |

Copilot uses AI. Check for mistakes.
Makefile Outdated
Comment on lines +309 to +310
$(KIND_BIN) get kubeconfig --name $(KIND_CLUSTER_NAME) > /tmp/kind-config
KUBECONFIG=~/.kube/config:/tmp/kind-config kubectl config view --merge --flatten > ~/.kube/config.tmp && mv ~/.kube/config.tmp ~/.kube/config && chmod $(KUBECONFIG_PERM) ~/.kube/config
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use-kind-cluster writes a kubeconfig containing client certs/keys to a fixed path in /tmp without setting restrictive permissions or cleaning it up. Use a mktemp file (chmod 600) and remove it after merging to avoid leaking kubeconfig material on multi-user systems.

Suggested change
$(KIND_BIN) get kubeconfig --name $(KIND_CLUSTER_NAME) > /tmp/kind-config
KUBECONFIG=~/.kube/config:/tmp/kind-config kubectl config view --merge --flatten > ~/.kube/config.tmp && mv ~/.kube/config.tmp ~/.kube/config && chmod $(KUBECONFIG_PERM) ~/.kube/config
@TMP_KUBECONFIG=$$(mktemp) && chmod 600 $$TMP_KUBECONFIG && \
$(KIND_BIN) get kubeconfig --name $(KIND_CLUSTER_NAME) > $$TMP_KUBECONFIG && \
KUBECONFIG=~/.kube/config:$$TMP_KUBECONFIG kubectl config view --merge --flatten > ~/.kube/config.tmp && \
mv ~/.kube/config.tmp ~/.kube/config && chmod $(KUBECONFIG_PERM) ~/.kube/config && \
rm -f $$TMP_KUBECONFIG

Copilot uses AI. Check for mistakes.
Comment on lines +346 to +357
helm upgrade --install agentregistry charts/agentregistry \
--kube-context $(KIND_CLUSTER_CONTEXT) \
--namespace $(KIND_NAMESPACE) \
--create-namespace \
--set image.pullPolicy=Always \
--set image.registry=$(DOCKER_REGISTRY) \
--set image.tag=$(VERSION) \
--set database.host=postgres-pgvector.$(KIND_NAMESPACE).svc.cluster.local \
--set database.password=agentregistry \
--set database.sslMode=disable \
--set config.jwtPrivateKey="$$JWT_KEY" \
--set config.enableAnonymousAuth="true" \
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

install-agentregistry hard-codes helm instead of using the Makefile’s HELM variable, and it never sets image.repository. If a developer overrides DOCKER_REPO (as documented in scripts/kind/README.md), the images built/pushed by docker-server won’t match what Helm deploys. Use $(HELM) and pass --set image.repository=$(DOCKER_REPO)/server (or otherwise derive the repository from the same variables used for building).

Copilot uses AI. Check for mistakes.

- Docker Desktop with Docker Compose v2+
- Go 1.25+ (for building from source)
- PostgreSQL with the [pgvector](https://github.com/pgvector/pgvector) extension
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Quick Start prerequisites now require a local PostgreSQL+pgvector install, but the default local dev flow (arctl starting the daemon via docker-compose) already runs PostgreSQL in a container (internal/daemon/docker-compose.yml). Consider rewording this prerequisite to clarify that PostgreSQL+pgvector is required for Kubernetes/Helm deployments, not for the default Docker Compose quick start.

Suggested change
- PostgreSQL with the [pgvector](https://github.com/pgvector/pgvector) extension
- PostgreSQL with the [pgvector](https://github.com/pgvector/pgvector) extension **for Kubernetes/Helm or external database deployments** (not required for the default Docker Compose quick start, which runs PostgreSQL in a container)

Copilot uses AI. Check for mistakes.
Makefile Outdated
.PHONY: install-postgresql
install-postgresql: ## Deploy standalone PostgreSQL/pgvector into the Kind cluster
kubectl --context $(KIND_CLUSTER_CONTEXT) apply -f examples/postgres-pgvector.yaml
kubectl --context $(KIND_CLUSTER_CONTEXT) -n $(KIND_NAMESPACE) wait --for=condition=ready pod -l app=postgres-pgvector --timeout=120s
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KIND_NAMESPACE is presented as configurable, but examples/postgres-pgvector.yaml hard-codes namespace: agentregistry and install-postgresql applies that manifest then waits in -n $(KIND_NAMESPACE). If KIND_NAMESPACE is overridden, the wait will target the wrong namespace and the install will appear to hang/fail. Either make the namespace fixed (remove the override) or template/patch the manifest so it deploys into $(KIND_NAMESPACE) consistently.

Suggested change
kubectl --context $(KIND_CLUSTER_CONTEXT) -n $(KIND_NAMESPACE) wait --for=condition=ready pod -l app=postgres-pgvector --timeout=120s
kubectl --context $(KIND_CLUSTER_CONTEXT) -n agentregistry wait --for=condition=ready pod -l app=postgres-pgvector --timeout=120s

Copilot uses AI. Check for mistakes.
Comment on lines +9 to +11
KIND="$(command -v kind)"
fi

Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If bin/kind is missing and kind is not on PATH, KIND="$(command -v kind)" can resolve to an empty string and the script will later fail with a confusing error. Consider explicitly validating that KIND resolves to an executable and printing a clear install hint before continuing.

Suggested change
KIND="$(command -v kind)"
fi
KIND="$(command -v kind || true)"
fi
if [ -z "${KIND}" ] || [ ! -x "${KIND}" ]; then
echo "Error: 'kind' binary not found." >&2
echo "Please install kind (https://kind.sigs.k8s.io/) and ensure it is either available at '${SCRIPT_DIR}/../../bin/kind' or on your PATH." >&2
exit 1
fi

Copilot uses AI. Check for mistakes.
Comment on lines 35 to 40
if ${KIND} get clusters | grep -qx "${KIND_CLUSTER_NAME}"; then
echo "Kind cluster '${KIND_CLUSTER_NAME}' already exists; skipping create."
else
kind create cluster --name "${KIND_CLUSTER_NAME}" \
${KIND} create cluster --name "${KIND_CLUSTER_NAME}" \
--config scripts/kind/kind-config.yaml \
--image="kindest/node:v${KIND_IMAGE_VERSION}"
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

${KIND} is executed unquoted here and later in the script. Quoting the command path (and other interpolated variables) avoids word-splitting surprises and makes the script safer if KIND ever contains spaces or additional args.

Copilot uses AI. Check for mistakes.
Makefile Outdated
@echo " prune-kind-cluster - Prune dangling images from Kind control-plane"
@echo " kind-debug - Shell into Kind control-plane with btop"
@echo " install-postgresql - Deploy PostgreSQL/pgvector into Kind"
@echo " install-agentregistry- Build images and Helm-install AgentRegistry into Kind"
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The help text has a missing space: install-agentregistry- Build... will render as a single word in make help. Update the string to include a space after the target name for readability.

Suggested change
@echo " install-agentregistry- Build images and Helm-install AgentRegistry into Kind"
@echo " install-agentregistry - Build images and Helm-install AgentRegistry into Kind"

Copilot uses AI. Check for mistakes.
Makefile Outdated
Comment on lines +321 to +322
docker exec $(KIND_CLUSTER_NAME)-control-plane crictl images --no-trunc --quiet | \
grep '<none>' | awk '{print $$3}' | xargs -r -n1 docker exec $(KIND_CLUSTER_NAME)-control-plane crictl rmi || :
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prune-kind-cluster is unlikely to work as written: crictl images --quiet outputs only image IDs (so grep '<none>' | awk '{print $3}' will never match), and xargs -r is not portable (fails on macOS/BSD xargs). Consider using the non-quiet crictl images output to filter dangling images, and avoid -r by guarding the xargs call when the input is empty.

Suggested change
docker exec $(KIND_CLUSTER_NAME)-control-plane crictl images --no-trunc --quiet | \
grep '<none>' | awk '{print $$3}' | xargs -r -n1 docker exec $(KIND_CLUSTER_NAME)-control-plane crictl rmi || :
docker exec $(KIND_CLUSTER_NAME)-control-plane crictl images --no-trunc | \
awk '$$1=="<none>" && $$2=="<none>" {print $$3}' | \
while read -r img; do \
if [ -n "$$img" ]; then \
docker exec $(KIND_CLUSTER_NAME)-control-plane crictl rmi "$$img"; \
fi; \
done || :

Copilot uses AI. Check for mistakes.
Comment on lines +342 to +357
@JWT_KEY=$$(kubectl --context $(KIND_CLUSTER_CONTEXT) -n $(KIND_NAMESPACE) \
get secret agentregistry \
-o jsonpath='{.data.AGENT_REGISTRY_JWT_PRIVATE_KEY}' 2>/dev/null | base64 -d); \
if [ -z "$$JWT_KEY" ]; then JWT_KEY=$$(openssl rand -hex 32); fi; \
helm upgrade --install agentregistry charts/agentregistry \
--kube-context $(KIND_CLUSTER_CONTEXT) \
--namespace $(KIND_NAMESPACE) \
--create-namespace \
--set image.pullPolicy=Always \
--set image.registry=$(DOCKER_REGISTRY) \
--set image.tag=$(VERSION) \
--set database.host=postgres-pgvector.$(KIND_NAMESPACE).svc.cluster.local \
--set database.password=agentregistry \
--set database.sslMode=disable \
--set config.jwtPrivateKey="$$JWT_KEY" \
--set config.enableAnonymousAuth="true" \
Copy link

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The install-agentregistry target passes the JWT signing key in the helm upgrade command via --set config.jwtPrivateKey="$$JWT_KEY", which exposes this secret in process arguments (e.g., ps output) to other local users while the command runs. A local attacker on the same host could capture this key and forge valid JWTs against the AgentRegistry instance, undermining authentication even after redeploys that reuse the same key. To mitigate this, avoid passing the key on the command line and instead load it from a Kubernetes secret or values file that is not exposed via process arguments.

Copilot uses AI. Check for mistakes.
@peterj
Copy link
Copy Markdown
Contributor

peterj commented Mar 11, 2026

some copilot comments to fix; but otherwise looks good to me!

@peterj peterj added this pull request to the merge queue Mar 11, 2026
Merged via the queue into agentregistry-dev:main with commit 68d7d16 Mar 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants