Skip to content

Latest commit

 

History

History
140 lines (115 loc) · 8.56 KB

README.md

File metadata and controls

140 lines (115 loc) · 8.56 KB

substrate

Helm chart for installing Agent Substrate.

The chart uses mTLS and requires a prepared PostgreSQL database. It requires the ClusterTrustBundle, ClusterTrustBundleProjection, and PodCertificateRequest feature gates plus the certificates.k8s.io/v1beta1 API.

Install the certificate controller as a separate release before preparing PostgreSQL or installing this chart. One controller serves the cluster; the application chart does not deploy it.

# CRDs
helm upgrade --install substrate-crds ./charts/substrate-crds

# Create CA pools, then install their certificate controller
go run ./cmd/ate-setup create podcertificate-controller-cas
helm upgrade --install substrate-podcert ./charts/substrate-podcert \
  --namespace podcertificate-controller-system --create-namespace --wait

# Install Substrate after creating the database identities and Secrets
helm upgrade --install substrate ./charts/substrate \
  --namespace ate-system --create-namespace \
  --set postgres.readWriteConnectionStringSecretRef.name=substrate-postgres-readwrite \
  --set postgres.ownerConnectionStringSecretRef.name=substrate-postgres-owner

By default, component images are pulled from ghcr.io/kagent-dev/substrate using the chart appVersion as the tag. Override image.registry and image.tag to install from a different image repository or tag.

The chart installs the Kubernetes credential provider and enables HTTPS egress interception. Create the egress-mitm-ca-pool Secret and configure actor trust as described in the credential provider setup. Namespace grants default to an empty list, denying credential access.

Render manifests without applying

helm template substrate ./charts/substrate

manifests/ate-install/ in the repo is the rendered mTLS output and is regenerated by hack/render-manifests.sh. The separate substrate-crds chart mirrors manifests/ate-install/generated/.

Values

See values.yaml for the full set; the important keys:

Key Default Notes
postgres.readWriteConnectionStringSecretRef substrate-postgres-readwrite Read the read/write connection from a pre-created Secret
postgres.ownerConnectionStringSecretRef substrate-postgres-owner Read the owner connection from a pre-created Secret
postgres.readWriteRole substrate_readwrite Role assumed by read/write connections
postgres.ownerRole substrate_owner Role assumed by owner connections
postgres.schema substrate Store the Substrate tables in this PostgreSQL schema
postgres.clientCertificates.enabled false Project separate owner and runtime login certificates into the API server
<component>.resources see values.yaml Requests and limits for each container, e.g. ateApi.resources and atenetRouter.agentgateway.resources
rustfs.enabled true Deploy an in-cluster S3-compatible RustFS bucket for snapshots
atelet.storageBackend s3 Default snapshot backend, wired to RustFS when rustfs.enabled=true
atelet.imageCredentialProviderConfig "" Host path to the kubelet credential provider config; set together with the bin directory
atelet.imageCredentialProviderConfigYAML "" Credential provider config as text, rendered into a ConfigMap, for nodes where that file cannot be authored or must differ from the kubelet's; mutually exclusive with the host path
atelet.imageCredentialProviderBinDir "" Host path to the kubelet credential provider binaries; both paths are mounted read-only
credentialProvider.namespacePolicies [] Default-deny atespace-to-namespace grants; the chart includes get-only Secret RBAC for the provider
ateApi.actorJWTIssuer https://idp.<namespace>.svc Issuer placed in actor JWTs; set a stable URL reachable by relying parties
snapshotPlugin.extraEnv [] Object storage credentials/settings shared by both snapshot plugins and atelet
snapshotPlugin.control.resources / snapshotPlugin.node.resources see values.yaml Resources for the API and node snapshot sidecars
ateApi.extraArgs [] Additional command-line arguments appended to the ateapi defaults
otel.endpoint "" Set to an OTLP endpoint to export traces, metrics, the actor lifecycle events and the router access log
otel.traces.enabled true Set to false to export no traces from the router; the Go components do not honor this yet
otel.traces.endpoint "" OTLP endpoint for traces, overriding otel.endpoint
otel.traces.samplingRatio 0.01 Fraction of parentless requests that start a trace, applied to the Go components and the router
otel.metrics.enabled true Set to false to disable OTLP metrics push; Prometheus scraping remains available
otel.metrics.endpoint "" OTLP endpoint for metrics, overriding otel.endpoint
otel.logs.enabled true Enable OTLP actor events from ateapi and the ateoms, plus the router access log. Actor events go to stdout when OTLP logs are disabled
otel.logs.endpoint "" OTLP endpoint for logs, overriding otel.endpoint

Upgrading sandbox and snapshot configuration

Upgrade substrate-crds before the application chart. Custom SandboxConfig resources now put assets and pauseImage under spec.versions[], with a named spec.defaultVersion. WorkerPools use spec.sandboxClasses[], and actor templates continue to select the config with sandboxConfig.configName.

WorkerPools now resolve spec.sandboxClasses[].configRef or the class default before creating their worker Deployment. Ensure each pool names an existing config of the same class, or that one is annotated sandboxconfig.ate.dev/is-class-default: "true". The chart supplies that annotation on gvisor-default. Sandbox class names are immutable; create a new pool to change runtime families. Configs referenced by a pool are protected from deletion until the pool switches configs or is deleted.

Update API clients for the simplified snapshot model: Snapshot.uuid, fidelity, locality, and durableSnapshot replace the storage[] entries, and durable snapshot status uses PENDING instead of IN_PROGRESS. The proto field numbers changed, so upgrade clients and components together. Persisted Actor and Tag snapshot metadata uses the same protobuf schema and is not migrated by this update. For development installs, delete affected Actors and Tags with the old version, then recreate them after upgrading.

Snapshot transfers now run in native sidecars in the API server and atelet pods. Move external object storage environment settings from atelet.extraEnv or the API environment ConfigMap to snapshotPlugin.extraEnv, so both plugins and atelet asset downloads receive them. ATE_STORAGE_BACKEND must not be set on the API server itself; select it through atelet.storageBackend. RustFS is wired automatically.

PostgreSQL setup

The chart does not deploy or initialize PostgreSQL. Prepare the database, schema, login users, and the configured readWriteRole and ownerRole before installing the chart. For development Helm installations in ate-system, run hack/install-postgres.sh after installing the separate certificate controller and its CA pools. It deploys certificate-protected PostgreSQL, runs the shared bootstrap SQL, and creates both connection Secrets. Use --kind for the smaller Kind deployment; set KUBECTL_CONTEXT to select a cluster. Standalone manifest installations can continue to use ate-setup.

When using hack/install-postgres.sh, install the application chart with --set postgres.clientCertificates.enabled=true. Its connection Secrets use the projected owner and runtime certificates. For external databases using password or provider-managed authentication, leave this setting disabled. Service-DNS and pod-identity certificates remain enabled for Substrate's internal connections. Standalone manifest generation enables the PostgreSQL certificates for the bundled development database.

For an operator-managed database, the operator must provision the identities, schema, and grants, and create both connection Secrets in the release namespace.

Substrate runs SET ROLE for each new connection. It rejects a login without the required membership.

Create both group roles and all grants before installation, then set postgres.readWriteRole and postgres.ownerRole to those names. Use distinct roles and table schemas for separate installs sharing one database. Give each install separate logins and grant each login membership only in its install's roles.