Synchronizer

The Synchronizer serves as a data synchronization engine between in-cluster and backend resources, designed to facilitate bi-directional data synchronization between the two. It operates as an efficient event-driven data pipeline, actively monitoring a predefined list of resources for any changes. When changes occur, it efficiently propagates these updates to the relevant endpoints.

flowchart LR

    subgraph Backend
    synchronizerserver["Synchronizer (server)"]
    pulsar[Pulsar]
    eventingester[Event Ingester]
    database[(Database)]
    synchronizerserver --- pulsar
    pulsar --- eventingester
    eventingester --- database
    end
    subgraph Cluster 1
    etcdcluster1[ETCD]
    synchronizercluster1["Synchronizer (client)"]
    etcdcluster1---synchronizercluster1
    end
    subgraph Cluster 2
    etcdcluster2[ETCD]
    synchronizercluster2["Synchronizer (client)"]
    etcdcluster2---synchronizercluster2
    end

    synchronizercluster2 --- synchronizerserver
    synchronizercluster1 --- synchronizerserver

Loading

Running the synchronizer locally

1. Run pulsar:

   ./scripts/pulsar.sh

2. Start synchronizer server:

   CONFIG=./configuration/server go run cmd/server/main.go

3. Start synchronizer client:

   SERVICES=./configuration/services.json CLUSTER_CONFIG=./configuration/clusterData.json CONFIG=./configuration/client go run cmd/client/main.go

Reconciliation after connecting

To repair cluster-to-backend gaps shortly after a connection is established, set backend.reconciliationTaskConfig.reconcileAfterConnectSeconds in the server's config.json:

{
  "backend": {
    "reconciliationTaskConfig": {
      "reconcileAfterConnectSeconds": 30
    }
  }
}

Each accepted connection schedules one reconciliation request after this delay, with 0–20% additional random jitter to spread reconnect bursts (30–36 seconds in this example). The delay starts when the backend adapter accepts the connection. Omitting the option or setting it to zero disables the feature; negative values also disable it. Values too large to represent the delay plus jitter are ignored with a warning. Clients must advertise synchronizer version v0.0.57 or newer, matching the existing reconciliation protocol requirement.

Pending requests are canceled when the connection closes, is replaced, or the server shuts down. Each request uses the single-client message even when sendBatch is enabled. This option works without a periodic schedule or minimum connection age; existing taskIntervalSeconds, cronSchedule, intervalFromConnectionSeconds, and periodic batching behavior stay unchanged. A periodic run may still request reconciliation near the one-off request.

Reconciliation treats the cluster as the source of truth. It does not replay backend-to-cluster commands. Cancellation prevents a pending send attempt; it cannot retract a request already enqueued to the message broker or guarantee that downstream reconciliation completes.

Backend message replay after reconnect

Server pods can retain backend messages for disconnected clients and replay them when the client connects. This covers reconnecting to another pod while the old pod still holds a dead connection. Enable it explicitly in the server config:

{
  "backend": {
    "replay": {
      "enabled": true,
      "ttlSeconds": 300,
      "maxMessages": 10000,
      "maxMessagesPerClient": 100,
      "maxBytes": 67108864
    }
  }
}

Replay is disabled by default. The limits above are the defaults when enabled; omitted or nonpositive limits use those defaults, as does an overflowing TTL. The byte limit counts retained payloads, property keys/values, and broker IDs across the pod; the count limits also bound per-entry metadata. Messages being replayed remain within these limits. Oldest queued messages are evicted when a limit is reached. An individual message larger than the byte budget is rejected; if all capacity is occupied by in-progress replay, a new message is rejected. Expired messages are never selected for replay and idle entries are swept every second. Shutdown releases the backlog as active sends finish.

Backlogs are isolated by account and cluster. The server replays supported PutObject, PatchObject, DeleteObject, GetObject, VerifyObject and reconciliation messages in receive order, before delivering later live messages for that client. An enqueue failure retains the failed message and remaining backlog until a later attempt or expiration, including a failed live enqueue. The server reads client responses while the backlog drains, and later live messages wait behind it. Malformed records are rejected before admission. With replay enabled, both Kafka and Pulsar use one handler worker regardless of consumerWorkers, preserving reader order. A slow client can therefore delay other backend traffic. Kafka ordering across different partitions is not defined.

Deploy updated clients before enabling server replay. Clients retain a cache of successfully applied mutation messages across socket reconnects. Configure it in the client config:

{
  "inCluster": {
    "deduplication": {
      "disabled": false,
      "ttlSeconds": 600,
      "maxMessages": 10000
    }
  }
}

Client deduplication is enabled by default with the limits shown; absent, nonpositive, or overflowing duration values use the defaults. Set disabled to true to turn it off. Only fixed-size SHA-256 fingerprints are retained. Entries expire from successful application time and are evicted oldest-first at capacity; duplicate arrivals do not extend their lifetime. Set retention longer than the server replay window and size the count for the client's mutation rate.

Senders must supply a stable, unique, nonempty MsgId for each logical message. The fingerprint includes the complete wire message because existing request/reply flows can share an ID across different operations. Exact mutation replays are skipped; changed contents with the same ID are not conflated. Failed applications remain retryable. Queries, reconciliation, and patches that request a full object for recovery can resend their responses instead of being suppressed.

This is bounded, in-memory recovery, not an application acknowledgement protocol. Cache eviction/expiry, process restarts, messages sent through a dead connection without a retained copy on the destination pod, missing IDs, and old clients limit the guarantee. The sender still cannot infer application success from enqueueing. Acknowledgements and sender retries remain a follow-up for confirmed delivery. The existing synchronizer.drops telemetry records replay_expired, replay_capacity, replay_oversized, and duplicate reasons.

Live namespace filtering

Synchronizer can watch a shared ConfigMap for namespace filter changes without a pod restart. This partially addresses kubescape/helm-charts#664; Helm wiring and live filtering in other components are separate changes.

Set inCluster.namespaceFilterConfigMapName to namespace-filters in the client's startup config.json. The default, an empty name, keeps existing startup-only filtering and requires no additional permissions. inCluster.namespace must be nonempty (normally populated from clusterData.json). Create the ConfigMap in that namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: namespace-filters
  namespace: kubescape
data:
  namespaceFilters.json: |
    {
      "includeNamespaces": [],
      "excludeNamespaces": ["kube-system"],
      "includeNamespacesRegex": [],
      "excludeNamespacesRegex": []
    }

This document matches the Operator's live-filter format. Both exact-name fields are required; regex fields are optional and reset to empty when omitted. Each field accepts a comma-separated string or an array of strings. Inclusion rules (exact names or regexes) take precedence over exclusions. Empty lists allow all namespaces. Regexes use Go syntax; surrounding whitespace and blank regex entries are ignored. Exact names are matched literally. Every valid document replaces all four lists together; startup lists are not merged into it.

Grant the Synchronizer service account permission to list and watch the named ConfigMap. Replace synchronizer below with the actual service account name:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: synchronizer-namespace-filters
  namespace: kubescape
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["namespace-filters"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: synchronizer-namespace-filters
  namespace: kubescape
subjects:
  - kind: ServiceAccount
    name: synchronizer
    namespace: kubescape
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: synchronizer-namespace-filters

Enabling this option initially requires deploying the updated binary and startup configuration. Subsequent edits to namespaceFilters.json take effect through a Kubernetes API watch, independent of ConfigMap volume refresh or subPath mounts. Synchronization waits for the first valid document; liveness remains available. Missing or invalid data and API/RBAC failures keep startup waiting. After the first valid document, invalid updates, deletion, and API outages retain the last valid rules. The watcher reconnects automatically. Malformed JSON, unknown fields, invalid types (including null), and invalid regexes reject the entire update. Applied changes are logged as namespace filters updated.

Live rules cover outgoing objects, patches, checksums, and fallback requests, including initial storage enumeration, queued events, backend reads, and reconciliation. The operator namespace and cluster-scoped resources remain exempt; existing parent-workload filtering is preserved. ConfigMap changes do not purge backend data, delete Kubernetes resources, trigger immediate resynchronization, or cancel already-dispatched messages. Newly included resources synchronize on subsequent events or reconciliation under the existing reconciliation rules. Backend-to-cluster writes keep their existing behavior, while outgoing responses containing excluded resource data are suppressed. Synchronizer controls resource synchronization; it does not itself stop scans in other components.

Cluster smoke test

1. Deploy a build containing this feature, the ConfigMap above, and its RBAC. Record the Synchronizer pod UID and container restart count. Use a disposable payments namespace outside the operator namespace with a Deployment that Synchronizer is configured to watch.
2. With both exact-name lists empty, change the Deployment and confirm its data reaches the test backend.
3. Run kubectl -n kubescape edit configmap namespace-filters and set excludeNamespaces to ["payments"]. Wait for namespace filters updated, change the Deployment again, and exercise backend reads and reconciliation. Confirm no updated object, patch, or checksum is sent for that namespace and existing backend records remain.
4. Clear exclusions, wait for the update log, and change the Deployment again. Confirm synchronization resumes. Re-exclude it and repeat the suppression check.
5. Submit an invalid regex, then delete and recreate the ConfigMap. Verify the last valid rules remain active until a new valid document is received.
6. Confirm the pod UID and restart count are unchanged throughout the filter edits. Repeat with missing ConfigMap permissions at startup, then restore RBAC and verify synchronization starts after a valid document loads.

A live cluster/backend is required for this smoke test; fake-client tests alone are not evidence that the Helm integration is complete.

API authentication

Every route except /healthz and /readyz requires Authorization: Bearer <Kubernetes service-account token> with audience kubescape.io. The token is validated with a TokenReview and the caller is authorized by RBAC (SubjectAccessReview) on kubescape.io resources: create on the resource named in the URL path (POST /apis/v1/<group>/<version>/<resource>).

• The service account running this component needs a ClusterRoleBinding to system:auth-delegator.
• KS_API_AUDIENCE: expected token audience (default kubescape.io).
• POD_NAMESPACE: required; the process exits at startup if it is unset.
• KS_API_AUTH_DISABLED=true: turns the check off.

To call the API by hand, mint a token for a service account that holds the matching Role:

TOKEN=$(kubectl create token <sa> -n <ns> --audience kubescape.io)
curl -H "Authorization: Bearer $TOKEN" ...

Design: in-cluster API authn/authz.

HTTP ingestion backpressure

The HTTP endpoint returns 429 Too Many Requests with Retry-After: 60 when its backend connection is known to be disconnected or reconnecting, or when admission capacity is exhausted. Route and method validation runs first; rejected requests are not read or parsed. Clients should respect Retry-After.

At most ten HTTP requests may read, parse, or wait for outbound dispatch at once. Non-alert resources (including network streams and node profiles) may occupy at most eight slots, reserving two slots for runtimealerts. Alerts share the same single WebSocket writer; the reservation provides admission capacity, not preemption or a delivery guarantee. Each request body is limited to 4 MiB; larger bodies receive 413 Request Entity Too Large.

Waiting for the writer ends when the request is cancelled or the synchronizer stops. WebSocket writes have a 30-second deadline so stalled writes eventually trigger reconnection. 202 Accepted means a message reached the outbound worker; backend delivery can still be retried after the HTTP response.

Operational metrics, tracing, logging, configuration and validation are documented in Operational telemetry.