[BUG] Declarative agents with skills append -full to IMAGE_TAG, producing an invalid reference when the agent image is digest-pinned #2586

What happened

When the agent runtime image is configured with a digest-pinned tag, every
declarative agent that declares spec.skills fails to roll out. The controller
composes the full-variant image by string-appending -full to IMAGE_TAG,
which produces a reference with the suffix after the digest:

ghcr.io/kagent-dev/kagent/app:0.10.0-rc3@sha256:0c1d336bd8e9c1d6542b002f2309d38c7be9e80c100e869b7811a6af17a55bce-full

An OCI digest is sha256: followed by hex and nothing may follow it, so this is
not a parseable image reference. The Deployment is created but no pod is ever
produced: the ReplicaSet stays at 0, the Deployment reports ReplicaFailure=True
with FailedCreate and eventually ProgressDeadlineExceeded. Any previously
running pod keeps serving, so the agent can look healthy while being unable to
roll forward.

On clusters with an admission controller that parses image references, the pod
CREATE is rejected outright at admission with invalid reference format. Without
such a controller the kubelet rejects it instead (InvalidImageName). Either way
the agent never starts.

Note the composed reference is wrong in two ways: besides the invalid syntax, it
pairs the digest of the non-full image with a -full suffix. The two tags are
genuinely different images — for 0.10.0-rc3, app:0.10.0-rc3 is
sha256:0c1d336bd8e9c1d6542b002f2309d38c7be9e80c100e869b7811a6af17a55bce and
app:0.10.0-rc3-full is
sha256:2c32b111dddb35093c2df9db353b3b2eca263bd8e850cff13bc52a6a65f3d014.

Code path

go/core/internal/controller/translator/agent/deployments.go (v0.10.0-rc3):

func resolveRuntimeImage(registry, repository, tag, digest, imageLabel string, full, pinDigest bool) (string, error) {
	if !pinDigest {
		if full {
			tag += "-full"
		}
		return fmt.Sprintf("%s/%s:%s", registry, repository, tag), nil
	}
	if d := normalizeImageDigest(digest); d != "" {
		return fmt.Sprintf("%s/%s@%s", registry, repository, d), nil
	}
	...
}

tag is DefaultImageConfig.Tag, which comes from the chart's IMAGE_TAG
(controller-configmap.yaml:
IMAGE_TAG: {{ coalesce .Values.controller.agentImage.tag .Values.tag .Chart.Version | quote }}).
Both the concatenation and the %s/%s:%s format assume IMAGE_TAG is a bare tag,
matching APP_FULL_IMAGE_TAG ?= $(VERSION)-full in the Makefile. Nothing validates
or rejects a digest-bearing value.

full is set in resolveInlineDeployment:

full := needsSRTSettings(agent, specRef.Sandbox)
pinDigest := agent.GetWorkloadMode() == v1alpha2.WorkloadModeSandbox

and needsSRTSettings returns true whenever spec.Skills != nil, so any
skills-enabled agent takes the -full path.

No supported way to digest-pin declarative agent images

--app-image-digest / --app-full-image-digest (and the golang-adk equivalents)
are only read in the pinDigest branch, and pinDigest is true only for
sandbox agents. So for declarative agents:

• the digest overrides are ignored, and
• the only way to express a digest is to embed it in IMAGE_TAG, which is exactly
  what breaks the -full case.

There is also no value of IMAGE_TAG that works for both: a normal agent renders
repo:TAG@sha256:HEX (valid, and this is why non-skills agents keep working), but
appending -full to that same string is invalid by construction. A single global
setting cannot satisfy both, so an operator who needs digest-pinned agent images
currently cannot use skills at all.

Why this surfaced in 0.10

-full selection is not new, but it used to be reachable only from the Go runtime.

Extending full-variant selection to the Python runtime (#2242) made the latent
concatenation bug reachable for Python agents, which is where it started biting.

Steps to reproduce

1. Install the chart with a digest-pinned agent image, e.g.
   controller.agentImage.tag=0.10.0-rc3@sha256:0c1d336bd8e9c1d6542b002f2309d38c7be9e80c100e869b7811a6af17a55bce.
   (Confirm IMAGE_TAG in ConfigMap/kagent-controller carries the digest.)
2. Apply any declarative Agent with a spec.skills.gitRefs entry.
3. Inspect the generated Deployment / its conditions.

Expected: the agent's container references the published full-variant image
(…/app:0.10.0-rc3-full, or its digest).

Actual: the container image is
…/app:0.10.0-rc3@sha256:0c1d…-full, which is not a valid reference, and the
agent never rolls out.

Agents without spec.skills on the same cluster are unaffected, because
repo:TAG@sha256:HEX happens to be a valid reference form.

Environment

• kagent 0.10.0-rc3 (controller and chart)
• Agent runtime: python, type: Declarative, non-sandbox workload mode
• Agent images pulled from ghcr.io/kagent-dev/kagent/*, pinned by digest

Suggested fix direction

Not prescriptive — a few directions that would each resolve it:

• Parse the configured reference properly (split tag / digest) instead of
  concatenating, and derive the full variant from the tag component, dropping
  or replacing a digest rather than appending after it.
• Provide a distinct setting for the full-variant image (e.g. a
  controller.agentFullImage.tag / APP_FULL_IMAGE_TAG equivalent surfaced in the
  chart) so operators can pin both variants explicitly and independently.
• Alternatively, honour --app-full-image-digest / --golang-adk-full-image-digest
  for declarative agents too, which would give digest-pinning parity with sandbox
  agents.

At minimum, it would help to fail fast with a clear error when IMAGE_TAG contains
a digest and a full-variant image is required, rather than emitting an unparseable
reference that only fails later at admission or on the kubelet.

I'm happy to put up a PR if you can indicate which direction you'd prefer.