Skip to content

[BUG] Git, OCI, and S3 Agent Plugin resources cannot be materialized while creating Substrate golden Actors #2604

New issue
New issue
Open
Bug
Open
[BUG] Git, OCI, and S3 Agent Plugin resources cannot be materialized while creating Substrate golden Actors#2604
Bug

Description

@supreme-gg-gg
supreme-gg-gg
opened on Aug 27, 2026

📋 Prerequisites

  • I have searched the existing issues to avoid creating a duplicate
  • By submitting this issue, you agree to follow our Code of Conduct
  • I am using the latest version of the software
  • I have tried to clear cache/cookies or used incognito mode (if ui-related)
  • I can consistently reproduce this issue

🎯 Affected Service(s)

App Service

🚦 Impact/Severity

No impact (Default)

🐛 Bug Description

Kagent advertises Git, OCI, and S3 sources for Agent Plugin resources, but an
AgentTemplate using a network-backed source cannot currently become ready on
the Substrate runtime. Both the Go ADK runtime and the Claude harness materialize
their selected resources during process startup, before their readiness endpoint
is available. Outbound networking is not usable at that point in the golden
Actor lifecycle, so the fetch fails with a closed connection and the process
exits.

This leaves the AgentTemplate pending, the golden Actor in RESUMING, and its
worker occupied. The failure is reproduced with both a public Git repository
and a public, digest-pinned OCI artifact. It does not depend on source
credentials.

🔄 Steps To Reproduce

  1. Configure a Kagent AgentTemplate using the Go ADK harness and select an
    Agent Plugin whose source is a public Git repository pinned to a commit.
  2. Apply the template and wait for its generated ActorTemplate to build a
    golden Actor.
  3. Observe that the template never becomes ready and the golden Actor remains
    in RESUMING.
  4. Inspect the golden Actor's container logs.
Initialized empty Git repository in /plugins/plugin-0/.git/
fatal: unable to access 'https://github.com/agentplugins/agent-plugins-example.git/': Send failure: Broken pipe
Failed to materialize Agent Plugins: materialize plugin 0: exit status 128

Observed with a public OCI skill artifact pinned by digest:

materialize skill "myskill": pull ghcr.io/supreme-gg-gg/myskill@sha256:...:
Get "https://ghcr.io/v2/": EOF
Failed to materialize Agent Plugins

The same class of failure occurs in the Claude harness while materializing a
plugin-selected skill, although its Git/TLS implementation reports the closed
connection as a TLS handshake failure.

🤔 Expected Behavior

An AgentTemplate using a valid Git, OCI, or S3 Agent Plugin resource reaches
Ready, and the plugin's selected skills and MCP configuration are available
to the agent. Golden creation must support any network access that Kagent
requires to materialize declared plugin resources.

📱 Actual Behavior

The runtime process attempts the Git fetch before it serves readiness. The
connection is closed during startup, materialization fails, and the process
exits before it can ever become ready. Golden creation consequently cannot
complete.

💻 Environment

No response

🔧 CLI Bug Report

No response

🔍 Additional Context

Temporary Workarounds:

  • Bake the required plugin or skill content into the harness image and avoid a
    runtime network fetch. This sacrifices normal Agent Plugin source semantics
    and requires a new image for every resource revision.
  • Pre-populate the exact materialization destination in a custom image only if
    the runtime's validation and path expectations remain satisfied. This is
    brittle and should be treated as a development-only workaround.

📋 Logs



📷 Screenshots


No response


🙋 Are you willing to contribute?


    

  •  I am willing to submit a PR to fix this issue
    • 

Activity
  1. supreme-gg-gg
    added the
    Bug
    issue type on Aug 27, 2026
  2. github-project-automation
    added this to  kagent kanbanon Aug 27, 2026
  3. github-project-automation
    moved this to Backlog in  kagent kanbanon Aug 27, 2026
  4. supreme-gg-gg
    changed the title [-][BUG] Git-backed Agent Plugins cannot be materialized while creating Substrate golden Actors[/-] [+][BUG] Git, OCI, and S3 Agent Plugin resources cannot be materialized while creating Substrate golden Actors[/+] on Aug 27, 2026
  5. supreme-gg-gg commented on Aug 27, 2026 
    @supreme-gg-gg
    supreme-gg-gg
    on Aug 27, 2026
    ContributorAuthor
    Note that in the case of Claude I was able to ask the Agent to clone the plugin repo and setup the plugin using the bash tool so it's definitely not a wrong harness setup. The issue is the network egress limitation by Substrate during golden image build.
    

    Codex looked into upstream code and wrote some analysis, I will paste the following core idea here for record-keeping:
    

    

    cmd/ateom-gvisor/main.go implements RunWorkload in this order:
    

      

    1. It explicitly deactivates Actor networking at lines 604-606.
      2. 

    2. It prepares the Actor's authenticated egress identity and installs the
      
network redirect at lines 621-629.
      3. 

    3. It creates and starts every application container at lines 679-700.
      4. 

    4. It waits for every configured readiness endpoint at lines 702-705.
      5. 

    5. Only after readiness succeeds does it call activateActorNetworking at
      
lines 706-708.
      6. 

    

    Preparing the redirect is not equivalent to allowing egress. In
    
internal/atunnel/egress.go, Egress.handle reads the current active tunnel at
    
lines 238-240 and immediately closes the downstream connection when it is nil at
    
lines 241-245. The harness therefore sees a TCP/TLS connection begin and then
    
close, which accounts for the Git Broken pipe, OCI EOF, and GnuTLS handshake
    
variants.
    

    This produces a direct circular dependency:
    

    Kagent readiness requires remote resource materialization
        -> materialization requires Actor egress
        -> Actor egress activation requires readiness

    

  6. erauner12 commented on Sep 20, 2026 
    @erauner12
    erauner12
    on Sep 20, 2026
    Contributor
    Confirmed this on:
    

      

    • kagent v1.0.0-alpha1
      • 

    • Substrate v0.2.0-beta4
      • 

    • Codex harness
      • 

    

    I tested the same minimal, digest-pinned OCI skill through two independent registries. The artifact could be pulled and inspected successfully outside the Actor, but both registries failed identically during golden Actor creation:
    

    materialize Codex skills: materialize skill "<redacted>":
pull <registry>/<artifact>@sha256:<redacted>:
Get "https://<registry>/v2/": EOF

    

    The AgentTemplate conditions reached:
    

      

    • Accepted=True
      • 

    • ResolvedRefs=True
      • 

    • Compatible=True
      • 

    • Ready=False, reason ActorTemplatePending
      • 

    

    The latest successful pre-skill revision remained available, while the skill-enabled desired revision never produced a golden snapshot.
    

    One potentially useful diagnostic: the Substrate egress gateway recorded no CONNECT or registry request during the failed materialization attempts. Normal Actor traffic after startup did appear in the gateway logs and was authenticated and routed correctly. This seems consistent with materialization occurring before outbound Actor networking becomes usable, rather than a registry, credential, certificate, or artifact-content problem.
    

    The test artifact was intentionally minimal—a scratch image containing only SKILL.md—so image size and extraction complexity were not contributing factors.
  7. QuentinBisson
    added 5 commits that reference this issue on Sep 24, 2026
    feat(substrate): the compiled ActorTemplate carries the revision's eg…
    db1a4d9
    feat(substrate): the compiled ActorTemplate carries the revision's eg…
    59193b5
    feat(substrate): the compiled ActorTemplate carries the revision's eg…
    1df24f3
    feat(substrate): the compiled ActorTemplate carries the revision's eg…
    47b9791
    feat(substrate): the compiled ActorTemplate carries the revision's eg…
    fdf8705
  8. ProbstenHias commented on Oct 4, 2026 
    @ProbstenHias
    ProbstenHias
    on Oct 4, 2026
    We hit the same thing on:
    

      

    • kagent v1.0.0-alpha7
      • 

    • Substrate 0.3.0-alpha3
      • 

    • Codex harness, local kind cluster
      • 

    

    We tried a public Git skill source pinned to a full commit, and a minimal digest-pinned OCI skill (a FROM scratch image with only SKILL.md) on two public registries. All attempts failed the same way during golden Actor creation:
    

    fatal: unable to access 'https://github.com/anthropics/skills/': Send failure: Broken pipe
configure Codex Harness: materialize Codex skills: materialize skill "mcp-builder": exit status 128

configure Codex Harness: materialize Codex skills: materialize skill "roast-style-guide": pull docker.io/<user>/skill-roast-style-guide@sha256:...: Get "https://index.docker.io/v2/": EOF

configure Codex Harness: materialize Codex skills: materialize skill "roast-style-guide": pull ttl.sh/<image>@sha256:...: Get "https://ttl.sh/v2/": EOF

    

    The Agent ends with Ready=False, reason ActorTemplateFailed, GoldenActorCrashed: golden actor crashed before its snapshot was taken. As in the comment above, the egress gateway has no request for these fetches. Normal Actor traffic after startup is logged and routed correctly. The only workaround we found is to remove spec.skills from the AgentTemplate.
    

    A related gap that will probably show up once materialization has egress: the translator adds exactly one origin per OCI source (appendArtifactSourceDestination in go/core/internal/translator/skills.go). For Docker Hub refs it adds https://registry-1.docker.io:443. A real pull with go-containerregistry, the library the harness uses, needs more hosts:
    

      

    • Docker Hub: index.docker.io (go-containerregistry's default registry name for docker.io, so it does not even hit registry-1.docker.io), auth.docker.io for the anonymous pull token, and production.cloudfront.docker.com for blob redirects.
      • 

    • GHCR: ghcr.io, plus pkg-containers.githubusercontent.com for blob redirects.
      • 

    

    I checked this with crane --verbose outside the cluster. I could not test it inside the cluster, because the startup bug fails first. Because the egress policy is derived and immutable per revision, users cannot add these hosts themselves. So the fix probably needs to allowlist the token realm and blob redirect hosts for OCI sources, or materialize skills outside the Actor's egress policy (for example in the controller).
  9. QuentinBisson
    added 10 commits that reference this issue on Oct 5, 2026
    feat(substrate): the compiled ActorTemplate carries the revision's eg…
    60277f9
    fix(a2agateway): end a send whose runtime is lost, and delete a pause…
    8d6384e
    fix(session): end a working turn whose runtime stopped saving events
    01dd828
    feat(session): suspend a runtime paused for input once the pause is o…
    9b9f776
    fix(session): leave an expired pause on a lost node alone, and delete…
    5ec449e
    feat(substrate): the compiled ActorTemplate carries the revision's eg…
    ae3efea
    fix(a2agateway): end a send whose runtime is lost, and delete a pause…
    5d3a144
    fix(session): end a working turn whose runtime stopped saving events
    b878562
    feat(session): suspend a runtime paused for input once the pause is o…
    213f267
    fix(session): leave an expired pause on a lost node alone, and delete…
    adfa896
  10. marcellodesales commented on Oct 16, 2026 
    @marcellodesales
    marcellodesales
    on Oct 16, 2026
    Reporting on the same here:
    

    kagent v1.0.0-alpha13
    
Substrate v0.5.0
    
Claude-code harness
    

    Is there a direction to this?
  11. supreme-gg-gg commented on Oct 17, 2026 
    @supreme-gg-gg
    supreme-gg-gg
    on Oct 17, 2026
    ContributorAuthor
    OCI skills should be available after #3203 which just merged, can you try v1 alpha14?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Metadata
Assignees
No one assigned
    Labels
    No labels
    No labels
    Type
    Bug
    Projects
    Milestone
    No milestone
    Relationships
    None yet
    Development
    No branches or pull requests
    Issue actions