[FEATURE]:Performance: inefficient repository exploration and repeated static prompt context cause excessive inference and tool-call overhead #37767

Feature hasn't been suggested before.

• I have verified this feature I'm about to request hasn't been suggested before.

Describe the enhancement you want to request

Summary

OpenCode appears to perform significantly more model inference and tool calls than necessary for repository-wide analysis tasks.

When testing OpenCode against a local vLLM-served model, I observed two related sources of overhead:

1. Repository-wide questions are often handled by reading files individually instead of first using repository-wide search, indexing, or batched inspection.
2. Large static context blocks, including project instructions, skill descriptions, and verbose tool definitions, appear in every model request.

The model itself is not the main bottleneck. vLLM metrics show that decoding remains reasonably fast, but OpenCode repeatedly invokes the model for low-information-gain decisions such as selecting and reading the next individual file.

In the same repository and with the same model, another coding-agent client completed a comparable analysis much faster by first using a repository-wide grep and then reading only relevant files.

This may be particularly visible with local models because inference cost, prompt size, prefix-cache behavior, and tool-call count are directly observable.

Environment

• OpenCode: v1.18.3
• Interface tested:
  • Desktop
  • TUI
• OS: Windows
• Backend: vLLM OpenAI-compatible API
• Model: NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4
• Context length: 262,144 tokens
• Speculative decoding: enabled
• Prefix caching: enabled
• Comparison client: Continue
• Both clients called the same vLLM endpoint and model directly

The Desktop interface appeared somewhat slower than the TUI, but the inefficient exploration behavior was present in both interfaces. This suggests that renderer overhead is secondary to the agent loop and context construction.

Example task

The task was a repository-wide analysis that required identifying comments and understanding the overall codebase.

A search-first implementation can reduce the candidate set with commands or tools equivalent to:

rg -l '//|/\*' <source directories>

It can then inspect representative matches and relevant files.

The comparison client followed approximately this strategy:

Search the repository for comment patterns.
Identify matching and non-matching files.
Read a small number of relevant files.
Perform semantic analysis on the reduced result set.

OpenCode instead followed approximately this pattern:

Read one entrypoint file.
Reason about which file to inspect next.
Read that file.
Invoke the model again.
Repeat for many files.

For example:
read main.tsx
model inference
read App.tsx
model inference
read another imported file
model inference
...

This turns a deterministic repository scan into a long sequence of model round trips.

Observed behavior from vLLM logs

The following values are approximate observations from the test runs, not controlled benchmark results.

OpenCode
Backend generation throughput was generally around 23–23.5 tokens/s.
Speculative decoding mean acceptance length was approximately 2.4–2.5.
Average draft-token acceptance was approximately 46–49%.
The agent issued many sequential tool calls.
Static instructions and tool definitions were present in repeated requests.
The complete repository-analysis task took approximately 30 minutes.
Comparison client
Backend generation throughput was approximately 27.8–28.1 tokens/s.
Speculative decoding mean acceptance length was approximately 2.6–2.7.
Average draft-token acceptance was approximately 54–58%.
It used a repository-wide search followed by only a few targeted reads.
The comparable task completed in approximately 5 minutes.

The roughly 20% decode-throughput difference does not explain the approximately 6x task-duration difference.

The dominant difference appears to be:

number of agent-loop iterations
× repeated prompt processing
× tool-call granularity
× accumulated transcript size

In other words, the model was generating at an acceptable speed, but OpenCode asked it to make too many small decisions.

Repeated static prompt context

vLLM request logging also shows large static blocks included in repeated requests, including:

project or AGENTS.md instructions
available skill names and descriptions
full tool descriptions and JSON schemas
extensive shell and PowerShell usage guidance
Git and GitHub rules
prior tool calls and tool results

One tool description alone contains extensive instructions about:

PowerShell syntax
path quoting
directory verification
file-operation restrictions
command sequencing
output truncation
Git workflow
pull-request workflow

These instructions may be useful, but resending them as model-visible prose on every agent iteration has several costs:

They continue to occupy the model context window even when prefix caching succeeds.
Small changes in tool ordering, prompt composition, dynamic metadata, or message boundaries may reduce prefix-cache reuse.
Verbose policy text competes with task-relevant repository context.
Instructions such as preferring individual Read, Glob, and Grep tools may unintentionally bias the model toward many granular operations.
Tool outputs accumulate in the transcript, increasing the prompt processed by later iterations.

I understand that stateless model APIs generally require instructions to remain available on every request. The issue is not merely that static instructions are present, but that they are verbose, repeatedly assembled, and combined with a high-iteration exploration strategy.

Expected behavior

For repository-wide questions, OpenCode should prefer high-information-gain deterministic operations before individual file reads.

A preferred workflow would be:

1. Inspect repository metadata and manifests.
2. List the relevant directory structure.
3. Use repository-wide search or an existing index.
4. Build a candidate file set.
5. Read candidates in batches.
6. Use the model for semantic interpretation.

Individual file reads should generally occur only after a broad search, index query, import-graph query, or repository survey has reduced the search space.

For example:
repository-wide question
↓
search/index/AST query
↓
candidate file set
↓
batch read
↓
semantic analysis

Rather than:
repository-wide question
↓
read one file
↓
model decides next file
↓
read one file
↓
repeat

Suggested improvements

1. Add a search-first repository exploration policy

Before reading files individually, classify whether the task can be reduced using:

content search
file metadata
symbol search
import/dependency graph
AST extraction
repository manifests
batched file reads

For repository-wide classification tasks, require a broad scan before sequential reads unless the model provides a reason why scanning is inappropriate.

This could be implemented as agent-loop policy rather than another long prompt instruction.

Example heuristic:

if task scope is repository-wide:
require one of:
repository search
symbol/index query
repository survey
batch metadata query
before repeated individual reads

2. Provide a repository survey tool

A single structured tool could return:

languages:
entrypoints:
manifests:
directory_summary:
major_modules:
symbols:
imports:
candidate_files:

This would allow the model to understand the project shape with one tool round trip instead of navigating the import tree file by file.

Possible interface:

{ "tool": "workspace.inspect", "arguments": { "depth": 3, "include_manifests": true, "include_entrypoints": true, "include_symbols": true, "include_import_graph": true, "max_files": 200 } }

3. Support batched reads

Allow the model to request multiple files or ranges in one tool call:
{ "tool": "workspace.read_batch", "arguments": { "files": [ "src/main.tsx", "src/App.tsx", "src/router.ts" ], "max_lines_per_file": 200 } }
The current model-tool-model sequence creates unnecessary latency when several reads are independent.

4. Keep the prompt prefix deterministic

OpenAI describes Codex as relying on exact-prefix prompt caching and specifically notes that nondeterministic tool ordering caused expensive cache misses. Tool definitions and stable instructions should remain byte-for-byte deterministic and appear before dynamic conversation content.

Reference:

OpenAI, “Unrolling the Codex agent loop”
https://openai.com/index/unrolling-the-codex-agent-loop/

Relevant implementation principles include:

stable prompt-prefix layout
deterministic tool ordering
dynamic content appended after stable content
context compaction for long-running sessions
avoiding mid-session mutations that invalidate the cached prefix
5. Reduce model-visible runtime policy

Rules that can be enforced by the runtime do not need to be repeatedly explained to the model in verbose prose.

For example, the runtime can enforce:

sandbox boundaries
path validation
permission checks
destructive-command restrictions
timeout limits
output-size limits
Git operation approval
allowed working directories

The model-facing tool description can then remain short:

Execute an approved command in the selected working directory.
The runtime validates paths, permissions, timeouts, and sandbox policy.

This reduces both context usage and instruction noise.

6. Load detailed skills and guidance on demand

At session start, expose only compact skill metadata:
name:
description:

Load the full skill body only after the model selects that skill.

Similarly, detailed Git, PowerShell, and tool guidance could be retrieved only when relevant rather than included in every inference request.

7. Add agent-loop observability

It would help users and maintainers diagnose this behavior if OpenCode exposed:

number of model requests
prompt tokens per request
cached prompt tokens or prefix-cache hit rate
generated tokens
tool-call count by tool
time spent in prefill
time spent decoding
time spent executing tools
context growth by iteration
repeated file reads
repeated or overlapping searches

A per-run summary might look like:
Model calls: 34
Tool calls: 31
Individual file reads: 24
Repository searches: 1
Input tokens processed: ...
Output tokens generated: ...
Tool execution time: ...
Model inference time: ...

This would make inefficient agent behavior visible even when a fast hosted model hides the latency.

Related implementation patterns
OpenAI Codex

OpenAI's published description of the Codex agent loop documents several relevant techniques:

exact-prefix prompt caching
deterministic tool ordering
stable instructions before dynamic context
stateless agent iterations with cache reuse
automatic context compaction

Reference:

https://openai.com/index/unrolling-the-codex-agent-loop/

Codex still makes instructions available across model turns, but it treats prompt layout and cache stability as a performance-sensitive interface.

Claude Code / other coding agents

Claude Code exposes repository-oriented tools such as file search, content search, shell execution, and file reads. In practice, effective coding-agent workflows use search or indexing to reduce the repository before performing targeted reads.

The important pattern is not a specific vendor implementation but:

deterministic retrieval first
semantic reasoning second

Search, grep, symbols, AST queries, and metadata should perform work that does not require model inference.

Why this matters

Hosted frontier models can partially hide an inefficient agent loop through:

high generation throughput
provider-side prompt caching
large context windows
highly capable tool-selection behavior

Local models make the cost more visible, but the same inefficiency still affects hosted models through:

increased token usage
longer task latency
unnecessary context growth
reduced cache efficiency
more opportunities for the agent to drift
lower information density in the transcript

This is therefore not only a local-model optimization.

It is an agent-runtime efficiency issue.

Reproduction material

I can provide redacted vLLM request and metrics logs showing:

repeated static prompt content
tool definitions in successive requests
request-level sampling parameters
generation throughput
speculative decoding acceptance
prefix-cache metrics
agent iteration timing

I can also run a smaller controlled reproduction if maintainers specify the preferred OpenCode version, task, logging configuration, and model endpoint.

Before sharing the full logs, I will redact local paths and project-specific content.

Questions
Is repository exploration strategy primarily controlled by OpenCode's system prompt, model-specific prompt, or hard-coded agent-loop policy?
Is there currently any heuristic requiring repository-wide search before repeated individual reads?
Are tool definitions and project instructions guaranteed to have stable ordering and byte-identical serialization across requests?
Does OpenCode track prompt-cache reuse or repeated static-prefix size?
Would the maintainers be open to a repository survey tool, batched reads, and agent-loop efficiency telemetry?

A fast hosted model can hide the latency, but it cannot eliminate the unnecessary model invocations or token consumption.

Thank you for considering this. I would be happy to help validate a fix against the same local vLLM setup.