Skip to content

Commit 6f592d4

Browse files
committed
Bundle application, workers, server, and Electron using Rollup
- Add UI components and credentials support for Remote MCP Basic and Bearer authentication - Convert core app and test sources from ES modules (.mjs) to TypeScript (.ts) - Expand isomorphic-git integration with new operations and repo management - Extract agent tools into modular definitions and index via a centralized registry - Implement HTTP request retry logic for network-bound tools - Queue ops replay on startup and browser reconnect - Re-architect test suites to execute against TypeScript using custom Jest resolvers - Reorganize Web Components into subdirectories with co-located styles/templates - Route task editor through store sync path
1 parent f33da4a commit 6f592d4

685 files changed

Lines changed: 83484 additions & 43719 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
.cache
22
.env
3+
assets/llamafile*
34
assets/v86*
45
coverage
6+
dist
7+
dist-electron
58
e2e-results
69
node_modules
7-
service-worker.js
8-
workbox-*.js
10+
out

‎AGENTS.md‎

Lines changed: 565 additions & 127 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 316 additions & 107 deletions
Large diffs are not rendered by default.

‎bin/build.sh‎

Lines changed: 41 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -3,34 +3,44 @@
33
# change directory to the project root
44
cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/.." || exit 1
55

6-
npm run test || exit 1
7-
8-
echo 'https://xt-ml.github.io/shadow-claw/' | node \
9-
bin/file-search-replace.mjs \
10-
'http://localhost:8888' \
11-
'manifest.json' || exit 1
12-
13-
echo 'base href="/shadow-claw/"' | node \
14-
bin/file-search-replace.mjs \
15-
'base href="/"' \
16-
'index.html' || exit 1
17-
18-
echo '#service-' | node \
19-
bin/file-search-replace.mjs \
20-
'service-' \
21-
'.gitignore' || exit 1
22-
23-
echo '#workbox-' | node \
24-
bin/file-search-replace.mjs \
25-
'workbox-' \
26-
'.gitignore' || exit 1
27-
28-
echo '// importScripts: [' | node \
29-
bin/file-search-replace.mjs \
30-
'importScripts: \[' \
31-
'./service-worker/workbox-config.cjs' || exit 1
32-
33-
node bin/touch-nojekyll.mjs \
34-
&& npm run build:pkg:meta "$(npm run -s build:pkg:get:meta)" \
35-
&& npm run -s clean:service-worker \
36-
&& npm run -s build:service-worker || exit 1
6+
npm run -s build:clean \
7+
&& mkdir -p dist/public/assets \
8+
|| exit 1
9+
10+
11+
# production only (only icons and screenshots get copied,
12+
# the rest of the assets are only used for local)
13+
if [[ "${NODE_ENV}" == "production" ]]; then
14+
cp -R assets/icons dist/public/assets/ \
15+
&& cp -R assets/screenshots dist/public/assets/
16+
else
17+
cp -R assets dist/public/
18+
fi
19+
20+
npm run -s rollup \
21+
&& npm run -s tsc \
22+
|| exit 1
23+
24+
# production only (only the manifest, index.html, and
25+
# metadata are updated for gh-pages)
26+
if [[ "${NODE_ENV}" == "production" ]]; then
27+
echo 'https://xt-ml.github.io/shadow-claw/' | node \
28+
bin/file-search-replace.mjs \
29+
'http://localhost:8888' \
30+
'dist/public/manifest.json' \
31+
|| exit 1
32+
33+
echo 'base href="/shadow-claw/"' | node \
34+
bin/file-search-replace.mjs \
35+
'base href="/"' \
36+
'dist/public/index.html' \
37+
|| exit 1
38+
39+
node bin/touch-nojekyll.mjs \
40+
&& npm run -s build:pkg:meta "$(npm run -s build:pkg:get:meta)" \
41+
|| exit 1
42+
fi
43+
44+
# build the service worker (all environments)
45+
npm run -s build:service-worker \
46+
|| exit 1

‎bin/file-search-replace.mjs‎

100755100644
File mode changed.

‎bin/rimraf.mjs‎

100755100644
File mode changed.

‎bin/touch-nojekyll.mjs‎

100755100644
Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,7 @@ import { fileURLToPath } from "node:url";
77

88
const __filename = fileURLToPath(import.meta.url);
99
const __dirname = dirname(__filename);
10-
const dataProjectRoot = join(__dirname, "..");
1110

12-
chdir(dataProjectRoot);
11+
chdir(join(__dirname, "..", "dist", "public"));
1312

1413
await writeFile(".nojekyll", "", "utf-8");

‎docs/README.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# ShadowClaw Documentation
2+
3+
> Deeper dives into the systems that make ShadowClaw tick — for humans _and_ agents alike.
4+
5+
## What's in here
6+
7+
### Architecture
8+
9+
How the core pieces fit together.
10+
11+
| Document | What it covers |
12+
| ------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
13+
| [System Overview](architecture/overview.md) | High-level architecture, data flow, design philosophy |
14+
| [Orchestrator & State Machine](architecture/orchestrator.md) | Main-thread state machine, message queue, invoke/compact lifecycle, EventBus |
15+
| [Worker Protocol](architecture/worker-protocol.md) | Worker ↔ main thread messages, tool-use loop, streaming, cancellation |
16+
| [Storage System](architecture/storage.md) | OPFS, local folders, write paths, zip export/import, group workspaces |
17+
| [Context Management](architecture/context-management.md) | Token estimation, dynamic windowing, output truncation, auto-compaction |
18+
| [Streaming](architecture/streaming.md) | SSE flow, StreamAccumulator, throttling, intermediate responses, proxy passthrough |
19+
20+
### Subsystems
21+
22+
Detailed docs for each major subsystem.
23+
24+
| Document | What it covers |
25+
| --------------------------------------------------------- | -------------------------------------------------------------------------------- |
26+
| [Shell Emulator](subsystems/shell.md) | JS shell via `just-bash` AST evaluation, OPFS bridge, supported commands |
27+
| [WebVM](subsystems/vm.md) | v86 Alpine Linux, boot modes, exclusivity guard, terminal bridge, 9p sync |
28+
| [Git Integration](subsystems/git.md) | isomorphic-git, LightningFS ↔ OPFS sync, merge conflicts, credentials |
29+
| [Channel System](subsystems/channels.md) | Channel registry, browser/Telegram/iMessage channels, router, multi-channel flow |
30+
| [Accounts & Credentials](subsystems/accounts.md) | Service account management, credential storage, auth bridges |
31+
| [Tools & Profiles](subsystems/tools.md) | Tool definitions, execution dispatch, profiles, adding new tools |
32+
| [Notifications & Scheduling](subsystems/notifications.md) | Web Push, VAPID, server-side SQLite scheduler, recursion guards |
33+
| [Providers & Adapters](subsystems/providers.md) | LLM provider registry, adapter pattern, OpenAI/Anthropic formats, Bedrock proxy |
34+
| [Electron Desktop](subsystems/electron.md) | Desktop app architecture, in-process server, power management |
35+
| [Reactive UI](subsystems/reactive-ui.md) | TC39 Signals, `effect()`, stores, `ShadowClawElement` base class, Web Components |
36+
| [Crypto](subsystems/crypto.md) | AES-256-GCM encryption for API keys at rest |
37+
38+
### Guides
39+
40+
Step-by-step instructions for common dev tasks.
41+
42+
| Document | What it covers |
43+
| ------------------------------------------------------------------- | ------------------------------------------------------------- |
44+
| [Adding a Provider](guides/adding-a-provider.md) | How to add a new LLM provider end-to-end |
45+
| [Adding a Tool](guides/adding-a-tool.md) | How to add a new agent tool |
46+
| [Adding a Shell Command](guides/adding-a-shell-command.md) | How to hook into the JS shell emulator |
47+
| [Adding a UI Page](guides/adding-a-page.md) | How to add a new Web Component page/section |
48+
| [Adding a Channel](guides/adding-a-channel.md) | How to add a new messaging channel |
49+
| [Service Accounts & Credentials](guides/adding-service-accounts.md) | How to manage encrypted credentials for channels and services |
50+
51+
### Decisions
52+
53+
The _why_ behind key choices.
54+
55+
| Document | Decision |
56+
| ----------------------------------------------------------------------------------- | --------------------------------------------------- |
57+
| [Bundled TypeScript Architecture](decisions/bundled-typescript-architecture.md) | Rationale for the transition to Rollup + TypeScript |
58+
| [Native Web Components and Signals](decisions/native-web-components-and-signals.md) | Why native standards for UI and reactivity |
59+
| [Worker-Isolated Agent Runtime](decisions/worker-isolated-agent-runtime.md) | Why the agent and VM run in dedicated workers |
60+
| [IndexedDB and OPFS Storage](decisions/indexeddb-and-opfs-storage.md) | Why IndexedDB and OPFS for persistent storage |
61+
62+
---
63+
64+
## How to use these docs
65+
66+
**Building ShadowClaw?** Start with the [System Overview](architecture/overview.md), then dive into whatever subsystem you're touching.
67+
68+
**Adding a feature?** Check the [Guides](#guides) section for step-by-step instructions.
69+
70+
**Wondering why something is the way it is?** The [Decisions](#decisions) section has you covered.
71+
72+
**AI agents working in this repo** should start with [AGENTS.md](../AGENTS.md) for conventions and guardrails, then come here for the deeper context. AGENTS.md is optimized for agent consumption; these docs are optimized for human understanding (though agents are welcome here too).
73+
74+
## Contributing to docs
75+
76+
- Keep docs accurate — update them when the code changes.
77+
- Architecture docs describe _what is_ and _how it works_.
78+
- Guides describe _how to do things_.
79+
- Decision docs describe _why decisions were made_ and are append-only (supersede, don't edit).
80+
- Use Mermaid diagrams generously — they render on GitHub and in most editors.
Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# Context Management
2+
3+
> Dynamic token-budget-aware context windowing replaces fixed message counts,
4+
> ensuring the LLM always gets the most relevant recent context within its token limit.
5+
6+
**Source:** `src/context/`
7+
8+
## Overview
9+
10+
Instead of sending a fixed number of recent messages, the orchestrator estimates token costs and builds the context window dynamically. Three modules work together:
11+
12+
```mermaid
13+
flowchart LR
14+
A[estimateTokens.ts] --> C[buildDynamicContext.ts]
15+
B[truncateToolOutput.ts] --> C
16+
C --> D[DynamicContextResult]
17+
D --> E{usagePercent > 80%?}
18+
E -->|yes + truncated| F[Auto-compact]
19+
E -->|no| G[Send to LLM]
20+
```
21+
22+
## Token Estimation
23+
24+
**File:** `src/context/estimateTokens.ts`
25+
26+
Lightweight heuristic: **~4 characters per token** (avoids shipping a full tokenizer).
27+
28+
```ts
29+
const CHARS_PER_TOKEN = 4;
30+
const MESSAGE_OVERHEAD_TOKENS = 4; // role, formatting
31+
32+
export function estimateTokens(text: string): number {
33+
return Math.ceil(text.length / CHARS_PER_TOKEN);
34+
}
35+
36+
export function estimateMessageTokens(message: StoredMessage): number {
37+
let contentTokens = 0;
38+
39+
if (typeof message.content === "string") {
40+
contentTokens = estimateTokens(message.content);
41+
} else if (Array.isArray(message.content)) {
42+
for (const block of message.content) {
43+
if (block.type === "text") contentTokens += estimateTokens(block.text);
44+
else if (block.type === "tool_use") {
45+
contentTokens += estimateTokens(block.name);
46+
contentTokens += estimateTokens(JSON.stringify(block.input));
47+
} else if (block.type === "tool_result") {
48+
contentTokens += estimateTokens(
49+
typeof block.content === "string"
50+
? block.content
51+
: JSON.stringify(block.content),
52+
);
53+
}
54+
}
55+
}
56+
57+
return contentTokens + MESSAGE_OVERHEAD_TOKENS;
58+
}
59+
```
60+
61+
## Output Truncation
62+
63+
**File:** `src/context/truncateToolOutput.ts`
64+
65+
Truncates large tool outputs at **line boundaries** (not mid-line):
66+
67+
```ts
68+
const TOOL_OUTPUT_MAX_CHARS = 25_000; // ~6K tokens
69+
70+
export function truncateToolOutput(
71+
content: string,
72+
maxChars = TOOL_OUTPUT_MAX_CHARS,
73+
): string {
74+
if (content.length <= maxChars) return content;
75+
76+
const slice = content.slice(0, maxChars);
77+
const lastNewline = slice.lastIndexOf("\n");
78+
79+
// Prefer clean line boundary if it's past 50% of budget
80+
const cutPoint = lastNewline > maxChars * 0.5 ? lastNewline : maxChars;
81+
const kept = content.slice(0, cutPoint);
82+
const truncated = content.length - cutPoint;
83+
84+
return `${kept}\n[...truncated ${truncated} chars]`;
85+
}
86+
```
87+
88+
**Why 25K chars?** A single tool output (e.g., reading a large file or `git_diff`) could consume the entire context window. The 25K limit (~6K tokens) ensures no single tool result dominates the budget while preserving enough content to be useful.
89+
90+
## Dynamic Context Building
91+
92+
**File:** `src/context/buildDynamicContext.ts`
93+
94+
### Algorithm
95+
96+
1. **Calculate available budget:**
97+
```text
98+
availableBudget = contextLimit − systemPromptTokens − maxOutputTokens
99+
```
100+
2. **Pre-truncate** all messages with large tool outputs
101+
3. **Walk newest-to-oldest**, accumulating token estimates:
102+
- Always include the most recent message (even if it exceeds budget)
103+
- Stop when the next message would exceed remaining budget
104+
4. **Return** the kept messages plus usage stats
105+
106+
```mermaid
107+
flowchart TD
108+
A[200 messages from DB] --> B[Truncate large tool outputs]
109+
B --> C[Walk newest → oldest]
110+
C --> D{tokens + msg fits budget?}
111+
D -->|yes| E[Include message, advance]
112+
E --> D
113+
D -->|no| F{Is this the only message?}
114+
F -->|yes| G[Include anyway]
115+
F -->|no| H[Stop here]
116+
G --> I[Return result]
117+
H --> I
118+
```
119+
120+
### Result shape
121+
122+
```ts
123+
interface DynamicContextResult {
124+
messages: StoredMessage[]; // Messages that fit in the budget (newest preserved)
125+
estimatedTokens: number; // Total tokens used by kept messages
126+
truncatedCount: number; // Number of older messages dropped
127+
contextLimit: number; // Model's context window size
128+
usagePercent: number; // 0–100 (rounded to 1 decimal)
129+
}
130+
```
131+
132+
## Auto-Compaction
133+
134+
The orchestrator monitors `usagePercent` from `buildDynamicContext()`:
135+
136+
| Condition | Action |
137+
| ----------------------------------------------------------------------- | --------------------------------------- |
138+
| `usagePercent > 80%` AND messages were truncated AND >10 total messages | Queue compaction via `queueMicrotask()` |
139+
| `usagePercent ≤ 80%` | Normal operation |
140+
141+
Compaction summarizes the conversation into a single message, freeing token budget for future messages while preserving important context.
142+
143+
## UI Integration
144+
145+
The orchestrator emits a `context-usage` event with `ContextUsage` data:
146+
147+
```ts
148+
interface ContextUsage {
149+
estimatedTokens: number;
150+
contextLimit: number;
151+
usagePercent: number;
152+
truncatedCount: number;
153+
systemPromptTokens: number;
154+
maxOutputTokens: number;
155+
}
156+
```
157+
158+
The chat component renders a color-coded progress bar:
159+
160+
| Usage | Color | Meaning |
161+
| ------- | ----- | ------------------------ |
162+
| 0–60% | Green | Plenty of room |
163+
| 60–80% | Amber | Getting full |
164+
| 80–100% | Red | Auto-compaction imminent |
165+
166+
## Context Limits by Model
167+
168+
Context limits are resolved per model family via `getContextLimit(model)` in `src/providers.ts`:
169+
170+
| Model Pattern | Context Limit |
171+
| --------------------------------- | ---------------- |
172+
| `claude` (all) | 200,000 tokens |
173+
| `gemini-1.5-pro`, `gemini-2` | 1,048,576 tokens |
174+
| `gemini` (other) | 32,768 tokens |
175+
| `gpt-4o`, `o1`, `o3`, `o4` | 128,000 tokens |
176+
| `gpt-4-turbo` | 128,000 tokens |
177+
| `gpt-4` | 8,192 tokens |
178+
| `gpt-3.5-turbo` | 16,385 tokens |
179+
| `llama-3.1-405b` | 131,072 tokens |
180+
| `llama` | 131,072 tokens |
181+
| `amazon.nova` | 300,000 tokens |
182+
| `mistral-large`, `command-r-plus` | 128,000 tokens |
183+
| `phi-4`, `gemini-nano` | 4,096 tokens |
184+
| Default fallback | 4,096 tokens |

0 commit comments

Comments
 (0)