Skip to content

Commit 062f88e

Browse files
steipetesteipete
authored
refactor: extract reusable AI runtime package (#99059)
* refactor: extract reusable AI runtime package * refactor: complete AI provider relocation * refactor: keep llm core internal * refactor(ai): make @openclaw/ai self-contained with host policy ports Move pure transport helpers (tool projections, strict-schema normalization, prompt-cache boundary, stream guards, anthropic/openai compat, request activity) from src into packages/ai; move utf16-slice into normalization-core. Inject host policy (guarded fetch, redaction, strict-tool defaults, diagnostics logging) through AiTransportHost with inert library defaults installed by src/llm/stream.ts. Narrow the public barrel to instance-scoped createApiRegistry/createLlmRuntime; the process-default runtime moves behind internal/ and registerBuiltInApiProviders takes an explicit registry. Delete the src/llm/api-registry re-export facade. * fix(ai): teach node, jiti, and vite resolvers the @openclaw/ai and utf16-slice subpaths The workspace alias tables in root-alias.cjs, plugin-sdk-native-resolver, sdk-alias, the shared vitest config, and the Control UI vite config only knew @openclaw/llm-core; Node-side plugin loading resolved @openclaw/ai through the pnpm symlink to the unbuilt dist (checks-node-compact CI failures), and the Control UI build broke on the new normalization-core/utf16-slice subpath. * chore(ui): drop leftover service-worker debug logging * build(release): ship @openclaw/ai with its own shrinkwrap and honest dependency set packages/ai declares only its six real runtime deps (kysely, chalk, json5, tslog, zod, fs-safe, and proxyline were never imported); orphaned root deps removed. generate-npm-shrinkwrap now treats publishable packages/* like publishable plugins so the AI tarball pins its transitive tree even though workspace deps are omitted from the root shrinkwrap. knip learns the package entry points; the tsdown dts neverBundle option moves to its documented deps.dts home; the README documents the no-semver internal/* contract and host ports. * docs(ai): add minimal external-consumer example app examples/ai-chat consumes only the public @openclaw/ai surface (built dist via the workspace link): isolated runtime, built-in provider registration, one streamed completion. Supports Anthropic/OpenAI via env keys and a keyless local Ollama target; live-verified against Ollama. * docs(ai): document the @openclaw/ai package and workspace shrinkwrap boundary * chore(check): include examples/ in duplicate-scan targets * fix: emit normalization package subpaths * fix: complete AI package boundary artifacts * fix: align AI package boundary contracts * fix(ci): stabilize package release contracts * test: align documentation contract checks * test: keep cron docs guard aligned * test: align restored docs contract guards * test: follow upstream docs contracts * docs: drop superseded talk wording
1 parent 9133118 commit 062f88e

230 files changed

Lines changed: 3123 additions & 1249 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/codeql/codeql-mcp-process-tool-boundary-critical-security.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ paths:
2828
- src/agents/agent-tools.abort.ts
2929
- src/agents/agent-tools.before-tool-call*.ts
3030
- src/agents/agent-tools.read.ts
31-
- src/agents/agent-tools-parameter-schema.ts
31+
- packages/ai/src/providers/agent-tools-parameter-schema.ts
3232
- src/agents/sessions/tools/**
3333
- src/agents/embedded-agent-runner/effective-tool-policy.ts
3434
- src/agents/embedded-agent-runner/tool-name-allowlist.ts

‎.github/workflows/openclaw-npm-release.yml‎

Lines changed: 58 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,23 @@ jobs:
170170
- name: Build
171171
run: pnpm build
172172

173+
- name: Pack AI runtime package
174+
id: ai_runtime_tarballs
175+
run: |
176+
set -euo pipefail
177+
PACKAGE_VERSION="$(node -p "require('./package.json').version")"
178+
package_version="$(node -p "require('./packages/ai/package.json').version")"
179+
if [[ "$package_version" != "$PACKAGE_VERSION" ]]; then
180+
echo "packages/ai version ${package_version} does not match openclaw ${PACKAGE_VERSION}." >&2
181+
exit 1
182+
fi
183+
ARTIFACT_DIR="$RUNNER_TEMP/openclaw-ai-runtime-packages"
184+
rm -rf "$ARTIFACT_DIR"
185+
mkdir -p "$ARTIFACT_DIR"
186+
pnpm --dir packages/ai pack --pack-destination "$ARTIFACT_DIR"
187+
(cd "$ARTIFACT_DIR" && sha256sum ./*.tgz > SHA256SUMS)
188+
echo "dir=$ARTIFACT_DIR" >> "$GITHUB_OUTPUT"
189+
173190
- name: Build Control UI
174191
run: pnpm ui:build
175192

@@ -233,30 +250,35 @@ jobs:
233250
RELEASE_REF: ${{ inputs.tag }}
234251
RELEASE_NPM_DIST_TAG: ${{ inputs.npm_dist_tag }}
235252
DEPENDENCY_EVIDENCE_DIR: ${{ steps.dependency_evidence.outputs.dir }}
253+
AI_RUNTIME_TARBALL_DIR: ${{ steps.ai_runtime_tarballs.outputs.dir }}
236254
run: |
237255
set -euo pipefail
238256
PACK_OUTPUT="$RUNNER_TEMP/npm-pack-output.txt"
239-
npm pack --json 2>&1 | tee "$PACK_OUTPUT"
257+
pnpm pack --json 2>&1 | tee "$PACK_OUTPUT"
240258
PACK_NAME="$(node - "$PACK_OUTPUT" <<'NODE'
241259
const fs = require("node:fs");
242260
const path = require("node:path");
243261
const input = fs.readFileSync(process.argv[2], "utf8");
244262
245263
function resolveTarballFileName(value) {
246264
const fileName = typeof value === "string" ? value.trim() : "";
265+
const resolved = path.resolve(fileName);
247266
if (
248267
!fileName.endsWith(".tgz") ||
249268
fileName.includes("\0") ||
250269
fileName !== path.basename(fileName) ||
251-
fileName !== path.win32.basename(fileName)
270+
fileName !== path.win32.basename(fileName) ||
271+
path.dirname(resolved) !== process.cwd()
252272
) {
253273
console.error(`npm pack reported unsafe tarball filename ${JSON.stringify(fileName)}.`);
254274
process.exit(1);
255275
}
256-
return fileName;
276+
return path.basename(resolved);
257277
}
258278
259-
function arrayEndFrom(start) {
279+
function jsonEndFrom(start) {
280+
const opening = input[start];
281+
const closing = opening === "[" ? "]" : "}";
260282
let depth = 0;
261283
let inString = false;
262284
let escape = false;
@@ -274,9 +296,9 @@ jobs:
274296
}
275297
if (char === "\"") {
276298
inString = true;
277-
} else if (char === "[") {
299+
} else if (char === opening) {
278300
depth += 1;
279-
} else if (char === "]") {
301+
} else if (char === closing) {
280302
depth -= 1;
281303
if (depth === 0) {
282304
return i + 1;
@@ -286,15 +308,15 @@ jobs:
286308
return -1;
287309
}
288310
289-
for (const match of input.matchAll(/\[/g)) {
311+
for (const match of input.matchAll(/[\[{]/g)) {
290312
const start = match.index;
291-
const end = arrayEndFrom(start);
313+
const end = jsonEndFrom(start);
292314
if (end === -1) {
293315
continue;
294316
}
295317
try {
296318
const parsed = JSON.parse(input.slice(start, end));
297-
const first = Array.isArray(parsed) ? parsed[0] : null;
319+
const first = Array.isArray(parsed) ? parsed[0] : parsed;
298320
if (first && Object.prototype.hasOwnProperty.call(first, "filename")) {
299321
process.stdout.write(resolveTarballFileName(first.filename));
300322
process.exit(0);
@@ -326,6 +348,8 @@ jobs:
326348
rm -rf "$ARTIFACT_DIR"
327349
mkdir -p "$ARTIFACT_DIR"
328350
cp "$PACK_PATH" "$ARTIFACT_DIR/"
351+
cp "$AI_RUNTIME_TARBALL_DIR"/*.tgz "$ARTIFACT_DIR/"
352+
cp "$AI_RUNTIME_TARBALL_DIR/SHA256SUMS" "$ARTIFACT_DIR/ai-runtime-SHA256SUMS"
329353
cp -R "$DEPENDENCY_EVIDENCE_DIR" "$ARTIFACT_DIR/dependency-evidence"
330354
printf '%s\n' "$RELEASE_TAG" > "$ARTIFACT_DIR/release-tag.txt"
331355
printf '%s\n' "$RELEASE_SHA" > "$ARTIFACT_DIR/release-sha.txt"
@@ -358,14 +382,16 @@ jobs:
358382
PREFLIGHT_ARTIFACT_DIR: ${{ steps.packed_tarball.outputs.dir }}
359383
run: |
360384
set -euo pipefail
361-
TARBALL_PATH="$(find "$PREFLIGHT_ARTIFACT_DIR" -maxdepth 1 -type f -name '*.tgz' -print | sort | tail -n 1)"
385+
TARBALL_PATH="$(find "$PREFLIGHT_ARTIFACT_DIR" -maxdepth 1 -type f -name 'openclaw-[0-9]*.tgz' -print | sort | tail -n 1)"
362386
if [[ -z "$TARBALL_PATH" ]]; then
363387
echo "Prepared preflight tarball not found." >&2
364388
ls -la "$PREFLIGHT_ARTIFACT_DIR" >&2 || true
365389
exit 1
366390
fi
367391
PACKAGE_VERSION="$(node -p "require('./package.json').version")"
368-
node --import tsx scripts/openclaw-npm-prepublish-verify.ts "$TARBALL_PATH" "$PACKAGE_VERSION"
392+
AI_TARBALL="$(find "$PREFLIGHT_ARTIFACT_DIR" -maxdepth 1 -type f -name 'openclaw-ai-*.tgz' -print | head -n 1)"
393+
node --import tsx scripts/openclaw-npm-prepublish-verify.ts \
394+
"$TARBALL_PATH" "$PACKAGE_VERSION" "$AI_TARBALL"
369395
370396
- name: Upload dependency release evidence
371397
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
@@ -716,6 +742,11 @@ jobs:
716742
echo "Prepared preflight tarball digest mismatch." >&2
717743
exit 1
718744
fi
745+
if [[ ! -f preflight-tarball/ai-runtime-SHA256SUMS ]]; then
746+
echo "Prepared AI runtime dependency checksums are missing." >&2
747+
exit 1
748+
fi
749+
(cd preflight-tarball && sha256sum -c ai-runtime-SHA256SUMS)
719750
echo "tarball_name=$ARTIFACT_TARBALL_NAME" >> "$GITHUB_OUTPUT"
720751
echo "tarball_sha256=$ARTIFACT_TARBALL_SHA256" >> "$GITHUB_OUTPUT"
721752
echo "tarball_path=$ARTIFACT_TARBALL_PATH" >> "$GITHUB_OUTPUT"
@@ -779,6 +810,22 @@ jobs:
779810
if [[ -n "${publish_target}" ]]; then
780811
publish_target="./${publish_target}"
781812
fi
813+
publish_if_missing() {
814+
local package_name="$1"
815+
local tarball_path="$2"
816+
local package_version
817+
if [[ -z "$tarball_path" || ! -f "$tarball_path" ]]; then
818+
echo "Prepared tarball for ${package_name} was not found." >&2
819+
exit 1
820+
fi
821+
package_version="$(node -p "require('./package.json').version")"
822+
if npm view "${package_name}@${package_version}" version >/dev/null 2>&1; then
823+
echo "${package_name}@${package_version} is already published; reusing it."
824+
return 0
825+
fi
826+
bash scripts/openclaw-npm-publish.sh --publish "./${tarball_path}"
827+
}
828+
publish_if_missing "@openclaw/ai" "$(find preflight-tarball -type f -name 'openclaw-ai-*.tgz' -print | head -n 1)"
782829
bash scripts/openclaw-npm-publish.sh --publish "${publish_target}"
783830
784831
- name: Verify extended-stable registry readback

‎config/knip.config.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,9 +138,13 @@ const config = {
138138
entry: rootEntries,
139139
ignoreDependencies: [
140140
"@openclaw/*",
141+
// Docker packaging stages @openclaw/ai without nested dependencies after
142+
// verifying the root owns its exact runtime dependency versions.
143+
"@mistralai/mistralai",
141144
"cross-spawn",
142145
"file-type",
143146
"playwright-core",
147+
"partial-json",
144148
"sqlite-vec",
145149
"tree-sitter-bash",
146150
...rootBundledPluginRuntimeDependencies,
@@ -164,6 +168,19 @@ const config = {
164168
ignoreDependencies: ["three"],
165169
project: ["src/**/*.{ts,tsx}!"],
166170
},
171+
"packages/ai": {
172+
// Mirror the published export map so knip sees every dist entry point.
173+
entry: [
174+
"src/index.ts!",
175+
"src/providers.ts!",
176+
"src/types.ts!",
177+
"src/validation.ts!",
178+
"src/utils/diagnostics.ts!",
179+
"src/utils/event-stream.ts!",
180+
"src/internal/*.ts!",
181+
],
182+
project: ["src/**/*.ts!"],
183+
},
167184
"packages/sdk": {
168185
entry: ["src/index.ts!"],
169186
project: ["src/**/*.ts!"],

0 commit comments

Comments
 (0)