This document tracks issues that cannot be closed by a code change in this repo alone. They require operator action, an external vendor, or production telemetry that does not yet exist. Each entry lists what it is, why it can't be closed inline, who needs to act, and what the acceptance criteria are.
Last reviewed: 2026-10-22 (outside review of SUB/1, the ledger, WARDEN wrap and KI-3/4/9/11; fixes in § Outside review fixes (2026-10-22)). Previous: 2026-08-27.
Resolution: The input_validity circuit now ships a PLONK backend
instead of Groth16. PLONK uses a universal structured reference string —
a single public Powers-of-Tau (phase 1) is sufficient, and there is no
per-circuit, multi-party phase-2 ceremony and no toxic waste to destroy.
This removes the entire reason KI-1 existed.
What was done (in-repo, no ceremony):
contracts/zk/scripts/setup_plonk.sh— compiles the circuit, reuses the public Hermez pot14 ptau, runssnarkjs plonk setup, exports artifacts. (Also fixes the dead ptau URL the old Groth16setup.shpointed at.)contracts/zk/verifier/verification_key.json— committed (protocolplonk, curvebn128, 3 public signals).contracts/zk/verifier/Verifier.sol— committed (PlonkVerifier,verifyProof(uint256[24], uint256[3])).aimarket_hub/zk_groth16.py— prover parametrised by proof system;AIMARKET_ZK_BACKEND=plonkselects the PLONK snarkjs subcommands.security/zk_artifacts.py::production_zk_issues()— acceptsplonk, checks for the PLONK artifacts (no ceremony hint).
Verified end-to-end: snarkjs plonk fullprove + snarkjs plonk verify
roundtrip on the committed artifacts returns OK! (witness built with
circomlibjs Poseidon matching the circuit).
Trust assumption (honest): PLONK is not transparent — it still relies on the public universal Powers-of-Tau ceremony (Hermez/Perpetual PoT), which thousands of projects reuse. That is a far weaker and more standard assumption than a bespoke single-operator setup; it is not "no trust at all" (that would need a STARK). Small demo payments run on Base mainnet; see the on-chain journal and the mainnet demo notice in the root README.
Optional Groth16 path (not required): smaller proofs / cheaper on-chain
verify, but it would need a per-circuit ceremony. Tooling for that still
exists (scripts/zk_ceremony_coordinator.sh, contracts/zk/scripts/setup.sh)
for operators who choose it at mainnet scale. PLONK is the default.
To enable the real backend: run setup_plonk.sh, then set
AIMARKET_ZK_BACKEND=plonk + AIMARKET_ZK_WASM/ZKEY/VKEY and drop
AIMARKET_ZK_SIMULATED. prod_startup_guard still blocks
AIFACTORY_PROD=1 + simulated ZK.
Where: contracts/evm/src/, contracts/solana/programs/aimarket-escrow/.
Why it can't be fixed in-repo: External audits by Trail of Bits, OpenZeppelin, Spearbit, ConsenSys Diligence, etc., are independent third-party services. The value comes from auditors having no prior context, which is the opposite of in-repo work.
Tooling already in repo (b4d64ebf):
contracts/audits/audit-response.md— disposition template (EXT-001 placeholder, status taxonomy: open / fixed / accepted / false-positive)scripts/run_contract_audit.sh— local Slither summary generator (slither-summary-*.md).github/workflows/contracts-ci.yml— Slither--fail-highon bothcontracts/evm/andacex/contracts/evm/.github/workflows/security-scan.yml— Python Bandit (fail on High), pip-audit, npm audit on frontend + TS SDK.github/dependabot.yml— pip / npm×2 / actions / cargo updatesscripts/run_dependency_audit.sh— one-shot local CVE scan
Required action (operator / business):
- Budget: USD 30k-100k (Trail of Bits typically USD 80k+ for a 6-week engagement; OpenZeppelin USD 40k-60k for similar scope; Spearbit competitions in the USD 30k-50k range).
- Scope letter covering:
AIMarketEscrow.sol(EVM payment channels)AIMarketCapabilityNFT.sol(ERC-721 entitlements)aimarket-escrowSolana program (Anchor)- The Groth16 verifier circuit (
contracts/zk/circuits/input_validity.circom) and itsVerifier.solafter KI-1 completes.
- Run internal scans first (
slither --fail-high,mythril,cargo audit,anchor verify) to clear the obvious findings before the audit clock starts. - Address all High/Critical findings + the majority of Mediums before mainnet deploy.
Acceptance criteria:
- Audit report PDF stored under
contracts/audits/. - All H/C findings fixed or formally accepted (signed off in
contracts/audits/audit-response.md). - Audit conclusion ("no critical findings remaining") published.
Until done: do not deploy to mainnet with real funds at scale. Testnet (Base Sepolia / Solana devnet), small demo sums (~$2 on Base), and dry-runs are fine.
Pet-project waiver: external audit (KI-2) is optional for self-host demos.
Use Slither + docs/pet-project-trust.md + multisig (KI-4) instead.
Where: entrypoint.sh — backend supervisor with exponential
backoff + circuit breaker.
What was found on 2026-10-22 (outside review; see the table under "Outside review fixes"):
- Production does not show it today.
aicom-app-1on factory-vps runsUVICORN_WORKERS=1on SQLite; its container log has no supervisor restart since the container started (2026-10-21 13:11 UTC). - One hypothesis below is ruled out. uvicorn (0.52) starts
--workerswithmultiprocessingspawn, and the master never imports the app, so no SQLite connection can be inherited across a fork. - A self-inflicted restart path, now closed. The supervisor kills the backend after five
failed
/api/healthprobes from 127.0.0.1, and the app's own per-address limiter (100 requests a minute) counted and throttled that probe with everything else on the address: under load the backend answered the probe 403 and was killed for being busy. In a local 15-minute soak with 4 workers, 13,779 of 17,907 health probes got 403. The probe is now unmetered (web/backend/middleware/firewall_http.py), the supervisor reads a 403/429 as alive, and the same soak gives 0 throttled probes. In production the limiter also refused a frontend container's server-side product reads 66 times in the previous run's log — a separate, user-facing effect of one address per frontend container (setAIFACTORY_TRUSTED_PROXY_IPSand forward the visitor's address to fix it). - Every worker ran every background loop (backups, stand-ups, UNI jobs, disk alerts): N
backups in one minute writing the same files. One worker, the
flockholder, runs them now. - The crash telemetry never survived. It was appended to
uvicorn-last-crash.logand overwritten by the next start's log rotation; it has its own file now (uvicorn-crash-telemetry.log). scripts/load_test_factory.shhit/api/pipeline/list, which no longer exists (404), and counted the limiter's 403 as a failure; it now reads a public pipeline route and reports throttled answers apart from failures.
Still open: the one-hour soak against PostgreSQL with real pipeline traffic below, before
raising UVICORN_WORKERS in production.
Symptom: Under sustained pipeline load, uvicorn workers crash and
the supervisor restarts them. With UVICORN_WORKERS > 1, the
multiprocess supervisor has been observed to enter crash loops.
Why it can't be fixed in-repo: The supervisor with backoff is a mitigation, not a fix. The actual root cause requires:
- Production-grade load (many concurrent invocations against
/api/health,/api/pipeline/*,/api/sandbox/*). - Memory profiling under that load (
py-spy dump,memray). - Inspection of
prometheus_client.multiprocessshared-memory state (suspected file descriptor leak or write contention onPROMETHEUS_MULTIPROC_DIR). - Inspection of SQLite connections forked between workers (suspected:
workers share a
sqlite3.Connectionobject that was opened beforefork(), leading to write corruption and subsequent crashes).
None of those can be done from inside the repo — they need a running production environment with realistic traffic.
Mitigation in place:
- Default
UVICORN_WORKERS=1(single worker = no shared-state issue). USE_SQLITE=trueforcesUVICORN_WORKERS=1even if the operator overrides (2026-06).- Cap at
UVICORN_WORKERS=4even if operator overrides (was 2 before Wave 2; raised after supervisor improvements). - Exponential backoff 5s → 60s prevents busy-loop.
BACKEND_MAX_RESTARTS=20inBACKEND_RESTART_WINDOW_SECS=1800triggers fail-fast (exit 1) instead of infinite restart.- Crash diagnostics persisted to
/app/data/logs/uvicorn-last-crash.logwith structured telemetry (workers, SQLite flag, Prometheus multiproc file count, log tail).
Required action (dev/ops):
- Deploy staging with
./scripts/run_prod_compose.sh up -d --build. - Run
./scripts/load_test_factory.sh --base-url http://127.0.0.1:9081 --duration 3600 --concurrency 10. - Attach
py-spy record -p $UVICORN_PID -o flame.svg --duration 3600. - Capture
PROMETHEUS_MULTIPROC_DIRfilesystem snapshot before / after. lsof -p $UVICORN_PIDto inventory open file descriptors over time.- Open a GitHub issue with the trace; link any fix back to this entry.
Acceptance criteria:
- Root cause identified and documented in a code comment at
entrypoint.sh(replacing or augmenting the current "KNOWN ISSUE" block). - Workers can safely run at
UVICORN_WORKERS=4+without the supervisor having to restart them more than once per hour under steady load. BACKEND_MAX_RESTARTScap can be lowered (e.g. to 5) without spuriously tripping.
Until done: keep UVICORN_WORKERS=1 in production. The supervisor
absorbs crashes when they happen; users see brief HTTP 503 during the
backoff window, not infinite errors.
Where: contracts/evm/src/AIMarketEscrow.sol, contracts/evm/src/AIMarketCapabilityNFT.sol.
What's missing: Both contracts use Ownable2Step (good — accidental
transferOwnership is recoverable), but owner after deploy is the EOA
that ran forge create. A compromise of one private key reassigns hub
authorizations, token whitelists, and (on NFT) global pause.
Why it can't be fixed in-repo: A Safe / Squads multisig is created
on-chain by the operator, not declared in source. The transfer step
must happen between forge create and first real-money channel.
Required action (operator):
- Deploy a Gnosis Safe (EVM) with 2-of-3 or 3-of-5 signers held by different humans / different geographies.
- From the deployer EOA, run:
cast send $ESCROW_ADDR "transferOwnership(address)" $SAFE_ADDR \ --ledger --from $DEPLOYER --rpc-url $RPC_URL
- From the Safe UI (or
cast sendvia a Safe transaction builder), callacceptOwnership(). Repeat for the NFT contract. - Document the Safe address in
contracts/audits/audit-response.mdunder the pre-mainnet checklist.
Acceptance criteria:
cast call $ESCROW_ADDR "owner()(address)"returns the Safe address.cast call $ESCROW_ADDR "pendingOwner()(address)"returns0x0.- Safe transaction history shows the
acceptOwnershipcall.
Until done: Single-EOA owner is acceptable for testnet drills and mainnet pilots with < $1k TVL. Do not run without multisig past $10k TVL on a single channel or > $100k cumulative.
Measured, not assumed (2026-10-22): python3 scripts/check_ki4_ownership.py reads the
live deployment from Base and exits 1 when an EOA owns a contract while the escrow holds the
limit or more (KI4_TVL_LIMIT_USD, default 1000). Today: AIMarketEscrow (0xa4cb…B2Eb) and
AIMarketCapabilityNFT are owned by the EOA 0x1218…Ad0a, no pending owner, and the escrow
holds 0 USDC — inside the line above. The transfer itself stays the operator's
(scripts/multisig_transfer_runbook.sh); the check is what an alerter should run.
Where: requirements.txt and transitive pins.
Status: FastAPI/Starlette upgraded to 0.141.1/1.6.0; the audit gate
contains zero ignored vulnerability IDs. A fresh Python 3.12 pip-audit is
required to remain green in CI.
Removed dead dependencies (2026-06-28):
| Package | Action | Cleared |
|---|---|---|
langgraph |
Removed from requirements.txt |
PYSEC-2026-83 + transitive checkpoint/langsmith |
langchain-core |
Removed from requirements.txt |
5 GHSA advisories |
Bumped inline (cleared, non-breaking):
| Package | From → To | Cleared |
|---|---|---|
aiohttp |
3.10.5 → 3.14.0 | 25 advisories |
PyJWT[crypto] |
2.10.1 → 2.13.0 | 7 (PYSEC-2025-183, PYSEC-2026-120/175-179) |
cryptography |
45.0.7 → 46.0.7 | PYSEC-2026-35/36, GHSA-r6ph-v2qm-q3c2 |
orjson |
3.10.7 → 3.11.6 | PYSEC-2026-107, GHSA-hx9q-6w63-j58v |
python-dotenv |
1.0.1 → 1.2.2 | GHSA-mf9w-mj56-hr94 |
python-multipart |
→ ≥ 0.0.18 | (earlier) |
Smoke-verified: jwt/crypto suites (28 passed) and LLM router/provider suites
(13 passed) green on the bumped versions; remaining test failures are the
pre-existing /app read-only sandbox path issue, unrelated to these bumps.
Acceptance criteria:
bash scripts/pip_audit_gate.shexits 0 with no--ignore-vulnarguments.- Production runtime smoke (
/api/health,/api/pipeline/list) green for ≥ 1 hour post-deploy.
The full unfiltered report remains the pip-audit-report CI artifact every run.
Frontend note (npm): Next 16 / React 19 migration — DONE (2026-06-06).
web/frontend now ships next 16.2.7, react/react-dom 19.2.7,
eslint 9, eslint-config-next 16, recharts 2.15.4. This cleared the
previous next (high) + postcss advisories, so the npm gate was raised from
critical to high (npm audit --omit=dev --audit-level=high, green; only 2
moderate-in-next remain, below the gate). The ws override on ^8.21.0
still clears the ethers → ws advisories without downgrading ethers.
What the migration required (vs. the originally-feared async-params rewrite, which turned out not to apply — those 9 App-Router files broke neither typecheck nor build):
- one-line CSS fix: Google-Fonts
@importmoved to the top ofstyles/globals.css(Turbopack, now the default builder, enforces CSS spec ordering; webpack tolerated it); next lintremoved → flateslint.config.mjsspreadingeslint-config-next/core-web-vitalsdirectly (FlatCompat hits a circular-JSON bug);next.config.js:images.domains→images.remotePatterns.
Verified green: tsc --noEmit (0 errors), eslint . (0 errors), next build
(32/32 pages), vitest (7 passed) under React 19.
Remaining tech-debt (advisory, tracked here): eslint-config-next 16
promoted a wave of React-Compiler-era hook rules to error on pre-existing code
— react-hooks/set-state-in-effect (41), react-hooks/refs (19),
react-hooks/immutability (3), plus @next/next/no-html-link-for-pages (8).
These are down-leveled to warn in eslint.config.mjs (parity with the
pre-existing exhaustive-deps warnings) rather than mass-refactored blind;
they should be cleared incrementally with a visual/e2e pass. React-19
runtime e2e for recharts + framer-motion is covered by
web/frontend/e2e/charts-motion.spec.ts (Playwright, CI frontend-build job).
Internal review of b4d64ebf / 90652fd8 surfaced issues fixed in
follow-up commits (not operator blockers):
| ID | Severity | Issue | Fix |
|---|---|---|---|
| NEW-1 | CRITICAL | ZK ceremony entropy in argv (ps leak) |
zk_ceremony_coordinator.sh: stdin pipe, no -e |
| NEW-2 | HIGH | OIDC nonce optional | Strict nonce required in verify_id_token |
| NEW-3 | HIGH | OIDC role stale after group change | _sync_oidc_user updates role every login |
| NEW-4 | MEDIUM | Vault token over HTTP | Refuse remote HTTP; warn on loopback |
| NEW-5 | MEDIUM | /zk/prove-input DoS |
Per-IP rate limit (AIMARKET_ZK_PROVE_RATE_LIMIT) |
| NEW-6 | MEDIUM | Truncated proof fields | Return full commitments/nullifier |
| NEW-7 | MEDIUM | zkey overwrite | Backup before install-secrets |
| NEW-8 | MEDIUM | OIDC post-login open redirect | safe_post_login_url() validation |
Still operator/vendor (unchanged): KI-1 … KI-11 above.
An outside reviewer read the public mirrors (hub, WARDEN, this file) and listed suspected defects and the tests that would settle them. Each was reproduced or ruled out; the fixes carry tests that fail on the code before them where a test could show it.
| ID | Severity | Finding | Status |
|---|---|---|---|
| LEDGER-1 | HIGH | Credits ledger: resolving a hold committed its status before moving the money, and carving a child out of an allowance committed the decrement before the child row. A process killed in between, or another request's failed statement rolling back the shared SQLite connection, lost the money (held but never released). | Fixed. Every money move is one transaction under the connection lock (db_backend.atomic). aimarket-hub/tests/test_ledger_crash_consistency.py: kills a real process at the last statement before each commit and interleaves a foreign rollback — 8 of 11 cases failed before. |
| SUB-1 | MEDIUM | JobStore.join() checked "parent running" and inserted the child in separate transactions: a child could join a call that had just finished (tree and receipt disagree). |
Fixed. Check and insert in one transaction with a write on the parent row (row lock on PostgreSQL). Deterministic test fails before (test_subcontract_races.py). |
| SUB-2 | HIGH | A child hold left held by a crash was released by no sweep (the invoke sweep skips children, the allowance sweep releases only the allowance): the buyer's money stayed frozen. |
Fixed. invoke_funding.sweep_orphaned_child_holds (with the allowance sweep, every 120 s) releases child holds older than an hour and gives the root mandate its budget back. |
| SUB-3 | MEDIUM | The grant (a bearer credential for the buyer's allowance) was sent to a provider at a plain-http public URL. |
Fixed. Sent only over TLS or to the operator's own hosts; otherwise the provider gets the job token alone. Checked and fine: the grant is stored hashed, never logged, never in a refusal, refused on /a2a, dead once the job closes. |
| CRED-1 | MEDIUM | Signup grant: budget checked and grant minted as two steps. On PostgreSQL 20 concurrent signups minted $0.30–$0.45 against a $0.25 daily budget (3 runs); on SQLite, 4 and 16 processes did not overshoot. | Fixed. CreditsLedger.open_signup_account: one transaction, serialized by a counter row; exact on both backends (thread and multi-process tests). Per-address and per-subnet signup limits were already in place. |
| MAND-1 | LOW | A mandate document unreadable mid-chain (deleted between two reads) was dropped, which shifted the revocation check of later links. | Fixed. Fails closed. Found by the new ruff gate. |
| WARDEN-1 | HIGH | wrap ignored sensitiveToolPatterns. |
Fixed (see KI-9). |
| PERF-1 | HIGH (CPU) | The pipeline worker re-parsed the YAML config for every product on every cycle: 92% of its CPU in a py-spy profile of the live process. | Fixed and deployed 2026-10-23 (ai-factory:perf-20261023): parse reused while the files do not change; ~89% of a core → 16%. |
| PERF-2 | MEDIUM | Storefront grid 3.4-7.3 s (three demo assessments per product, each walking node_modules); trust strip 4-5.7 s (all 3,510 tasks loaded to count product states), blocking the event loop and with it /api/health. |
Fixed and deployed 2026-10-23: grid median 7.3 s → 0.22 s, trust strip 4.7 s → 0.18 s, health p95 3.8 s → 0.43 s at 20 users; responses byte-identical before and after (docs/load-testing.md). |
| PERF-3 | MEDIUM | Monitor ran 17 blocking satellite polls on its event loop: /api/health p95 1.8-4.3 s, one state read 62 s. |
Fixed (bd4276c28), deployed 2026-10-23 09:21 UTC on monitor.modelmarket.dev: /api/health p95 1.8 → 1.1 s, slowest 4.3 → 1.4 s (20 clients, 90 s). UNI monitor still on the previous image. |
| LOG-1 | LOW | 311 Agent Capital Exchange tracebacks a day on the apex hub, from ledgers that are off by configuration. | Fixed and deployed 2026-10-23 (modelmarket-hub:acex-20261023): one warning per ledger, same 503 answers. |
| SIGNER-1 | MEDIUM | The escrow signer's forward on the Docker bridge was open to every container on aicom_aicom_net (fifteen, product frontends among them). |
Fixed 2026-10-23 on factory-vps: only the hub container's address passes (escrow-signer/deploy/signer-hub-only.*). |
| ARGUS-1 | LOW | ARGUS checked sensitivity by the prefixed name (fs__delete_file), its MCP host by the server's (delete_file): an anchored pattern blocked the tool instead of asking. |
Fixed (needsApproval). |
| KI3-1 | HIGH (multi-worker) | With UVICORN_WORKERS > 1 every worker ran every background loop (backups, stand-ups, UNI jobs, disk alerts). |
Fixed. One worker, the flock holder, runs them; another takes over when it dies (tests/test_background_leader.py). |
| KI3-2 | MEDIUM | KI-3's crash telemetry was appended to uvicorn-last-crash.log and overwritten by the next start's log rotation: none ever survived. |
Fixed. Own bounded file, uvicorn-crash-telemetry.log. |
| KI3-3 | HIGH | The app's per-address limiter counted and throttled the supervisor's /api/health probe: a busy backend answered it 403 and was killed (13,779 of 17,907 probes in a 4-worker soak). |
Fixed. Probe unmetered (deny list and enforced ACL still apply); supervisor reads 403/429 as alive; the same soak gives 0 throttled probes. |
| IDEMP-1 | MEDIUM | A paid invoke retried after a timeout on the credits rail or a ledger channel was a second purchase. | Fixed. Idempotency-Key (aimarket_hub/idempotency.py, migration 061): replay, 422 on another body, 409 while running or when the outcome is unknown; never re-run. |
| LOAD-1 | — | No load test touched the money path (the Locust mix is read-only by design). | Added. aimarket-hub/scripts/load_money_path.py: paid invokes, keyed retries and racing signups on a local hub, then a ledger audit. SQLite and PostgreSQL, 2,300 requests at 64 concurrent: every invariant held, each keyed group charged once, grants stopped at the budget. |
| CI-1 | MEDIUM | The hub's CI ran tests only. | Fixed. Ruff bug rules over the money/auth paths, and a PostgreSQL job for the money races (aimarket-hub/.github/workflows/ci.yml). |
Deployed 2026-10-23 (owner's go-ahead): apex hub modelmarket.dev on modelmarket-hub:review-20261022 (12 files of main ba1b10b2d over the running image; migration 061 applied; smoke on a copy of the live DB first: every row kept, held amounts match held holds on all 208 accounts, rollback image starts on the migrated copy); factory aicom-app-1 on ai-factory:review-20261022. Rollback notes: /root/hub-review-20261022/ROLLBACK.txt and /root/factory-deploy/review-20261022/ROLLBACK.txt on factory-vps. The other five hubs (UNI, lab, hunt, attested, Independent) still run the 2026-10-21/22 package.
Not changed, on purpose: the custodial channel rail (KI-11 — documented, with
AIMARKET_ESCROW_REQUIRED as the switch to the contract rail); factory products not listed as
hub capabilities (a positioning decision, docs/ecosystem-architecture.md §3); an on-chain
payment that is not refunded when delivery then fails (SUB/1's allowance is refunded; an x402
transfer cannot be recalled by the hub).
Where: oracles/ satellite — oracle-core signing, Chronos VDF, Sortes ECVRF, sixteen
other domain oracles, lottery on-chain consumers.
What reviewers correctly note (2026-07):
- Chronos (Wesolowski VDF) — parameters and modulus choice are documented in source
(
oracles/chronos/SECURITY.md), but there is no external audit, no formal verification, and no published parameter-selection guide (iteration countT↔ wall-clock, hardware profiles, security margins). Links to third-party review do not exist yet. - Signing & oracle-core — Ed25519 + optional hybrid ML-DSA-65 is implemented
(
core/oracle_core/signing.py), but the hybrid extension is implementation-defined (not frozen inaimarket-protocol), off by default, Hub verifies Ed25519 only, and there is little proof-of-correctness beyond unit tests — no independent crypto review. - Overall tier — seventeen oracles with real math in ~two months of focused work is ambitious research/prototype quality (demos, testnet lottery, Hub integration), not a fully hardened cryptographic service suitable as the sole trust anchor for mainnet-scale TVL.
Why it can't be fixed in-repo alone: External cryptographic audits, formal methods, and published operator attestation programs require third-party engagement and calendar time — the same class of blocker as KI-2 (smart-contract audit).
In-repo honesty (done):
oracles/docs/crypto-maturity.en.md(+ RU) — ecosystem-facing maturity statement.oracles/core/docs/SIGNING.md— what hybrid signing does today (not a normative spec).- Banner in
oracles/README.md.
Required action (operator / business):
- Commission external crypto audit (scope:
oracle-coresigning, Chronos VDF, Sortes ECVRF minimum; budget similar order of magnitude to KI-2, narrower scope). Still open — the owner decided (2026-10-12) to self-verify first; nothing below replaces this item. Publish Chronos parameter guide + operator attestation checklist— done 2026-10-12:oracles/chronos/docs/PARAMETERS.md(+ RU), with measured costs (10.5 µs per squaring in Python on the production host, 0.88 µs with GMP on a laptop — sodifficultyis not a wall-clock promise).Freeze hybrid PQC fields in— done 2026-10-12: spec §7.3.8 (+ §7.3.7, the strict Ed25519 rule),aimarket-protocolwith negative test vectorsreceipt-hybrid-signed.jsonand four negative vectors, checked byconformance/run.py. The Hub already verifies both layers when present (Signer.verify_hybrid).Oracle key-management runbook— done 2026-10-12:oracles/docs/key-management.en.md(+ RU).
Self-verification 2026-10-12 (what the standard vectors found, all fixed and deployed):
- Sortes ECVRF skipped RFC 9381
validate_key: for the identity public key a proof could be forged for any input, always with the same output, andsortes.verifytakes the key from the caller. Fixed (small-order keys refused; responses say whether the key is the oracle's own). - Ed25519 verification across the ecosystem accepted the identity-key forgery (
R= base point,S= 1, valid for every message) because OpenSSL, tweetnacl and WebCrypto do. Every verifier now refuses small-order keys andR; RFC 8032 vectors pin the signers. - Chronos:
yandN − yboth verified for the same input (−1 has order 2), one bit of prover choice in a beacon. Outputs are now reported canonically (y_canonical,output).
Acceptance criteria:
- Audit report PDF linked from
oracles/docs/crypto-maturity.en.md. - Chronos parameter guide committed under
oracles/chronos/docs/. - Protocol PR merged in
aimarket-protocolfor PQ extension + Hub parity. - KI-6 entry deleted; one-line
CHANGELOG.mdnote.
Until done: Oracles are fine for testnet, demos, bounded pilots, and education
(including courses that map to lottery/ code). Do not market the family as
production-hardened crypto or rely on it alone for large real-money flows — same pre-mainnet
posture as KI-2.
Where: Root factory — pipeline_worker.py, config/fragments/, web/backend/services/,
entrypoint.sh, shipped demos under docs/sample-output/.
What reviewers correctly note (2026-07):
- Volume vs time — full multi-agent factory with quality gates in ~two months implies technical debt (supervisor KI-3, conditional agent graph, director re-queue paths).
- Over-engineered default — twelve config fragments and many stages are heavy for simple landing/MVP outputs; fair critique for pet-project scope.
- Shipped products — public build replays and live demos skew to MVP storefronts, not complex long-lived software — honest tier labeling required.
- Production readiness — load testing, multisig, external audit acknowledged open (KI-2–4).
In-repo actions (done / ongoing):
docs/factory-pipeline-profiles.md— minimal vs full profiles.docs/ecosystem-maturity-review.en.md— scorecard + matrix.docs/sample-output/README.md— MVP-tier label.
Required action (operator):
- Run KI-3 soak:
./scripts/load_test_factory.sh --duration 3600 --concurrency 10 - Prefer minimal pipeline profile for landing-only builds.
- Complete KI-4 multisig before >$10k TVL.
Acceptance criteria:
- KI-3 closed (workers stable at
UVICORN_WORKERS=4+under load). - Documented default profile choice in operator guide.
- At least one non-landing sample replay OR honest “MVP only” banner on all public demos.
Until done: Present Factory as self-host research/prototype — powerful demo pipeline, not proven SaaS factory at scale.
Where: metis/ — distributed coordinator, confidence gate, verifier-with-retry, economy
metering hooks.
What reviewers correctly note:
- Distributed cluster — TLS+HMAC multi-node path exists but ** lacks production soak** (partition, stale registry, clock skew, partial node compromise).
- Confidence gate — fail-closed on structured
TaskSpec, but trusts council-assigned confidence; high-score subtle hallucinations can proceed if ambiguities omitneeds_user_input. - Verifier-with-retry — strong concept; limited adversarial/red-team benchmark coverage vs encoding tricks, contradictory evidence, or confident wrong answers.
- Economy metering — interesting; enforcement is consumer-side (Factory/Hub debit), not a hard kernel-level cap inside Metis alone.
In-repo actions:
metis/docs/en/MATURITY.mdmetis/tests/test_adversarial_gates.py— regression seeds for known gate bypass patterns- Benchmark docs already state confidence ≠ accuracy ceiling
Required action (dev/ops):
- Multi-node soak test script (48h, 3+ workers, inject node failure).
- Expand
metis/benchmarks/with trap/adversarial suite; publish scores indocs/benchmarks/. - Wire Factory architect stage to hard-fail on
verified: falsewhen configured.
Acceptance criteria:
- Soak report committed under
metis/docs/benchmarks/. - Adversarial benchmark ≥ N cases with documented pass/fail thresholds.
- Distributed mode labeled beta until soak green.
Until done: Use Metis for cognition + confidence signal on bounded tasks; do not treat distributed Metis as HA production inference mesh without soak evidence.
Where: warden/src/, policy defaults in argus.config.example.json.
What reviewers correctly note:
- WARDEN is real — static scan, threat feed, pinning, egress guard; unit tests on textbook poisoning.
- Two months is short for a firewall against sophisticated attacks: obfuscated injections, multi-hop tool chains, runtime-only exfil, model-side compliance with smuggled instructions after vetting, compromised benign servers post-pin approval.
- Permissive defaults —
allowUnknownServersand LUMEN degrade-to-neutral preserve autonomy but weaken fail-closed posture when oracle unreachable.
In-repo actions:
- §Limitations in
argus/docs/security-warden.md argus/test/adversarial-warden.test.ts— documents known bypass class (obfuscated injection)
Required action (dev/security):
- Red-team fixture corpus (≥20 cases) in
argus/test/fixtures/warden-adversarial/. - Tighten default policy doc for high-security profile (
allowUnknownServers: false). - Optional Immunefi / private bug bounty before claiming “production firewall”.
Acceptance criteria:
- Red-team report or CI job with adversarial fixtures; regressions tracked.
- High-security preset documented and tested.
Where it stands (2026-10-22, WARDEN 0.20.0): a red-team corpus exists and is re-run per
release (warden/scripts/redteam/: 23 code-aware attacks against the flow guard, 2 still
delivered and named in warden/docs/flow-guard.md; MCPTox held-out half 210 of 218 blocked with 0
of 45 clean servers; AgentDojo replay 0 of 579 attack actions). The high-security preset is
wrap --require-approval with {"allowUnknownServers": false}. An outside review found that
warden-mcp wrap accepted policy.sensitiveToolPatterns and passed matching calls straight to
the server; since 0.20.0 each such call waits for a person through the client's MCP elicitation
form, or is refused when the client has none (warden/test/wrap.test.ts, 7 tests).
Until done: WARDEN is strong default for casual MCP — not a guarantee against targeted attack; combine with egress allowlists, human approval on sensitive tools, encrypted keystore.
Where: aimarket-hub/ — federation crawler, channels, invoke, slash sync, plugin registry.
What reviewers correctly note:
- Protocol v2 — solid normative base + reference hub.
- Federation + micropayments — implemented but early; almost no third-party hub mesh or external invoke volume → real-world edge cases (stale manifests, channel races, slash disagreements, plugin ordering) mostly untested in the wild.
- Adoption ≈ 0 outside operator deployments — fair critique.
In-repo actions:
Required action (ecosystem):
- Publish federation testnet playbook (2+ hub peers, cross-crawl, slash pull).
- Chaos tests: invoke during channel close, manifest rotation mid-flight.
- First external peer hub (even read-only mirror) to validate crawl/slash sync.
Acceptance criteria:
- Documented federation drill with logs committed.
- Negative test vectors in
aimarket-protocolfor channel edge cases. - At least one non-operator peer listed in ecosystem map.
Until done: Hub is reference implementation + demo deployment — not proven multi-operator marketplace at scale.
Catalogue invoke is no longer this issue. A Hub catalog sale pays USDC
on-chain to the listing's payout_address; the hub verifies
PAYMENT-SIGNATURE / the transfer and does not custody that money
(aimarket_hub/settle.py, same invariant as HESTIA). An operator share is
available on that rail through MarketSplitter and is still not custody — the
contract forwards both legs inside the buyer's transaction and the hub verifies
the legs rather than the route. Rails, and what each asks you to trust, are
mapped in aimarket-hub/docs/money-rails.md.
KI-11 below is the channel / deposit rail only.
Where: aimarket-hub/aimarket_hub/channels.py,
aimarket-hub/aimarket_hub/escrow_bridge/,
contracts/evm/src/AIMarketEscrow.sol.
What is actually true today (re-verified 2026-07-29 against the code and the live
modelmarket.dev container — the 2026-07-25 wording said "no runtime code path calls
debitChannel", which is no longer the whole story in either direction):
| Step | Where it happens |
|---|---|
| Deposit | On-chain, but by default as a plain transfer to the platform settlement wallet (on_chain.platform_recipient) — not into AIMarketEscrow. Verified for recipient, amount, token, confirmations and sender before credit. A client may instead fund the contract and pass escrow_channel_id; that branch exists, but the custodial one is still the default and cannot be switched off. |
| Payer binding | On-chain sender plus an EIP-191 proof-of-control signature, required by default in every mode (AIMARKET_CHANNEL_ALLOW_UNPROVEN_PAYER=1 opts out; non-EVM payers are refused). |
| Debit authorization | Live runtime path. On a paid invoke against an escrow-bound channel the hub captures and verifies the depositor's EIP-712 DebitAuthorization before the provider runs, and returns 402 on failure — fail-closed. The buyer's receiptId is reused as the ledger receipt so the on-chain and off-chain replay keys cannot drift. |
Debit submission (debitChannel) |
Human-initiated only. The one submission site (escrow_bridge/mirror.py) is reachable from python -m aimarket_hub.escrow_bridge.cli submit --yes and from nothing else: GET /ai-market/v2/escrow/plan is hardcoded to PlanOnlySigner, no route broadcasts, and there is no scheduler or timer. |
| Prod's actual capability | Can broadcast, through a separate policy signer — re-verified 2026-09-11 and no longer what this row claimed. config.describe() on the live container now returns strategy: "external", may_broadcast: true, private_key_set: false, signer_url_set: true. The 2026-07-29 wording ("cannot broadcast at all", strategy: "plan") is stale and must not be cited as a mitigation. What still holds is the part that matters: no hot wallet is keyed into the hub (private_key_set: false). The authorized key lives in escrow-signer/ on a different machine, reached over AIMARKET_ESCROW_SIGNER_URL=http://172.18.0.1:9500/sign — a systemd-managed SSH local forward (escrow-signer-tunnel.service, ExitOnForwardFailure=yes, dedicated IdentitiesOnly key) to the signer host. So the gate moved from "the hub is incapable" to "a separate process with its own policy decides": it re-verifies the buyer's EIP-712 against state read from chain, pins one escrow and one chain, and enforces its own spend windows. Residual, worth knowing: the forward binds the Docker bridge gateway (-L 172.18.0.1:9500), so every container on aicom_aicom_net can reach /sign, not just the hub. Verified 2026-09-11 — aicom-app-1 and aicom-grafana-1 both get a response from the endpoint. That was three neighbours on 2026-09-11 (aicom-app-1, aicom-prometheus-1, aicom-grafana-1); on 2026-10-22 it is fifteen, among them product frontends the factory generated (aicom-frontend-*-20261011). AIMARKET_ESCROW_SIGNER_TOKEN plus the signer's own policy are what stand between them and a signature. escrow-signer/deploy/signer-hub-only.sh (+ .service/.timer) narrows the port to the hub container's current address and rejects everyone when the hub is not running. Installed on factory-vps 2026-10-23 (signer-hub-only.timer, every minute): the hub gets 200 from /health, aicom-grafana-1 and aicom-app-1 are refused, the host's own 127.0.0.1:9500 is untouched; no other container carries signer settings. The first install failed safe (a tcp-reset REJECT needs -p tcp in the same rule; nothing reached INPUT), fixed in the script. Undo steps are in the script's header. |
channel/close, remainder |
Off-chain only. The unspent remainder is recorded as a payout obligation (channel_payout_obligations, migration 017); refund_executed_usd is still 0.0 on the ledger rail. |
The three mainnet settlements in the journal (§3h, §3i, §3j) were made by hand, not by
the hub. Proof, not inference: the production bridge store held exactly three rows, all
pending with an empty tx_hash, and two of their receipt ids
(0x937253cb…, 0x409356b3…) return usedReceipts() == true on
0x0606983c…72C25D. The chain had collected those debits while the hub still reported
$0.16 owed on them. The hub did not know it had been paid, because it was not the payer.
Bug found and fixed while verifying this (2026-07-29). The store's forward-only state
machine had no pending → confirmed edge, so a debit collected out of band could never be
resolved — and because submission is strictly nonce-ordered per channel, each stuck row
blocked every later row on its channel permanently. Head-of-line blocking on money.
Fixed by guard 0 in the mirror: read usedReceipts(receiptId) first, before the queue and
deadline guards, and resolve an already-collected receipt as confirmed (a read failure
blocks rather than assuming "unpaid"). Verified against Base: the two collected receipts
resolve, the genuinely-unpaid third stays queued.
Consequences that must not be papered over:
- Custodial by default. The default deposit lands in the operator's wallet, so channel
balances are an operator liability, not escrowed funds.
AIMarketEscrow's non-custodial guarantees describe the contract, which the default rail does not use. - The two rails are mutually exclusive. If a contract channel and a ledger channel back
the same money, on-chain
usedAmountstays0however much the ledger consumed, sorefundChannel/expireChannelwould return a fully-consumed deposit in full. - Collection depends on a person. A verified, collectable claim is captured on every paid invoke, but nothing collects it unless an operator runs the CLI. An unattended hub accrues claims it never presents.
Both remaining dev items landed 2026-09-19. What is left is an operator decision, not code:
AIMARKET_ESCROW_REQUIRED=1makes the custodial deposit branch unreachable —/channel/openthen accepts only anescrow_channel_idthe contract confirms. It is deliberately NOT gated on the bridge also being enabled: requiring escrow with the bridge off would have left the custodial path open, which is the operator asking for the strongest setting and silently getting the weakest. That misconfiguration now refuses BOTH doors andpayment_readiness()reportsescrow_required_but_bridge_disabledat boot rather than letting it be discovered one refused open at a time.AIMARKET_ESCROW_COLLECT_INTERVAL_Sstarts a bounded pass in the hub's own lifespan (escrow_bridge/collector.py), so an authorized debit no longer waits for a person to run the CLI. Off by default; bounded by the bridge's existing submit policy and byMAX_USD_PER_PASS/PER_DAY, so calling more often cannot spend more. Each cycle runsconfirm()beforerun()— a debit collected out of band leaves a row that blocks every later row on its channel, and resolving those first is what keeps the queue moving.
Consequence 1 is therefore closable and consequence 3 is closed. What remains is an operator hot wallet with ETH for gas, and the decision to turn both switches on.
Required action (dev + operator) — what was left:
Land an escrow bridge that signs/submitsThe bridge, its guards, its store and its CLI exist. What remains: (a) an unattended trigger — a bounded background pass in the hub's lifespan, or a host timer invokingdebitChannelper settled receipt.cli submit --yes; (b) an operator hot wallet keyed into prod (SUBMIT_STRATEGY,SUBMIT_CONFIRM, and a key or signer URL — all absent today;debitChannelneeds ETH for gas, not USDC, and must NOT reuse the0x1218funds key).(c) a value bound.Done 2026-07-29 —AIMARKET_ESCROW_MAX_USD_PER_PASS($5 default) andAIMARKET_ESCROW_MAX_USD_PER_DAY($25 default, rolling, read from the store so it survives a restart), both reported byconfig.describe(). Debits collected out of band and refused submissions do not consume the budget. What remains unbounded is the hot wallet's own ETH balance, which is an operational choice (fund it thinly) rather than something the hub can enforce.- Point
/channel/openatAIMarketEscrow.openChannelinstead of the settlement wallet. Today escrow is available in addition to, at the client's option; anescrow_requiredswitch is needed so an operator can make the custodial branch unreachable. Reconcile the two rails so the double-refund case is unreachable.Done 2026-07-29 —payment_readiness()refuses a configuration whereAIMARKET_PAYMENT_RECIPIENTequals the escrow address, which is the one setting that backs both rails with the same money. The reverse direction was already unreachable and is now asserted rather than re-implemented: a transfer into escrow cannot be claimed as a ledger deposit, because verification requires the tx to paypayment_recipient, which this interlock keeps distinct. 14 tests, where there had been none for the list behindpayment_configured.Expose the outstanding obligation total on an operator surface.Done — served over HTTP and admin-gated.
Acceptance criteria:
- A hub-issued invoke produces an on-chain
ChannelDebitedevent for its receipt id without a human at a shell. (Events of the right shape exist; nothing the hub runs produced them.) channel/closeproduces aChannelSettledevent whoserefundRecipientleg matches the depositor, andrefund_executed_usdstops being hardcoded to0.0.channel_payout_obligationsholds no owed obligation for a contract-settled channel.- This entry deleted; one-line
CHANGELOG.mdnote.
Until done: describe the hub rail as an off-chain, custodial metering ledger with verified on-chain funding and operator-initiated on-chain collection — never as "escrowed", "non-custodial" or "mirrored on-chain", and never as if collection were automatic. Keep deployments below the value ceiling in KI-4.
When one of these is resolved:
- Delete the entry from this file in the same commit that resolves it.
- Add a one-line entry to
CHANGELOG.md(create if missing):- KI-1 closed: ZK ceremony complete, Verifier.sol committed (see docs/zk-ceremony.md). - If the resolution itself introduces new tracked work, open a new KI entry here for that.
If a new "can't be fixed in-repo" item appears, add it as KI-N with
the same shape: Where / Why / Required action / Acceptance / Until done.