What happens between you typing a sentence and a graded result appearing. Written to be read alongside the code, so every claim below names the file it lives in.
The shape is the same at every stage, and worth holding onto:
A stage is a model session with a small set of tools and its instructions in a markdown file. The model decides what to do; the tools do anything that must be exact and refuse anything that must not happen. Nothing reaches disk except through a tool that checked it first.
There is no pipeline. Each stage is a conversation you can interrupt, correct, and resume.
| Piece | Where | What it is |
|---|---|---|
| Stage | session.py |
A live model session, held open across turns, emitting typed events |
| Instructions | skills/<stage>/SKILL.md |
How that stage works, in prose. Editable without touching code |
| Tools | tools.py, world/tools.py, scenario_tools.py, run/tools.py |
The exact half: they execute, validate, and refuse |
| Artifacts | artifacts/sessions/<id>/ |
What each stage leaves behind for the next |
| Conversation | chat.py |
Holds one agent's journey through the stages |
| Session | sessions.py |
One conversation, one folder: chat and artifacts together |
The artifacts, each the input to the next stage:
contract.json → world.sqlite + handlers/ + simulator_prompt.md + sub_goals.json
→ scenarios/<name>/ (+ scenarios.json as the index) → runs.json
All of it, plus the conversation that produced it, lives in one folder per session. There is nothing held in memory that is not also on disk, so closing the page, restarting the server or coming back tomorrow all resume the same way: by reading the folder.
reception.py
A stage with Read, Glob, Grep and one tool, point_at_agent. You say where your agent
lives; it looks, confirms the path exists, picks a short name, and calls that tool.
It looks from the workspace root (the directory holding your repos), not from inside
agent-learning-kit, because the agent under test is almost never inside the harness.
point_at(name, path, kind) refuses a path that does not exist, and refuses an unknown kind.
kind selects an AgentSource from sources.py — repo (code on disk, gets file tools) or
spec (a prompt and tool schema pasted in, gets no file tools). A new kind of agent is one
registration, not a new code path.
understand.py, skills/understand-agent/SKILL.md, gate in tools.py
The session gets read-only file tools and one submission tool. It reads the agent's source and
calls submit_contract.
The contract (contract.py) is the anti-hallucination device for everything downstream:
| Field | Why it matters |
|---|---|
tools[] — name, args, arg_types, arg_values, description |
The agent's action space. arg_values are the real permitted values — the menu, the enum, the lookup |
hard_constraints[] |
Rules the agent must follow. Told to the agent under test, and graded by the judge |
base_environment |
Its real starting data, reproduced row for row |
real_use_cases[] |
What it is actually for |
notes |
Free-form: whatever else the reader judged worth carrying forward |
amendments[] |
Anything not read from source — see below |
How it is written: accept_contract in tools.py validates before anything reaches disk. It
refuses a contract with no tools, no use cases, duplicate tool names, types for arguments that do
not exist, or — the one that mattered most in practice — every tool having no arguments, which
means the arguments were read and then not recorded. Problems are returned into the
conversation, so the model corrects and resubmits rather than a bad contract landing.
How it is changed later (amend.py). The contract is not frozen, but every change is
recorded with a reason in amendments[], so months later you can still tell what came from the
agent and what came from us:
amend_contract— let an argument accept a value it did not beforeadd_rule/drop_rule— a hard constraint the source did not state, or one misreadfix_tool— correct argument names, types, description, or remove a tool that does not exist
Each demands a why. A contract that can be rewritten invisibly is no longer evidence.
build.py, skills/build-environment/SKILL.md, tools in world/tools.py
This is the part that makes the whole thing worth doing. The repository's own services, migrations, seed process and tool code run in an isolated environment. A call for something that is not there is refused by the submitted implementation, not by a mock or a rewritten handler.
It builds three things, all shared by every scenario: the world, the simulator prompt for a conversational agent, and the sub-goal catalogue. The stage has sixteen tools and no file access at all:
| Tool | Does |
|---|---|
create_schema |
Run the CREATE TABLE statements |
seed |
Insert rows — the agent's real catalogue |
change_data |
One UPDATE or DELETE, for fixing a row put in wrong |
adopt_tool |
Bind and smoke-test the agent's own implementation |
write_env_file / run_env_command |
Container orchestration only; never agent behavior |
run_tool |
Call a defined tool and see what the world does |
declare_sequence / drop_sequence |
A series of calls whose end state must hold |
inspect_world |
Look at what is in the world |
amend_contract, add_rule, drop_rule, fix_tool |
Correct the contract |
check_world |
Run every probe, report without saving |
save_world |
Freeze it — refused unless it holds up |
Bindings are small adapters to the callable the repository ships. There is no generated-handler fallback. If the callable cannot be imported or the shipped service cannot start, the build stops and reports the missing runtime/configuration seam.
The distinction the whole design turns on (runtime.py):
ToolError— the world saying no. The id does not exist; the item is unavailable. This is the world working.- Any other exception — our bug.
They are recorded differently and never confused. _is_refusal matches the agent's own refusal
type across the class hierarchy.
world/probe.py. Every probe restores the world to a frozen baseline first, so probes cannot
inherit each other's rows.
| Probe | Asks |
|---|---|
happy |
A valid call built from the contract's permitted values. A refusal is acceptable; a crash never is |
edge |
A nonexistent id → must refuse, not succeed and not crash. A missing required argument → must refuse |
coverage |
Every contract tool has a handler; no handler for a tool the agent lacks; and each handler actually reads the arguments the contract says it takes |
data |
Every identifier the contract permits exists in the world — catches a whole category left unseeded, which otherwise looks exactly like correct strictness |
sequence |
Each declared sequence, run from the frozen world, leaves the state it claims |
| unknown tool | Calling a tool that does not exist must refuse |
save_world refuses on three counts: score below 0.85; no declared sequence (calls that
each work alone can still forget what the last one did); and a dirty world — rows left over
from building, which would otherwise appear in every scenario as somebody else's order already
in the cart.
Then world/snapshot.py writes world.sqlite, handlers/*.py, world.py and manifest.json.
The snapshot is the base state every scenario restores from.
scenarios.py, skills/write-scenarios/SKILL.md, tools in scenario_tools.py
A scenario is a change on the base environment, and it owns a folder (folder.py):
scenarios/<name>/
scenario.json the instruction, the reference solution, which sub-goals it names
setup.py def setup(world) what this scenario changes first
ready.py def ready(world) is the world ready for it
checks/<goal>.py def check(world, calls) one per deterministic sub-goal
scenarios.json is an index over those folders, regenerated from them, so anything wanting the
whole suite at a glance has it.
Code lives in files, never duplicated into JSON, and the file is the artifact: each check file is
written with a __main__ block so it runs standalone against what a run left behind, and a test
proves the file and the harness give the same answer.
setup is code rather than a list of rows: what a scenario changes is not necessarily the
database alone. There is no persona and no opening line. Variability comes from real
conditions that live in setup, and the base world stays the shared starting point.
sub_goals are names from the catalogue the environment step defined, not restated wording.
That is what makes results roll up: the same sub-goal failing in seven of twelve scenarios is one
sentence.
solution is what a correct agent would do. It is never run against the agent — it exists so the
scenario can be proved.
The writer can look (inspect_world) and rehearse (try_calls — run calls against a
throwaway copy and see what state they leave), so a solution is written from what was observed.
prove.py, called by submit_scenario before a scenario is kept:
| Run | Must | |
|---|---|---|
| Ready | reset → setup → ready |
hold |
| Solvable | reset → setup → the solution → the checks | pass |
| Not vacuous | reset → setup → nothing → the checks | fail |
If the first fails, the world does not hold what the scenario presumes, and running it would test us rather than the agent: the agent would fail for a precondition we got wrong, and it would read as a finding about the agent. If the second fails, either the scenario cannot be passed or the check is wrong. If the third passes, the checks grade nothing while reporting a result, which is the failure that makes a suite quietly green.
Vacuity is judged on all checks passing, because one check surviving an empty run ("no
unavailable item was ordered") is legitimate. A single check that survives is still reported, in
Proof.weak: sub-goals are shared, so a check that cannot fail without calls would roll up as a
pass for an agent that did nothing at all.
save_scenarios additionally refuses a suite where no sub-goal is shared by two scenarios,
because nothing would roll up.
run/
The simulation is not a loop written here. ALK already owns placing a call, driving the
synthetic user, and producing a transcript; the harness supplies the world, the instruction, and
the grading. Against the real hosted agent that is run/live.py and run/call.py:
world + setup ──► webhook ──► public url ──► the assistant's OWN tools repointed
│
ALK's voice case places the call ──┘
│
the world afterwards + the calls ──► the sub-goals' checks
- Restore the frozen world and apply this scenario's
setup. Its own copy, so nothing leaks between scenarios. - Stand up the webhook (
run/voice.py) and bind that world to it. A hosted voice agent executes its tools by calling a webhook, so answering that webhook fromhandle_tool_callis the entire integration. - Expose it (
cloudflared, orHARNESS_WEBHOOK_URL), because a hosted agent cannot reach loopback. - Repoint the assistant.
pointed_atcopies the agent's own tools and changes onlyserver.url. Nothing about the agent is redefined — rebuilding its tools would mean testing an agent we wrote. - Place the call through ALK's own voice case, with the scenario's filled simulator prompt driving the caller.
- Grade from the world afterwards plus the recorded calls, through the same checks the gates
used. A sub-goal marked
judgedis reported as judged, never silently counted.
run/alk.py is the same story for the text path: the world goes in as environment= to ALK's
ChatEnvironment, which owns the turn loop.
Running is also a stage of the conversation, not only a command (run/stage.py,
skills/run-scenarios/SKILL.md, tools in run/tools.py: preflight, list_scenarios,
run_scenario, read_results). The stage exists because reading a failure is judgement: it has
to sort every failure into one of four causes — the agent was wrong, the world wrongly refused,
the check is wrong, or the simulated caller never asked for the thing — and only the first is a
finding about the agent. Each run's record lands in runs.json with the instruction, the
per-sub-goal verdicts, every tool call, and the transcript.
This is what the environment was built for. The previous run's known issues — "mocked tools
always succeed, including removing an item that was never added", "mock responses do not vary by
argument", "world state does not change unless a scenario sets state_updates" — are all the
same defect, and a world that really holds rows and can really refuse answers all three.
The split is deliberate and worth defending:
| Judgement (the model) | Exact (code) |
|---|---|
| Reading unfamiliar source | Whether the contract is structurally usable |
| Designing a schema | Whether a handler crashes or refuses |
| Choosing what is worth testing | Whether an expectation resolves against real tables |
| Whether a claim held | Whether the state matches |
The model never decides whether something passed. It decides what to try.
- A new agent kind → a class in
sources.pyand one registration - A new world kind (browser, filesystem, queue) → a class in
world/kinds.pyimplementingvalues_present/mutable_state/describe, and one registration. Browser is registered and stubbed - A new place the agent runs (Vapi, LiveKit, a hosted endpoint) → a class in
run/targets.pywithopen/say/close, whose tool calls reach the sameworld.handle_tool_call - A change to how a stage works → edit its
SKILL.md. No code
- Browser worlds: registered, not implemented
- Snapshots are local files, not S3
- Judged sub-goals are reported as judged, not actually sent to a judge yet
- Results do not post to the platform
- Nothing reports which of the contract's use cases have no scenario
A HarnessJob accepts guidance, a list of operator instructions, and
require_tool_workflow, which defaults to false. Both travel through understanding,
environment authoring, scenario authoring, adjustment rewinds, and runtime repairs.
Use require_tool_workflow: true only when every scenario must exercise a tool.
With this requirement enabled, authoring rejects a contract unless a declared tool has an executable import, constructed callable, or service entrypoint. Runtime validation checks the contract again after repairs. ALK does not invent tools or modify submitted agent code to satisfy the requirement.
Declarations do not prove execution. Local, resumed, certified, and hosted execution
require captured tool calls in every scenario. Saved job requirements also apply to
interactive scenario tools and cannot be disabled by a caller passing false. A conversation without tool evidence follows
the existing evidence_missing retry and errored-receipt policy, even if its generated
scenario claims tool evidence is optional. Ordinary chat and voice jobs keep their
existing behavior when the requirement is omitted. Scenarios intended to refuse a
request without calling tools belong in a separate job without this requirement.
Use runtime.execution: native with a source-owned command, or
runtime.execution: compose with a repository-relative compose_file and service.
Omitting execution retains automatic selection. Explicit native selection ignores
unrelated Compose files. Explicit Compose selection preserves the selected service's
transitive dependencies and resolves build contexts relative to its Compose file.
Selection does not add execution support: unsupported images, engines, installers,
and container launch semantics fail with a capability error.
runtime.required_environment maps process names to required environment-variable
names. Only values delivered to that process satisfy its declaration. Source-wide
credential discovery remains an inventory of possible requirements. The preflight
response preserves ready_to_submit as its legacy inventory result and separately
reports can_submit_for_authoring and configuration_status. Submission permission
is not proof that the selected runtime will start or complete a conversation.
For an OpenAI-compatible ingress, runtime.interface.model records the exact model
or profile identifier accepted by that endpoint. It is passed unchanged in the
openai_chat request's model field; it may differ from the backing provider model.
Omitting it preserves the legacy agent-learning-target default. Authoring should
record a source-evidenced identifier when the endpoint requires one. For a custom
JSON envelope, json_template can instead supply a literal model and
"messages": "{{messages}}" in request_template; openai_chat does not use templates.
agent.config.target_model supplies the caller's desired backing model, for example
vertex_ai/gemini-3.7-flash. Authoring and runtime repair receive this preference and
the names of supplied target environment values. They must derive the repository's
provider configuration and initialization requests from source and verify a real
chat. Secret values, vault references, simulator credential names, and arbitrary
connection settings are not included in this briefing. The backing model preference
does not replace runtime.interface.model, which may identify an application profile.
runtime.extras selects optional dependency groups declared in the selected Python
project. For a uv workspace root, a group absent from the root must have exactly
one owner among included members. The installer then includes workspace packages
and only the named extras. Root-owned groups use the root project alone; ambiguous,
unknown, invalid, and unsupported selections fail before installation. Compose
applies these selections only to the control process. Existing pip project paths
install the selected .[extras]; requirements-only sources cannot declare extras
without a project manifest. Locked uv installs and their stale-lock retry preserve
the same selection. Source manifests and lockfiles are not rewritten by selection.
runtime.install remains descriptive metadata; arbitrary installation instructions
are not executed by the process compiler. Declare supported optional groups through
runtime.extras rather than relying on that string. A complete local editable declaration
such as pip install -e sdk[cloud] -e tools is checked against the selected
project and its uv workspace. If the exact source owner declares an installable
group omitted from extras, compilation returns generated_runtime_extra_omitted
through the runtime-plan repair path. A repair must preserve the proven source
owner and select the group; deleting the install text does not resolve the error.
Unsupported install syntax remains documentary and never executes.
runtime.requirements_files selects additional Python dependency manifests relative
to the selected component. Both compilers install these into the same environment
after project dependencies and extras, preserving every installed package version.
Conflicting supplemental requirements fail instead of replacing the selected baseline.
Selection accepts registry package requirements
and local -r / -c includes; paths must remain inside the submitted component,
exist in the build context, and have an acyclic bounded include graph. URLs and
other pip options fail explicitly. The bounded chat-readiness repair may add
source-evidenced manifests while retaining existing selections and all other runtime
fields. runtime.install alone does not select a requirements file.
HTTP endpoint paths can reference canonical inputs and earlier setup captures, such
as /threads/{{session_id}}/turns. Setup runs first; the main path then renders for
each call without changing the stored endpoint. Each substituted value is encoded
as one path segment, so it cannot introduce a new host, path separator, query, or
fragment. Missing values fail before the dependent request. Placeholders outside
the URL path are unsupported. Traces retain the endpoint template and redact captured
values, including their encoded forms, from failure diagnostics.
HTTP status failures take precedence over success-response extraction. For an API
that returns application failures with a successful HTTP status, declare
runtime.interface.failure_rule, for example:
{"status_path": "state", "failure_values": ["failed"], "message_path": "detail"}Each runtime.interface.setup_requests entry also accepts failure_rule. Its
application status is checked after the accepted HTTP status and before captures or
the next request. For example, {"status_path":"code","success_values":[0]}
rejects an HTTP 200 setup response whose application code is 102. Comparisons preserve
JSON scalar types; numeric zero does not match boolean false or the string "0".
ALK does not infer failure merely because an arbitrary response contains an error
field. Failed calls retain partial conversation evidence and bounded diagnostics.
Excerpts are omitted when the caller cannot supply credential context. HTTP bodies
larger than 1 MiB are rejected explicitly. Process tails survive reset and teardown;
retention reads at most 64 KiB per file, sanitizes before persistence, and retains up
to 8,192 characters per tail with a 512-log attempt budget. This is diagnostic
redaction, not a guarantee that arbitrary personal data has been removed.
Public PEM keys and certificate-only chains may pass source and artifact checks when the complete bounded content parses as public material. Private, mixed, malformed, oversized, and credential-named files remain blocked.
For generic source-owned HTTP or callable chat agents, ALK runs one representative opening request after scenario setup and ready checks, before source-data invariant review. It uses the execution transport, edited persona opening or a greeting, and the declared model, setup requests, response mapping, and tools. Valid text, clarification, refusal, or structurally usable tool requests establish interface readiness. ALK does not replay returned tools or judge task success in this check. The target itself may execute tools during the request.
Failed requests prevent certification. Candidate configuration errors use the
existing source-backed runtime-plan repair budget. Authentication and unexplained
application failures remain errors; they do not permit rewriting scenario outcomes.
Each attempt retains a bounded sanitized receipt under generic-harness/chat-readiness.
ALK resets a successful disposable instance before continuing and closes failed
instances. Certificates require explicit chat_ingress evidence. Voice and external
connect-only providers record that this check is not applicable.
One successful opening does not prove every scenario or a complete workflow. Socket operations share a deadline, but urllib DNS resolution and response-header parsing are not independently interruptible. The enclosing job and sandbox limits remain the final execution bounds; cancellation drains the client worker before cleanup.