GitHub - cmrhyq/TestAgents: API Automated Testing Agent Based on Large Models

LLMTestAgent

LLM-Driven API Automation Testing Agent

Talk to your APIs in natural language. The agent picks its own tools — under a hard permission boundary.

Quick Start · Model Setup · Agent Runtime · Memory · Development

中文 | English

Features

Space-scoped sandbox — every test space owns a real directory (AGENTS.md, docs/, testcases/, artifacts/); file tools can never escape it
Mode = tool whitelist — agent / ask / plan are enforced on the execution path, not suggested in a prompt. In ask mode the model literally has no write_file to call
Streaming agent runtime — hand-written ReAct loop with a 6-event SSE protocol (start / text_delta / tool_call / tool_result / final / error); every tool call is visible in the UI
Skills — SKILL.md folder convention, builtin + per-space discovery, zip import with zip-slip protection, each skill narrows the tool set further
Three-layer memory — in-request context, cross-session memory per space (DB as source of truth, mirrored to memory/MEMORY.md), and a document knowledge base with chunking + optional vector search
OpenAPI 3.x parsing (JSON / YAML) with automatic endpoint extraction and persistence
Assertion engine — JSONPath-based rules (==, contains, matches, status_code, response_time, …) exposed to the agent as tools
Production guard — environments flagged is_production reject write methods and out-of-domain URLs
Graceful degradation — ChromaDB and embeddings are off by default; the knowledge base falls back to lexical search instead of failing
Everything persisted — messages, per-tool-call traces with HTTP request/response payloads, and per-LLM-call token and cost rows; a run can be replayed offline (see Test Reports and Usage)

Project Structure (Monorepo)

LLMTestAgent/
├── backend/          # Python backend (FastAPI + ReAct agent runtime)
│   ├── src/          # Source code
│   ├── skills/       # Builtin SKILL.md folders
│   ├── workspaces/   # Per-space sandbox directories (gitignored)
│   ├── tests/        # Unit / API tests
│   ├── pyproject.toml
│   └── ...
├── frontend/         # React frontend (Vite + Tailwind v4 + shadcn/ui)
│   ├── src/
│   ├── package.json
│   └── ...
├── doc/              # Design documentation
└── .github/          # CI workflows (backend.yml + frontend.yml)

Architecture

The runtime is four stages, and only the third one touches an LLM:

flowchart LR
    client["Browser / curl"] -->|"POST /api/v1/chat/stream"| api["FastAPI"]
    api --> prep["① conversation<br/>+ context assembly<br/>(no LLM)"]
    prep --> runner["② ReAct loop<br/>(LLM + tools)"]
    runner -->|"SSE events"| client
    runner -->|"save answer"| db[(SQLite)]
    runner -.->|"file tools"| ws[("workspaces/&lt;space&gt;/")]
    runner -.->|"send_request"| target[API under test]

Stage	Component	LLM
① Conversation	`ChatStreamService.ensure_conversation` — reuse or create conversation, persist user message	❌
② Context	`prepare()` + `agent/context.py` — snapshots space, history, memories, skills, tool set, `ModePolicy`	❌
③ Reasoning	`agent/runner.py` — ReAct loop, stream events	✅
④ Persist	`api/v1/chat.py` — render SSE frames, save answer	❌

Because stages ① ② ④ are deterministic, "which tools were available this turn" and "why did this message not reach the model" are answerable offline, without trusting the model's self-report.

See Architecture.md and SystemFlowchart.md for the full picture.

Quick Start

1. Clone and Install

git clone https://github.com/cmrhyq/LLMTestAgent.git
cd LLMTestAgent

# Backend
cd backend
uv sync --extra dev
cd ..

# Frontend
cd frontend
npm install
cd ..

2. Configure the Model

All configuration lives in backend/config/config.yaml — there is no .env and no ${ENV} interpolation. Fill in your provider credentials directly in model_list:

llm:
  default_model: smart-router      # complexity-based auto routing

model_list:
  - model_name: glm-4-flash
    litellm_params:
      model: zai/glm-4-flash
      api_key: your-api-key
  - model_name: deepseek-v4-pro
    litellm_params:
      model: deepseek/deepseek-v4-pro
      api_key: your-api-key

config.yaml holds plaintext API keys — make sure it stays out of version control.

3. Start Services

# Backend
cd backend
uv run python app.py
# or
uv run uvicorn app:app --host 0.0.0.0 --port 8000 --reload

# Frontend (another terminal)
cd frontend
npm run dev

On first run the database (backend/db/TestAgents.db), all 12 tables, and the migrations are created automatically. Space directories under backend/workspaces/ are created the first time you create a space.

Project Structure

LLMTestAgent/
├── backend/
│   ├── app.py                           # FastAPI entry (lifespan: config → db → migration)
│   ├── config/config.yaml               # Application config (only source of config)
│   ├── db/TestAgents.db                 # SQLite database (auto-created)
│   ├── skills/                          # Builtin skills (SKILL.md folders)
│   ├── workspaces/                      # Per-space sandbox dirs (auto-created, gitignored)
│   └── src/
│       ├── agent/                       # Agent runtime
│       │   ├── modes.py                 # Mode → tool whitelist (the permission boundary)
│       │   ├── context.py               # system prompt assembly (≤12k chars)
│       │   ├── events.py                # SSE event protocol
│       │   ├── runner.py                # Hand-written ReAct loop
│       │   └── tools/                   # fs / http / db / memory / knowledge / skill / executor tools
│       ├── workspace/                   # Space directory + path sandbox
│       ├── skill/                       # SKILL.md parsing, discovery, zip import, enable flags
│       ├── memory/                      # Long-term memory + knowledge base
│       ├── api/                         # FastAPI route layer
│       ├── core/                        # config, database, llm gateway, chroma, cache, logging
│       ├── data/                        # models / schemas / repositories / services / migration
│       └── utils/                       # snowflake IDs, HTTP helpers, OpenAPI parser
├── frontend/src/
│   ├── pages/                           # dashboard / spaces(+7 tabs) / chat / skills / run / report
│   ├── components/chat/                 # mode & skill selectors, tool-call cards, bubbles
│   ├── hooks/                           # TanStack Query hooks + use-chat (stream orchestration)
│   └── lib/                             # api client, SSE parser, typed domain models
└── doc/                                 # Design docs

Agent Runtime

Modes

Mode	Meaning	Tools	Max steps
`agent` (default)	Pick tools and actually test	17	100
`ask`	Answer questions only	12 (read-only)	100
`plan`	Produce a plan, execute nothing	12 (read-only)	80

ask / plan never receive send_request, write_file, delete_file, remember, or forget. Selecting a skill can only narrow this set further, never widen it.

Tools

Group	Tools	Guard
Files	`read_file`, `write_file`, `list_directory`, `delete_file`	Resolved path must stay inside the space root; `AGENTS.md` and `memory/MEMORY.md` are protected
HTTP	`list_environments`, `send_request`	Write methods rejected on production environments; URL must belong to that environment
Data	`get_space_endpoints`, `get_endpoint_detail_by_id`, `search_space`	Always filtered by `space_id`
Memory	`recall`, `remember`, `forget`	Writes are mirrored to `memory/MEMORY.md`
Knowledge	`knowledge_search`	Falls back to lexical search when vectors are unavailable
Skills	`list_skills`, `read_skill`	Read-only
Assertions	`evaluate_assertions`, `extract_jsonpath`	Pure functions

Tool calls are recorded in agent_tool_call (arguments, result, HTTP request/response with sensitive headers masked), so any run can be replayed offline at /runs/:id.

SSE events

Event	Payload
`start`	`conversation_id`, `space_id`, `mode`, `tools`
`text_delta`	`content`
`tool_call`	`id`, `name`, `args`
`tool_result`	`id`, `name`, `ok`, `content`, `duration_ms`
`final`	`answer`, `usage`
`error`	`message`

Tool failures do not break the stream — they come back as ok=false results so the model can correct itself.

Memory System

Layer	Storage	Editable	Purpose
Context	Assembled per request	❌	What the model sees this turn
Cross-session	`space_memory` table + `memory/MEMORY.md` mirror, isolated per space	✅	Conventions, past conclusions, known pitfalls
Knowledge base	`knowledge_document` + `knowledge_chunk`, optional vectors	✅	Requirement docs, API specs

Memory recall uses keyword extraction plus SQL LIKE rather than vector search: entries are few and precise, and explainability matters more than recall here. The knowledge base is the layer that benefits from embeddings.

# Add a memory to a space
curl -X POST http://localhost:8000/api/v1/spaces/<space_id>/memories \
  -H "Content-Type: application/json" \
  -d '{"title": "Auth header", "content": "All endpoints need X-Token", "category": "fact"}'

# Upload a document to the knowledge base
curl -X POST http://localhost:8000/api/v1/spaces/<space_id>/documents \
  -F "[email protected]"

Skills

backend/skills/api-test-design/
└── SKILL.md

---
name: api-test-design
description: Design API test scenario matrices
allowed_tools: [get_space_endpoints, read_file]
---

Methodology injected into the system prompt…

Discovery is two-level: backend/skills/ (builtin) and <space>/.skills/ (space-scoped, overrides builtin by name). Import a zip via POST /api/v1/skills/import (≤20MB, path escape rejected).

Usage

API Modules

Module	Path Prefix	Function
Spaces	`/api/v1/spaces`	CRUD, workspace tree, `AGENTS.md`, memories, documents
Endpoints	`/api/v1/endpoints`	Endpoint definition CRUD
Environments	`/api/v1/environments`	Environment CRUD (incl. `is_production`)
Test Runs	`/api/v1/agent/runs`	Execution history, tool-call traces, cleanup
Reports	`/api/v1/reports`	Per-run HTTP report aggregated from `send_request` calls
Chat	`/api/v1/chat`	`POST /stream` — the single agent entry point (SSE)
Conversations	`/api/v1/conversations`	Session list, messages, rename, delete
Skills	`/api/v1/skills`	List / detail / zip import / enable
Memory	`/api/v1/memories`	Update / delete an entry
Knowledge	`/api/v1/documents`	Chunks / reindex / delete
Usage	`/api/v1/usage/stats`	Token and cost aggregated by space / conversation / window
OpenAPI	`/api/v1/spaces/parse/openapi/{space_id}` · `/api/v1/workflows/upload/openapi`	Parse and persist a spec

Example Requests

# Parse an OpenAPI document into a space
curl -X POST "http://localhost:8000/api/v1/spaces/<space_id>/parse/openapi" \
  -F "file=@httpbin_service.json"

# Ask the agent to test something (streaming)
curl -N -X POST http://localhost:8000/api/v1/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"instruction": "Test the login endpoint with valid and invalid credentials",
       "space_id": <space_id>,
       "mode": "agent",
       "skill_names": ["api-test-design"]}'

The response carries an X-Conversation-Id header so clients can continue the same session.

Test Reports

A report is no longer a file row — it is one execution's HTTP calls, aggregated on the fly from agent_tool_call where tool_name = 'send_request':

Endpoint	Returns
`GET /api/v1/reports?space_id=&status=&page=&page_size=`	One row per `run_id` that issued HTTP requests, with counts and success rate
`GET /api/v1/reports/{run_id}`	Run summary + per-request list (no bodies)
`GET /api/v1/reports/{run_id}/requests/{request_id}`	One request's full request/response payload (lazy-loaded by the UI)

Success rate is computed from HTTP 2xx, not the tool-layer ok flag: send_request never raises on 4xx/5xx, so ok=1 + status 500 is a real combination. Failures are split into error_count (response received, non-2xx) and exception_count (no status code at all — network error, invalid args, or blocked by the production gate).

The old report table and the file-download endpoint are gone; migration v4 drops the table if it exists.

Database

SQLite + SQLAlchemy 2.0, snowflake IDs (non-autoincrement), created and migrated on first run.

Table	Description
`space`	Test space (owns a workspace directory)
`environment`	Environment config (`is_production` gate)
`endpoint`	Parsed API definitions
`agent_run`	One row per ReAct turn: mode, status, tool counters, token and cost totals
`agent_tool_call`	One row per tool call: arguments, result, HTTP request/response
`conversation` / `message`	Sessions (`mode`, `skill_names`, `environment_id`)
`space_memory`	Cross-session memory
`knowledge_document` / `knowledge_chunk`	Knowledge base
`skill`	Skill discovery cache + enable flag
`llm_log`	LLM call accounting: real model, tokens, routing tier, cost, attributed to space / conversation / run

Reset Database

rm backend/db/TestAgents.db     # macOS / Linux
del backend\db\TestAgents.db    # Windows

ER Diagram

See ER.md and DatabaseDesign.md.

ChromaDB (Optional)

ChromaDB is off by default. With it disabled the knowledge base still works through lexical search.

chroma:
  enabled: false                   # set true to use vector search
  host: "your-chromadb-host"
  port: 8000
  auth_credentials: your-token
  tenant: default_tenant
  database: default_database

embedding:
  enabled: false                   # also needs to be true for vectors
  model: zai/embedding-3           # any litellm-compatible embedding model
  api_key: your-api-key
  dimension: 1024

When both flags are on, knowledge_search uses vectors and automatically degrades to lexical search on failure. Existing documents can be re-indexed with POST /api/v1/documents/{id}/reindex.

Development

# Backend tests
cd backend
uv run pytest
uv run pytest --cov=src --cov-report=html

# Frontend checks
cd frontend
npm run type-check && npm run lint && npm run build

Add a Tool

Write a make_*_tools(...) factory in backend/src/agent/tools/ that returns @tool functions
Register it in build_tools() (backend/src/agent/tools/registry.py)
Classify it in src/agent/modes.py — add it to SIDE_EFFECT_TOOL_NAMES unless it is read-only
Add a test; tests/agent/test_skill_tool_contract.py will fail if a skill declares a tool that does not exist

Add a Skill

Create backend/skills/<name>/SKILL.md with name, description, and optional allowed_tools. allowed_tools must reference real tool names — a typo silently removes that tool from the bundle.

Database Changes

Models live in backend/src/data/models/. New tables are picked up by init_database_from_orm(); schema changes to existing tables need a migration module (see data/migration/v2_agent_migration.py), because SQLite cannot alter CHECK constraints in place.

Tech Stack

Component	Technology
Agent orchestration	Hand-written ReAct loop (no LangGraph)
LLM framework	LangChain messages + tool binding, LiteLLM Router
Vector database	ChromaDB (optional)
Web framework	FastAPI + Uvicorn
Database ORM	SQLAlchemy 2.0
Data validation	Pydantic 2
HTTP client	requests
Logging	structlog
Configuration	PyYAML

FAQ

Where is the database file? backend/db/TestAgents.db, created on first run. Change database.url in config.yaml.

Why can't the agent write files in ask mode? By design — ask and plan only bind read-only tools. It is not a prompt instruction the model might ignore; the tool simply is not in the list.

Why does knowledge_search not use vectors? chroma.enabled and embedding.enabled both default to false. Enable both, then call reindex on existing documents.

Changing HTTP timeouts in config.yaml has no effect. There is no timeout section in the config schema. send_request takes its timeout as a tool argument (default 30s); the old execution block inherited from the previous engine has been removed from both config.yaml and the template. If you need a global default, wire it into src/agent/tools/http_tools.py rather than adding YAML nobody reads.

No report after running? Verify the OpenAPI document parses (3.0.x / 3.1.x), the model credentials are valid, and the space actually contains endpoints.

Why is cost_usd null in /api/v1/usage/stats? It means no call in the window could be priced — not free. Usually the model is missing from litellm's price table (zai/* is). Add a rate in config.yaml (unit is USD per single token):

pricing:
  - model: zai/glm-4-flash
    input_cost_per_token: 0.0000001
    output_cost_per_token: 0.0000002

Check unpriced_calls alongside it: cost_usd only sums the priced calls, so reading it alone understates the real spend.

License

Licensed under the Apache License 2.0.

Name		Name	Last commit message	Last commit date
.githooks		.githooks
.github/workflows		.github/workflows
backend		backend
doc		doc
frontend		frontend
.gitignore		.gitignore
.impeccable.md		.impeccable.md
.pre-commit-config.yaml		.pre-commit-config.yaml
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
README_zh.md		README_zh.md
package-lock.json		package-lock.json

Navigation Menu

Repository navigation

About

Repository files navigation

Features

Project Structure (Monorepo)

Architecture

Quick Start

1. Clone and Install

2. Configure the Model

3. Start Services

Project Structure

Agent Runtime

Modes

Tools

SSE events

Memory System

Skills

Usage

API Modules

Example Requests

Test Reports

Database

Reset Database

ER Diagram

ChromaDB (Optional)

Development

Add a Tool

Add a Skill

Database Changes

Tech Stack

FAQ

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Features

Project Structure (Monorepo)

Architecture

Quick Start

1. Clone and Install

2. Configure the Model

3. Start Services

Project Structure

Agent Runtime

Modes

Tools

SSE events

Memory System

Skills

Usage

API Modules

Example Requests

Test Reports

Database

Reset Database

ER Diagram

ChromaDB (Optional)

Development

Add a Tool

Add a Skill

Database Changes

Tech Stack

FAQ

License

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages