harmony

Harmony classifies products against the US Harmonized Tariff Schedule — and it's the only classifier that will tell you its own error rate.

Paste a product (or upload a catalog CSV) and a Gemini agent walks the real 2026 HTS tree, cites real US Customs (CROSS) rulings, and answers with a 10-digit code, the duty rate, and an honest confidence. Every night — and on demand — it grades itself against federal rulings it has never seen, reads its own misses over the Phoenix MCP server, patches its own prompt, and only ships the patch if the measured score goes up. Then it publishes the falling error curve.

"Never trust an agent that won't show you its error rate."

Built for the Google Cloud Rapid Agent Hackathon — Arize track. AI inside the product is Google only (Gemini via Vertex AI, Vertex AI embeddings) plus Arize Phoenix.

---

The four surfaces

1. Classify (/) — one product in, one verified answer out: the code in big mono digits, duty rate, confidence with broker flags, the reasoning tree level by level, cited CROSS rulings linking to rulings.cbp.gov, a live Phoenix trace link — and, when its record warrants it, the agent quoting its own past graded mistake before it answers.
2. Catalog (/catalog.html) — the same job at volume: CSV in → classified, flagged, traced table → CSV out (≤200 rows per run so every row stays traced).
3. Report Card (/report-card.html) — measured accuracy round over round, the agent's self-named failure families with before/after, round history, prompt lineage with the gate's refusals visible, the self-written retrieval glossary, and Improve now: a real grade → introspect → fix → gate round, watchable in about a minute.
4. Settings (/settings.html) — prompt pinning, thresholds, batch sizes, the promotion-gate margins, Phoenix/MCP status and deep links, data controls.

The self-improvement loop

Each night at the configured hour (and whenever someone clicks Improve now):

1. Grade — the agent classifies a sample of held-out CROSS rulings (all post-dating the model's training cutoff). Recorded as a Phoenix experiment: exact-match code evals at 4/6/10 digits + Gemini-as-judge on reasoning quality.
2. Introspect — an ADK agent with the Phoenix MCP toolset reads the misses and its own experiment history over MCP and clusters them into named failure families ("String lights filed as lamps, not electric garlands"). Each family gets a Phoenix dataset, so the weakness stays measurable forever.
3. Fix — per family, two levers: a new classification rule appended to the prompt (saved as a new versioned Phoenix prompt) and/or a retrieval-glossary entry ("fairy lights" → "electric garlands") so the rulings search stops missing the right precedent.
4. Gate — the candidate prompt re-runs the same exam. It is promoted only if it clears every margin (10-digit ≥ +2.0pt, no backsliding at 4/6 digits, judge ≥ incumbent). Refusals stay visible in the lineage.
5. Feed — live visitor classifications (text only) and wrong-code reports become candidate eval cases for the next round.

What is real

• Pasted products and uploaded CSVs run the full live pipeline — Vertex AI embedding → CROSS rulings retrieval → Gemini walks the actual 2026 HTS tree → mid-answer self-introspection over Phoenix MCP → answer with citations. There are no canned answers; a stranger's product text works cold.
• The HTS tree and duty rates ship from the public-domain 2026 export at hts.usitc.gov (backend/scripts/fetch_hts.py rebuilds them).
• The cited rulings come from a scraped corpus of real CROSS rulings; every citation links to the live federal page at rulings.cbp.gov.
• The held-out exam is a separate set of recent rulings (strictly newer than everything in the retrieval corpus), so the agent can never have seen — or retrieved — an exam answer.
• Every trace link is real: OpenInference instruments each agent step into Phoenix Cloud; the link on a result card opens that very answer's spans.
• The memory callout is real: retrieved over the Phoenix MCP server from actually-graded past misses. It doesn't appear when no relevant miss exists, and it disappears when Phoenix is unreachable rather than being faked.
• Before the first graded round the UI shows honest empty states ("first round runs tonight") — never projections.

Disclosed fallback: wherever the Phoenix MCP server lacks a write surface the loop needs (creating datasets, experiments, prompt versions, annotations), Harmony performs that one call with the official arize-phoenix-client instead. Everything the agent reads about itself mid-answer and during introspection goes over MCP.

Architecture

repo/
├─ frontend/            static HTML/CSS/JS (no framework, no build step) + PWA manifest
│  ├─ index.html        Classify (working steps, clarify, result card, memory callout)
│  ├─ catalog.html      CSV upload → streaming table → export
│  ├─ report-card.html  curve, families, Improve-now live stages, history, lineage
│  ├─ settings.html     the dials
│  └─ js/, css/, art/   api client, page logic, design tokens, final art + OFL fonts
├─ backend/
│  ├─ harmony/
│  │  ├─ main.py        FastAPI app: API + serves frontend/ (one Cloud Run service)
│  │  ├─ agent/         ADK agent: prompts, tools, classifier pipeline, prompt versions
│  │  ├─ loop/          graded rounds: evals, introspection (MCP), gate, scheduler
│  │  ├─ api/           routes (SSE classify/catalog/rounds) + catalog jobs
│  │  ├─ hts.py         2026 HTS tree (indent-driven), duty resolution
│  │  ├─ rulings.py     CROSS corpus + Vertex embeddings retrieval + learned glossary
│  │  ├─ memory.py      graded-miss memory (read over Phoenix MCP, ranked by embedding)
│  │  ├─ mcp_client.py  persistent session to @arizeai/phoenix-mcp
│  │  ├─ phoenix_io.py  datasets / experiments / prompts / annotations (phoenix-client)
│  │  ├─ tracing.py     OpenInference → Phoenix Cloud, trace deep links
│  │  └─ store.py       Firestore (deployed) / in-memory (dev) app store
│  └─ scripts/          data pipeline: fetch_hts, scrape_cross, build_embeddings
└─ data/                shipped datasets: HTS tree, rulings corpus + embeddings,
                        held-out exam, sample catalog (demo input)

Request flow (classify): browser → POST /api/classify (SSE) → ADK LlmAgent (Gemini on Vertex AI) calls hts_* tools, rulings_search (Vertex embeddings over the corpus, glossary-expanded), and check_my_record (Phoenix MCP → graded-misses dataset) → final JSON is validated against the real HTS tree (the cited code must exist; cited rulings must have been retrieved) → result + live trace link.

Data stores: Phoenix Cloud holds traces, experiments, datasets and versioned prompts (the public evidence). Firestore holds the app's own small state: rounds, families, glossary, eval-candidate pool, prompt lineage. The visitor's own history lives in their browser (Settings → "Forget me" purges it).

Running locally

Prereqs: Python 3.12, Node 20+ (for the Phoenix MCP server via npx), Google Cloud ADC (gcloud auth application-default login).

cd backend
python -m venv .venv && .venv/Scripts/activate     # or source .venv/bin/activate
pip install -r requirements.txt
cd ..
cp .env.example .env                                # fill in your values
python -m harmony.main                              # http://localhost:4100

The repo ships with prebuilt data (data/). To rebuild from the live sources:

python backend/scripts/fetch_hts.py          # 2026 HTS from hts.usitc.gov (~30s)
python backend/scripts/scrape_cross.py       # CROSS rulings corpus + held-out exam (~25 min, throttled)
python backend/scripts/build_embeddings.py   # Vertex AI embeddings for the corpus

Deploying (Cloud Run)

gcloud run deploy harmony \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --min-instances 1 --max-instances 1 \
  --memory 1Gi \
  --set-env-vars GOOGLE_GENAI_USE_VERTEXAI=TRUE,GOOGLE_CLOUD_LOCATION=global,PHOENIX_BASE_URL=...,PHOENIX_PROJECT=harmony,HARMONY_STORE=firestore,HARMONY_SCHEDULER_TOKEN=... \
  --set-secrets PHOENIX_API_KEY=PHOENIX_API_KEY:latest

• --min-instances 1 --max-instances 1: the nightly scheduler and the single global "Improve now" run live in-process; one instance keeps them coherent (round state is also persisted to Firestore, so restarts are safe).
• Nightly rounds run from the in-process scheduler. For production-grade scheduling, point Cloud Scheduler at POST /api/rounds/run with header X-Harmony-Scheduler-Token: $HARMONY_SCHEDULER_TOKEN.
• Firestore: create a Native-mode database once (gcloud firestore databases create --location=us-central1).

Environment variables

Variable	Default	What it does
GOOGLE_GENAI_USE_VERTEXAI	TRUE	Use Vertex AI via ADC (the deployed path). FALSE + GOOGLE_API_KEY is a dev-only fallback.
GOOGLE_CLOUD_PROJECT	—	GCP project for Vertex AI + Firestore.
GOOGLE_CLOUD_LOCATION	global	Vertex location for Gemini 3.x models.
HARMONY_EMBED_LOCATION	us-central1	Regional Vertex endpoint for text embeddings.
PHOENIX_BASE_URL	—	Phoenix space URL, e.g. https://app.phoenix.arize.com/s/<space>.
PHOENIX_API_KEY	—	Phoenix API key (tracing + client + MCP). Keep it in Secret Manager.
PHOENIX_PROJECT	harmony	Phoenix project name receiving traces.
HARMONY_GEMINI_MODEL	gemini-3.5-flash	The agent's model (Vertex AI).
HARMONY_JUDGE_MODEL	gemini-3.5-flash	Gemini-as-judge model.
HARMONY_EMBEDDING_MODEL	text-embedding-005	Vertex embedding model (must match data/embeddings_meta.json).
HARMONY_STORE	auto	firestore | memory | auto (firestore when a project is set).
HARMONY_FIRESTORE_DB	(default)	Firestore database id.
HARMONY_FIRESTORE_PREFIX	harmony	Collection prefix.
PORT	4100	Server port (Cloud Run sets this).
HARMONY_SCHEDULER_TOKEN	empty	Shared secret enabling POST /api/rounds/run. Empty = route off.
HARMONY_NIGHTLY_ENABLED	true	Nightly round on/off (editable in Settings).
HARMONY_NIGHTLY_TIME_UTC	02:00	Nightly round time (UTC).
HARMONY_NIGHTLY_BATCH	100	Held-out rulings per nightly round. Sized for Phoenix free-tier span budget (~25k spans/month) — raise it on a paid space.
HARMONY_IMPROVE_BATCH	12	"Improve now" batch — scoped so the live round stays watchable.
HARMONY_GATE_TEN_DIGIT_MIN_PT	2.0	Gate: minimum 10-digit gain to promote.
HARMONY_GATE_BACKSLIDE_MAX_PT	0.5	Gate: max allowed 4/6-digit backslide.
HARMONY_FOLD_LIVE_TRAFFIC	true	Fold visitor classifications (text only) into next round's eval candidates.
HARMONY_BROKER_THRESHOLD	0.62	Below this confidence, answers carry the broker flag.
HARMONY_HIGH_THRESHOLD	0.85	At or above this confidence, the chip reads High.
HARMONY_TOP_K_RULINGS	8	CROSS precedents retrieved per answer.
HARMONY_ALLOW_CLARIFY	true	Allow the agent one clarifying question.
HARMONY_THINKING_BUDGET	0	Gemini thinking tokens for the classifier. Ships at 0: answers land fast, and the graded loop measures whether the baseline needs more deliberation.
HARMONY_JUDGE_THINKING_BUDGET	0	Thinking tokens for the Gemini judge.
HARMONY_INTROSPECT_THINKING_BUDGET	1024	Thinking tokens for the introspection agent.
HARMONY_ROUND_CONCURRENCY	12	Parallel classifications during graded rounds.
HARMONY_CATALOG_CONCURRENCY	6	Parallel classifications in Catalog mode.
HARMONY_MEMORY_SIMILARITY	0.46	Minimum similarity for a graded miss to surface as the memory callout.
HARMONY_DATA_DIR / HARMONY_FRONTEND_DIR	./data, ./frontend	Asset locations (set by the Dockerfile).

Honesty contract (held by the code, not the copy)

• The UI shows only measured or sourced numbers. Before round 1 every stat reads "—" / "first round runs tonight".
• A cited ruling must have been returned by the retrieval step or it is dropped from the answer.
• An answered code must exist as a 10-digit statistical line in the 2026 schedule, or the answer is downgraded and says so.
• The memory callout only renders when the MCP read actually returned a relevant graded miss.
• The gate's refused prompt versions stay visible — in the lineage and in Phoenix prompt history.
• No red anywhere: the agent's mistakes are data it owns, not alarms.

Harmony is a classification aid, not customs advice.

License

MIT — see LICENSE.