Bhashini Voice-to-Action Gateway

A multilingual voice-to-action gateway for rural Indian commerce. It takes a (possibly code-mixed Hindi/English "Hinglish") voice command like "Maa ko 500 rupaye bhej do" and turns it into a structured, validated, explicitly confirmed database action — a money transfer, a fertilizer order, a balance check — designed for users who can't navigate a conventional app UI.

This is an AI engineering project demonstrating production-grade backend patterns for a real, large-scale Indian problem: millions of Tier 2/3 users are locked out of digital commerce by language and literacy barriers, not by lack of a smartphone.

voice transcript ─▶ ASR ─▶ LangGraph intent router ─▶ Pydantic-validated
                                                          intent
                                                            │
                                          ┌─────────────────┴─────────────────┐
                                          ▼                                   ▼
                                 missing a slot?                     all slots present
                                          │                                   │
                                   ask follow-up                  issue confirmation_token
                                  (same request)                  + TTS-style confirm prompt
                                                                              │
                                                              ── separate HTTP request ──
                                                                              │
                                                                     user says yes/no
                                                                              │
                                                              ONLY this path can reach
                                                              the wallet/DB action layer

Implementation Status (Honest Architecture Seam)

Component	Classification	Implementation Status & Details
FastAPI Async API	IMPLEMENTED	Endpoints POST /v1/voice/command, POST /v1/voice/confirm, GET /health, GET /v1/wallet/{phone}/balance.
LangGraph Intent Router	IMPLEMENTED	Dynamic state machine (app/graph/intent_graph.py) enforcing slot-filling and human-in-the-loop confirmation.
Pydantic Strict Validation	IMPLEMENTED	ParsedIntent enforces type safety, bounds checks (e.g. max_transfer_inr), and confidence thresholds.
Transactional Core & Locking	IMPLEMENTED	Real PostgreSQL SELECT ... FOR UPDATE row locking + in-process asyncio.Lock for SQLite demo; unique idempotency_key prevents duplicate execution from replayed confirmation tokens.
Fertilizer Order Persistence	IMPLEMENTED (LOCAL DEMO RECORD)	Confirmed orders are written to a fertilizer_orders table with the same unique-idempotency-key replay protection as money movement. This is a local record only — there is no real supplier, pricing, inventory, or fulfilment integration.
Bhashini ASR Client	IMPLEMENTED	Fully implemented BhashiniASRProvider in app/providers/asr.py conforming to Government of India Dhruva ULCA pipeline inference schema (POST /services/inference/pipeline) with retry policies, timeouts, and language normalization.
ASR Mock / Offline Fallback	MOCK / OFFLINE	MockASRProvider in app/providers/asr.py provides deterministic offline testing with canned and plain-text audio passthrough.
Rule-Based NLU Fallback	MOCK / OFFLINE	RuleBasedIntentExtractor enables full pipeline execution with zero API keys.
Live Bhashini Inference	CONFIGURATION-DEPENDENT	Requires setting BHASHINI_API_KEY and BHASHINI_USER_ID (issued by bhashini.gov.in).
Claude Sonnet NLU	CONFIGURATION-DEPENDENT	Active when ANTHROPIC_API_KEY is provided (app/providers/intent_extractor.py); default model claude-sonnet-4-6.
NMT / TTS	NOT IMPLEMENTED IN MVP	Confirmation prompts are currently text-rendered; audio playback is on roadmap.

Architecture

┌────────────────────────────────────────────────────────────────────────┐
│                     CLIENT (mobile app / IVR / USSD)                    │
│   Records voice note → uploads (chunked/resumable in production)       │
└──────────────────────────────┬───────────────────────────────────────────┘
                                ▼
┌────────────────────────────────────────────────────────────────────────┐
│                POST /v1/voice/command  (FastAPI, async)                 │
│   1. ASR provider (Bhashini or Mock) → transcript + detected language  │
│   2. LangGraph intent pipeline:                                        │
│        extract_intent → check_slots → ask_followup | prepare_confirm   │
│                                       (cycle back to extract on retry)  │
│   3. Pydantic ParsedIntent validates EVERY field before it's trusted   │
└──────────────────────────────┬───────────────────────────────────────────┘
                                ▼
              needs_followup            needs_confirmation
              (ask user, same             (persist PendingIntent,
               conversation)               return confirmation_token)
                                                    │
                                     ── separate request ──
                                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│              POST /v1/voice/confirm  (only reachable path to money)     │
│   - Loads PendingIntent by token, checks not already resolved          │
│   - Executes against Postgres/SQLite with row-level locking +          │
│     idempotency key so retries never double-execute                    │
└────────────────────────────────────────────────────────────────────────┘

The hard boundary: there is no code path from extract_intent directly to the action executor within a single request. A misparsed intent — and ASR word-error-rates on noisy rural audio are realistically 15–30% — cannot move money without a human explicitly confirming a spoken-back summary.

Quickstart (zero external services)

git clone https://github.com/rishav579/bhashini-voice-gateway.git
cd bhashini-voice-gateway
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env   # defaults work offline: SQLite + fakeredis + rule-based NLU

uvicorn app.main:app --reload

Two demo users are auto-seeded on first run: a self-account (+919900011111, ₹2000 balance) with "Maa" already saved as a contact (+919900022222).

Try it

# 1. Send a voice command (transcript_override bypasses audio recording for demo/testing)
curl -X POST http://127.0.0.1:8000/v1/voice/command \
  -F "session_id=demo-1" \
  -F "phone_number=+919900011111" \
  -F "transcript_override=Maa ko 500 rupaye bhej do"

# → { "status": "awaiting_confirmation", "confirmation_token": "...", ... }

# 2. Confirm it (this is the ONLY call that can touch the wallet)
curl -X POST http://127.0.0.1:8000/v1/voice/confirm \
  -H "Content-Type: application/json" \
  -d '{"confirmation_token": "<paste token>", "confirmed": true}'

# → { "status": "completed", "transaction_id": "...", "new_balance_inr": 1500.0 }

Or run the scripted walkthrough:

python demo/demo_cli.py

Frontend (React dashboard)

The repository includes a full-stack demo UI in frontend/ — React 18 + TypeScript + Vite, with only react/react-dom as runtime dependencies. It talks to the same FastAPI backend over plain HTTP; nothing is mocked between them.

What it shows:

• A demo wallet header (persona "Ramesh") whose balance always comes from the live API.
• A large command bar with scenario chips, plus an optional browser dictation preview.
• A seven-stage pipeline view (Command → ASR → Intent → Guardrails → Confirm → Action → Result) driven by real API responses.
• A structured parsed-intent panel: intent, entities, confidence, raw transcript, detected language.
• An explicit confirmation card showing recipient/item and balance-before → after before anything executes.
• Distinct result states for completed / cancelled / rejected / follow-up / failed actions.
• A terminal-style event log of every pipeline step.

Frontend quickstart (backend running on port 8000):

cd frontend
cp .env.example .env   # sets VITE_API_BASE_URL=http://127.0.0.1:8000
npm install
npm run dev            # http://localhost:5173

VITE_API_BASE_URL is the only configuration knob — no environment is hard-coded into application logic. For a production build: npm run build (strict tsc + Vite bundle).

Honest scope notes for the frontend:

• The browser mic is a dictation preview, not the production speech path. It uses the Web Speech API where available (Chrome/Edge) and is labeled as a preview in the UI. Production audio would go to the Bhashini ASR provider, which remains configuration-dependent until credentials exist. Typed commands exercise the identical backend pipeline.
• The wallet runs on SQLite locally (demo.db, gitignored); the docker-compose setup swaps in Postgres for production-grade locking.
• This is not a production payment or commerce integration — no KYC, no UPI/bank rails, no supplier network. It demonstrates the engineering seams where those integrations would attach.

For screenshots or a short demo video: start both servers, then walk one scenario chip end-to-end (command → confirmation card → completed receipt), then show the over-limit chip being rejected by guardrails. Confirmation tokens are intentionally truncated in the UI; keep any full tokens out of frame.

Enabling Real Bhashini ASR & Claude NLU

To switch from offline mocks to live services, set your environment variables:

# Live Bhashini ASR (Government of India ULCA Inference)
export BHASHINI_API_KEY="your-bhashini-api-key"
export BHASHINI_USER_ID="your-bhashini-user-id"

# Real LLM-based intent extraction via Claude Sonnet (default model: claude-sonnet-4-6)
export ANTHROPIC_API_KEY="sk-ant-..."

uvicorn app.main:app --reload

Running the Tests

pytest -v

24 automated tests cover:

• Full happy path (command → confirmation → executed transfer with balance deduction)
• LangGraph slot-filling follow-ups and error loops
• User cancellation / declined confirmations
• Replay protection (re-executing a token returns 409)
• Balance bounds, overdraft rejections, and absurd amount constraints
• Fertilizer-order persistence: confirmed orders land in fertilizer_orders as explicit local-demo records, and a replayed idempotency key returns the original order instead of duplicating it
• Bhashini ULCA request serialization, language normalization, timeout handling, HTTP 4xx/5xx handling, and retry recovery
• Controlled upstream-failure isolation: live Bhashini errors return a clean 502/503 envelope (never a raw 500, no leaked credentials)

The React frontend has no automated tests yet; its verification today is TypeScript's strict compiler (tsc), the Vite production build, and manual end-to-end runs against the live API.

Production Deployment (Postgres + Redis)

docker compose up --build

This swaps DATABASE_URL to Postgres (enabling real SELECT ... FOR UPDATE row locking in app/action_executor.py). Idempotency does NOT depend on Redis — it is enforced by the database's unique transactions.idempotency_key constraint. The compose file also scaffolds a Redis instance for future multi-process session state; the current request pipeline does not read or write Redis.

Security & Authentication Limitations

This is a portfolio/demo system, and the boundaries below are deliberate scope decisions — they must be hardened before any real deployment:

• No authentication layer exists. There is no signup, login, JWT, or session handling anywhere in the codebase.
• Phone number = identity. Any caller can act as any phone number; unknown numbers are auto-provisioned with a seeded ₹2000 demo wallet so the API stays curl-able.
• Confirmation tokens are bearer-style. Whoever holds the token can execute the action — tokens are not bound to a user session or device.
• Confirmation tokens have no TTL. A pending intent remains executable until it is resolved.
• transcript_override is a deliberate demo/testing hook that bypasses audio recording entirely (documented in the request schema). In production this field would be removed so only real audio reaches the ASR stage.
• CORS currently allows wildcard origins for local/demo convenience (app/main.py). A real deployment must restrict this to the actual frontend origin.

None of these are hidden: they are the honest boundary of what this project currently is.

Project Structure

app/
├── main.py                    # FastAPI app + startup/demo seed
├── config.py                  # env-driven settings
├── schemas.py                 # strict Pydantic contracts (the safety layer)
├── db.py / db_models.py       # SQLAlchemy async engine + ORM models
├── redis_client.py            # Redis/fakeredis factory (scaffolded for future multi-process use; not on the request path)
├── user_service.py            # demo user/wallet/contact provisioning
├── action_executor.py         # money-moving logic, locking, idempotency
├── providers/
│   ├── asr.py                 # BhashiniASRProvider + MockASRProvider + factory
│   └── intent_extractor.py    # Claude-backed + rule-based NLU
├── graph/
│   └── intent_graph.py        # LangGraph state machine
└── routers/
    └── voice.py               # the two-endpoint API surface
tests/
├── conftest.py                    # pytest setup and fixtures
├── test_end_to_end.py             # e2e tests against FastAPI + DB
├── test_asr_upstream_error.py     # controlled upstream-failure envelope
├── test_bhashini_provider.py      # Bhashini ASR client & factory
└── test_fertilizer_persistence.py # order persistence + replay idempotency
demo/
└── demo_cli.py                    # scripted terminal walkthrough
frontend/                           # React 18 + TypeScript + Vite dashboard (see Frontend section)
├── src/api/                       # typed API client; base URL from VITE_API_BASE_URL
├── src/components/                # wallet header, command bar, pipeline, intent, confirmation, results, event log
├── src/hooks/                     # gateway state machine + dictation preview
└── src/styles.css                 # hand-rolled design system (no UI framework)