OpenAid Provisioner

AI agent that turns humanitarian data requests into governed Fivetran pipelines—search HDX, plan connectors, get human approval, sync to BigQuery, deliver field briefings.

Comment

Inspiration

Crises do not wait for data engineers.

When floods hit Kenya or a disaster displaces thousands, field teams need answers fast: Where are medical supplies? Which districts are covered? What datasets can we trust today? Much of that data already lives on HDX, the UN humanitarian data exchange. The hard part is not finding data. It is getting it into a clean, governed pipeline that analysts and coordinators can actually use.

We kept hearing the same pain in different words: "Our dashboard is empty and we do not have time to build another connector." Humanitarian workers think in missions. Data platforms think in schemas, sync schedules, and connector configs. That gap costs hours, sometimes days, when minutes matter.

That gap is why we built OpenAid Provisioner.

We wanted an agent that does more than chat. It should understand a real-world request, search live humanitarian sources, check what is already synced in Fivetran, propose a pipeline plan, and pause for human approval before changing anything. For the Google Cloud Rapid Agent Hackathon, Fivetran track, that felt like the right mission: use Gemini as the brain, Fivetran MCP as the superpower, and keep humans in control.

What it does

OpenAid Provisioner turns a plain-language humanitarian request into a governed data pipeline plan.

An operator types something like:

"We need the latest medical supply data for flooded regions in Kenya, but our dashboard is empty."

From there, the agent:

  1. Interprets the mission with Gemini on Vertex AI (region, intent, HDX search terms)
  2. Searches HDX live for matching datasets and CSV/JSON resources
  3. Checks Fivetran for existing connection coverage and gaps
  4. Plans connector strategy, including Fivetran Connector SDK for HDX sources
  5. Stops for approval so no pipeline is created without a human sign-off
  6. Provisions the pipeline (demo simulation by default, live mode supported)
  7. Applies schema policy for safe column changes
  8. Delivers a field briefing coordinators can act on

Everything streams step by step to a Next.js dashboard, so operators see the agent think, search, plan, and wait for approval in real time.

In short: OpenAid is a humanitarian data provisioning copilot. It connects crisis language to real data infrastructure.

How we built it

We split the system into a brain, a superpower, and a control layer.

Brain: Gemini + Google ADK on Vertex AI

We use Google ADK with Gemini 2.5 Flash for mission interpretation and connector strategy. Reasoning runs through Vertex AI with Application Default Credentials, aligned with GCP hackathon expectations.

Superpower: Fivetran MCP + REST fallback

We integrated the official Fivetran MCP server over stdio using Google ADK's MCP session manager. The agent can list connections and connector metadata like a data engineer. If MCP hits an edge case, our gateway falls back to the Fivetran REST API so the mission still completes.

Discovery: HDX CKAN API

Dataset search is live against data.humdata.org, not mocked. The orchestrator ranks results and prefers CSV/JSON resources that fit fast provisioning.

Custom ingest: Fivetran Connector SDK

We built a Python Connector SDK connector that pulls HDX CSV/JSON URLs into a warehouse table. Local fivetran debug validated the flow with 24,696 records synced in testing.

Control layer: FastAPI orchestrator + Next.js UI

An 8-step mission pipeline runs in FastAPI and streams events over SSE. The frontend shows each step, surfaces the approval card with the full plan JSON, and renders the final briefing.

Safety by default

DEMO_MODE=true and FIVETRAN_ALLOW_WRITES=false keep demos safe. Live provisioning is ready when credentials and writes are enabled.

Challenges we ran into

GCP auth was stricter than expected

Our org disallowed API keys on Agent Platform, so we moved to ADC with gcloud auth application-default login. That was the right long-term setup, but it added setup friction early on.

Fivetran MCP was powerful but picky

MCP spawned correctly, but some tool calls failed on a schema_file validation requirement. We built a unified gateway so MCP is preferred and REST is the reliable fallback.

Connector testing failed at first for a simple reason

Our example HDX URL was a placeholder and returned 404. Once we swapped in a real CKAN download link, fivetran debug worked and synced thousands of rows.

Local debug is slow, and that is normal

Upserting tens of thousands of rows locally through the SDK tester takes minutes. It looks broken if you do not expect it, but it proved the connector logic is sound.

Balancing speed vs safety

For humanitarian use cases, autonomous writes felt wrong. We made approval a hard gate in the mission flow, not an optional nice-to-have.

Accomplishments that we're proud of

  • Built a real multi-step agent, not a chat wrapper. It plans, calls tools, pauses, and completes a mission.
  • Integrated Gemini on Vertex AI + Google ADK with a practical orchestration model.
  • Wired Fivetran partner MCP into the agent workflow with REST resilience.
  • Connected to the live HDX API for real humanitarian dataset discovery.
  • Shipped a working Connector SDK path for HDX CSV/JSON ingestion.
  • Designed a clean approval-first UX that fits NGO operations and hackathon judging criteria.
  • Published the full project as open source with phased commits and a judge-friendly README.

Most of all, we are proud that the demo tells a clear story: a coordinator asks for Kenya flood medical data, the agent does the heavy lifting, and a human still decides what gets provisioned.

What we learned

  • The best agent demos are workflow agents. Judges and users respond when they see tool calls, intermediate decisions, and a final artifact, not just a polished answer.
  • Partner MCP is a force multiplier. Fivetran MCP gave our agent credible data-engineering actions: inspect connections, inspect connector catalog, plan provisioning.
  • Human-in-the-loop is a feature, not a limitation. In crisis data systems, trust and governance matter as much as speed.
  • Hackathon quality comes from end-to-end credibility: real HDX calls, real Vertex auth, real Fivetran credentials, and a connector that actually syncs data in local testing.
  • Shipping in phases works. Foundation, reasoning, partner integration, UI, connector SDK, docs. That kept the project demo-ready while leaving room for live provisioning and BigQuery briefings next.

What's next for OpenAid Provisioner

Phase 3: Live provisioning end to end

Move from demo simulation to real create_connection calls through Fivetran MCP/REST, with paused connections by default.

Phase 4: Live BigQuery briefings

Query synced tables for row counts, freshness, and region coverage instead of template text.

Phase 5: Hosted deployment

Deploy backend to Cloud Run and frontend to Vercel so judges can try a public URL.

Phase 6: Smarter missions

Support multi-dataset provisioning in one approval flow, plus automatic Connector SDK deploy using the HDX resource URL selected during search.

Longer term

Schema drift alerts when HDX publishers change columns, role-based approver permissions for large NGOs, and reusable playbooks for recurring crisis types (floods, epidemics, displacement).

OpenAid started as a hackathon project. The vision is bigger: make humanitarian data infrastructure as fast as the crises it serves.

Built With

Share this project:

Updates