UCP Conformance Test Suite

This suite contains language-agnostic integration tests that run against a running UCP Merchant Server instance. It verifies the server's adherence to the Universal Commerce Protocol (UCP) specification.

Architecture

The test suite follows a 4-tier modular architecture per UCP RFC #520:

• framework/: Shared testing harness decoupled from any business vertical. Provides capability discovery, platform profile evaluation, mock webhook and agent servers, base test cases, and capability decorators.
• core/: Agnostic protocol tests (protocol_test, binding_test) that only need the discovery document. Can be run against any UCP server regardless of business vertical.
• common/: Cross-cutting protocol capabilities such as payment credentials and extensions (common/payments/). These currently use the shopping checkout as their vehicle and are gated on the capabilities they need.
• shopping/: Retail shopping domain tests organized by functional domain: checkout/ (including idempotency), discount/, fulfillment/, order/ (including order-event webhooks), and validation/.
• platforms/: Platform certification profiles (e.g. google.yaml) that mandate required capabilities and payment handlers.

Tests declare the capabilities they exercise with @requires_capability, and are skipped automatically when the server under test does not advertise them in its discovery profile. This is what allows the same suite to run against a server implementing any subset of UCP.

Testing a Custom Server (Agnostic)

The conformance tests are designed to be run against any UCP server implementation. To run tests against your own server, follow these steps:

1. Seed Your Server with Test Data

Your server must be populated with test data (products, inventory, shipping rates, discounts) that match the expectations defined in your test configuration.

You can use the default flower_shop test data as a reference for what data is needed.

2. Configure Conformance Input (conformance_input.json)

Create a conformance_input.json file to define your merchant-specific values and expectations. The tests use this file to know what to assert against.

Example conformance_input.json:

{
  "ucp_version": "2026-04-08",
  "required_capabilities": [
    "dev.ucp.shopping.checkout",
    "dev.ucp.shopping.order"
  ],
  "currency": "USD",
  "items": [
    {
      "id": "item_1",
      "title": "Valid Item",
      "price": 1000
    }
  ],
  "out_of_stock_item": {
    "id": "item_out_of_stock",
    "title": "Out of Stock Item"
  },
  "non_existent_item": {
    "id": "non_existent",
    "title": "Non-existent Item"
  }
}

• ucp_version: The spec version your server targets.
• required_capabilities: The capability names your server is expected to declare in discovery.
• items: List of items available on your server. The first item in this list will be used for standard checkout tests.
• out_of_stock_item: An item ID that should trigger an out-of-stock error (status 409) when trying to complete checkout.
• non_existent_item: An item ID that does not exist in your catalog, used to test invalid input handling.

3. Configure Test Fixtures (test_fixtures.json)

If your server's test data differs from the default, create a test_fixtures.json to map the exact values for calculations and assertions.

Example test_fixtures.json:

{
  "test_fixtures": {
    "valid_item": {
      "sku": "item_1",
      "expected_price": 10.0,
      "quantity": 1
    },
    "valid_discount_code": "PROMO10",
    "expected_discount_percentage": 10.0,
    "valid_discount_code_2": "PROMO20",
    "expected_discount_percentage_2": 20.0,
    "valid_fixed_discount_code": "FIXED5",
    "expected_fixed_discount_reduction": 5.0,
    "known_customer": {
      "full_name": "John Doe",
      "email": "[email protected]",
      "addresses": [
        {
          "street_address": "123 Main St",
          "address_locality": "Springfield",
          "address_region": "IL",
          "postal_code": "62704",
          "address_country": "US"
        }
      ]
    },
    "known_customer_without_address": {
      "full_name": "Jane Doe",
      "email": "[email protected]"
    },
    "free_shipping_min_subtotal": 100.0,
    "free_shipping_item_sku": "item_1",
    "dynamic_fulfillment": {
      "domestic": {
        "destination": {
          "street_address": "123 Main St",
          "address_locality": "Springfield",
          "address_region": "IL",
          "postal_code": "62704",
          "address_country": "US"
        },
        "expected_option_id": "exp-ship-us"
      },
      "international": {
        "destination": {
          "street_address": "25 King St W",
          "address_locality": "Toronto",
          "address_region": "ON",
          "postal_code": "M5V 2H1",
          "address_country": "CA"
        },
        "expected_option_id": "exp-ship-intl"
      }
    }
  },
  "shipping_locations": {
    "domestic_destination": {
      "street": "123 Main St",
      "city": "Anytown",
      "state": "CA",
      "postal_code": "12345",
      "country": "US"
    }
  }
}

• valid_item: The SKU and expected price (in major units, e.g., 10.00) of the item used in tests.
• Discounts Configuration:
  • valid_discount_code & expected_discount_percentage: Required for basic discount flow tests. The code must be valid on your server, and the percentage is expressed as a number from 0 to 100.
  • valid_discount_code_2 & expected_discount_percentage_2: Optional. Used for testing multiple discount codes applied sequentially. If your server does not support multiple discounts or you don't configure this, the corresponding test will be skipped.
  • valid_fixed_discount_code & expected_fixed_discount_reduction: Optional. Used for testing fixed-amount discounts. If your server only supports percentage discounts or you don't configure this, the test will be skipped.
• Fulfillment Configuration:
  • known_customer: Optional. A buyer your server recognizes by email and stores addresses for, with the stored address contents (response-schema field names). Used by the stored-address injection, selection, and reuse tests; each skips if this is not configured. Address ids are read from your server's responses, so only the contents must match.
  • known_customer_without_address: Optional. A buyer your server recognizes but has no stored addresses for. The corresponding test skips if not configured.
  • free_shipping_min_subtotal: Optional. Order subtotal (major units) at which your server offers a zero-cost fulfillment option. The threshold test skips if not configured.
  • free_shipping_item_sku: Optional. An item SKU eligible for free shipping regardless of order value. The item test skips if not configured.
  • dynamic_fulfillment: Optional. Destination/option pairs for the dynamic fulfillment test: each case declares a destination address (response-schema field names) and the option id your server returns for that destination. The test verifies that updating the fulfillment address to the domestic destination makes that destination's option available, and likewise for the international destination; it skips if not configured.
• shipping_locations: Addresses used for fulfillment tests.

4. Run the Tests

Install dependencies:

uv sync

Run the tests pointing to your server:

SERVER_URL=https://your-merchant-server.com \
SIMULATION_SECRET=your-sim-secret \
uv run pytest \
  --conformance_input=path/to/your/conformance_input.json \
  --fixture_config=path/to/your/test_fixtures.json

Alternatively, use the ucp-conformance runner, which understands the tiered layout and can select suites or apply a platform certification profile:

uv run ucp-conformance \
  --server_url=https://your-merchant-server.com \
  --simulation_secret=your-sim-secret \
  --conformance_input=path/to/your/conformance_input.json \
  --fixture_config=path/to/your/test_fixtures.json \
  --suite=all

--suite accepts core, common, shopping, all, or a comma-separated combination. Running only --suite=core is useful for a server that does not implement the shopping vertical. A single module, class or file path also works, to run one test module on its own:

uv run ucp-conformance --suite=shopping/discount/discount_test.py
uv run ucp-conformance --suite=shopping.checkout.lifecycle_test.CheckoutLifecycleTest

To validate against a platform's certification requirements, pass a profile from platforms/. The run then fails if the server does not advertise every mandated capability and payment handler:

uv run ucp-conformance \
  --server_url=https://your-merchant-server.com \
  --simulation_secret=your-sim-secret \
  --platform=google \
  --suite=all

Use -k/--filter to run a subset of tests by name, and --dry-run to list what would execute without contacting the server.

---

Example Walkthrough: Running against the Python Sample Server

For a quick demonstration or local development, you can run the suite against the reference Python sample server from the separate samples repository.

NOTE: These instructions assume the samples, python-sdk, and conformance repositories are sibling directories and commands are executed from the directory containing this README.

Prerequisites

Clone the sample server and SDK next to this repository, then sync both environments:

git clone https://github.com/Universal-Commerce-Protocol/samples.git ../samples
git clone https://github.com/Universal-Commerce-Protocol/python-sdk.git ../python-sdk
uv sync
uv sync --directory ../samples/rest/python/server/

1. Initialize the Sample Database

Populate the SQLite databases with the default "flower shop" test data:

DATABASE_PATH=/tmp/ucp_test
rm -rf ${DATABASE_PATH} && mkdir -p ${DATABASE_PATH}

uv run --directory ../samples/rest/python/server import_csv.py \
    --products_db_path=${DATABASE_PATH}/products.db \
    --transactions_db_path=${DATABASE_PATH}/transactions.db \
    --data_dir=../../../../conformance/shopping/fixtures/flower_shop

2. Start the Sample Server

Start the server in the background:

SIMULATION_SECRET=super-secret-sim-key
MERCHANT_SERVER_PORT=8182

uv run --directory ../samples/rest/python/server server.py \
    --products_db_path=${DATABASE_PATH}/products.db \
    --transactions_db_path=${DATABASE_PATH}/transactions.db \
    --port=${MERCHANT_SERVER_PORT} \
    --simulation_secret=${SIMULATION_SECRET} &
MERCHANT_SERVER_PID=$!

3. Run the Tests

Run the tests using the default flower shop configurations:

SERVER_URL=http://localhost:${MERCHANT_SERVER_PORT} \
SIMULATION_SECRET=${SIMULATION_SECRET} \
uv run pytest

4. Clean Up

Stop the background server:

kill ${MERCHANT_SERVER_PID}

Examining Database State (Optional)

After running tests, you can inspect the sample server's database state:

# Dump transactions table
uv run --directory ../samples/rest/python/server dump_transactions.py \
    --transactions_db_path=${DATABASE_PATH}/transactions.db

# Dump request logs
uv run --directory ../samples/rest/python/server dump_log.py \
    --transactions_db_path=${DATABASE_PATH}/transactions.db