feat!: add Anthropic Message Batches API by cdoern · Pull Request #5577 · ogx-ai/ogx

cdoern · 2026-04-16T19:25:11Z

Summary

Adds all 5 Anthropic Message Batches API endpoints under /v1/messages:

POST /v1/messages/batches — create a batch of message requests
GET /v1/messages/batches — list batches with pagination
GET /v1/messages/batches/{id} — retrieve batch status
POST /v1/messages/batches/{id}/cancel — cancel an in-progress batch
GET /v1/messages/batches/{id}/results — stream results as JSONL

The implementation follows the same architecture as the existing OpenAI Batches provider: KVStore persistence, background asyncio tasks with semaphore-based concurrency control, and reuse of the existing create_message() flow for processing individual batch requests.

Breaking change

MessagesConfig now requires a kvstore backend (KVStoreReference). Existing distributions are updated via codegen; out-of-tree configs will need to add this field.

Test plan

Integration tests added at tests/integration/messages/test_message_batches.py covering create / retrieve / list / cancel / results / duplicate-custom_id / 404, running under the messages-openai suite (already wired into tests/integration/ci_matrix.json)
Recordings committed against openai/gpt-4o so replay-mode CI works without an API key
Pre-commit checks pass (ruff, mypy, API spec, Anthropic coverage)
Unit tests pass
OpenAPI spec auto-regenerated with new endpoints
Distribution configs (ci-tests, starter) updated via codegen

Locally verified:

./scripts/integration-tests.sh \
  --stack-config server:ci-tests --setup gpt \
  --file tests/integration/messages/test_message_batches.py
# 6 passed in 1.76s

🤖 Generated with Claude Code

github-actions · 2026-04-16T19:25:51Z

✱ Stainless preview builds

This PR will update the llama-stack-client SDKs with the following commit message.

feat(messages): add Anthropic Message Batches API

⚠️

llama-stack-client-node studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️ → build ⏭️ → lint ⏭️ → test ✅

⚠️

llama-stack-client-openapi studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️

⚠️

llama-stack-client-go studio · conflict

Your SDK build had at least one warning diagnostic.

⚠️

llama-stack-client-python studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️ → build ⏭️ → lint ⏭️ → test ✅

This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
If you push custom code to the preview branch, re-run this workflow to update the comment.
Last updated: 2026-05-13 15:42:32 UTC

…ranslation (ogx-ai#5587) ## Summary - Adds a new `messages-openai` integration test suite that reuses `tests/integration/messages` with the `gpt` setup, so requests flow through `_anthropic_to_openai` / `_openai_to_anthropic` in `providers/inline/messages/impl.py` instead of the native passthrough branch that the existing `messages` (ollama-reasoning) suite hits. - Registers the suite in `tests/integration/ci_matrix.json` so replay-mode CI runs it, and adds a `gpt` / `messages-openai` row to `.github/workflows/record-integration-tests.yml` so future rerecord runs refresh it. - Includes 12 recordings captured against `openai/gpt-4o` so replay mode works out-of-the-box (no API key required in CI). This is a precursor to ogx-ai#5577 — the batches tests there need a suite that targets OpenAI, because batches go through `create_message()` which only exercises translation when the provider is not in `_NATIVE_MESSAGES_MODULES`. ## Test plan Recorded locally and verified against `openai/gpt-4o`: ```bash ./scripts/integration-tests.sh \ --stack-config server:ci-tests \ --suite messages-openai \ --inference-mode record-if-missing ``` Result: 13/13 tests pass. Recording URLs point at `https://api.openai.com/v1/chat/completions`, confirming the translation path is being exercised (not the Anthropic-native `/v1/messages` endpoint the ollama suite hits). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Charlie Doern <[email protected]>

cdoern · 2026-04-22T19:48:11Z

not sure why backwards compatibility test is failing, I added ! in.

Add the 5 Anthropic Message Batches endpoints under /v1/messages: - POST /v1/messages/batches (create batch) - GET /v1/messages/batches (list batches) - GET /v1/messages/batches/{id} (retrieve batch) - POST /v1/messages/batches/{id}/cancel (cancel batch) - GET /v1/messages/batches/{id}/results (stream JSONL results) The implementation follows the same pattern as the existing OpenAI Batches provider: KVStore for persistence, background asyncio tasks with semaphore-based concurrency control, and reuse of the existing create_message() flow for processing individual batch requests. The MessagesConfig now requires a kvstore backend, which is a breaking change to the provider configuration. Integration tests run under the messages-openai suite and cover create / retrieve / list / cancel / results / duplicate-custom_id / 404 paths against openai/gpt-4o, with recordings committed for replay-mode CI. Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Charlie Doern <[email protected]>

leseb

Looks good, AI caught this one too:

 Cancel during processing can misreport completed requests as canceled

  Results are only persisted once after asyncio.gather completes, but cancellation finalization depends on previously persisted results (existing_data). If cancellation happens mid-run, completed requests are not yet saved, so _finalize_batch_canceled marks all not-yet-recorded requests as canceled, losing true
  succeeded/errored outcomes and producing incorrect counts.

leseb · 2026-05-07T08:03:33Z

+        limit: int = 20,
+        before_id: str | None = None,
+        after_id: str | None = None,


use a ListMessageBatchesRequest

leseb · 2026-05-07T08:03:47Z

+
+    async def retrieve_message_batch_results(
+        self,
+        batch_id: str,


ditto for Request class

leseb · 2026-05-07T08:04:11Z

+
+    async def _process_message_batch(
+        self,
+        batch_id: str,


why isn't batch_id part of the request class?

leseb · 2026-05-07T08:04:18Z

+
+    async def _process_message_batch_impl(
+        self,
+        batch_id: str,


leseb · 2026-05-07T08:04:22Z

+
+    async def _finalize_batch_canceled(
+        self,
+        batch_id: str,


leseb · 2026-05-07T09:47:49Z

+    requests: list[MessageBatchRequestParams] = Field(
+        ...,
+        min_length=1,
+        max_length=100000,


isn't 100000 a little too much for asyncio, let's do 1000 instead?

leseb · 2026-05-07T09:50:02Z

+        if batch.processing_status == "ended":
+            raise ValueError(f"Cannot cancel batch '{batch_id}' that has already ended")
+
+        async with self._update_lock:


let's acquire the lock earlier so we avoid the read-check-write race? we need to get the batch in the lock too, then cancel the task after releasing the lock.

- Introduce request classes for list/retrieve/cancel/results endpoints and thread them through the Messages protocol, FastAPI routes, and inline impl. - Lower CreateMessageBatchRequest.requests max_length from 100000 to 1000 to keep batch size reasonable for asyncio scheduling. - Bundle batch_id with its original request via an internal _BatchContext so private helpers take a single argument. - Fix cancel_message_batch race by reading and updating the batch under the update lock, and cancelling the processing task only after the lock is released so the task can acquire it to finalize. - Track partial batch results in memory as each request completes so a mid-run cancellation finalizes with the true succeeded/errored counts rather than marking already-completed requests as canceled. Signed-off-by: Charlie Doern <[email protected]>

Signed-off-by: Charlie Doern <[email protected]>

The messages suite defaults to the ollama setup which uses ollama/llama3.2:3b-instruct-fp16, but the recorded HTTP responses for the batch tests only existed for ollama/gpt-oss:20b. The ollama suite batch tests therefore had no matching recording and fell through to the live container which returned 400. Add llama3.2-keyed recordings cloned from the existing gpt-oss responses so replay covers both parameter variants. Signed-off-by: Charlie Doern <[email protected]>

The messages suite was switched from ollama-reasoning (gpt-oss:20b) to ollama (llama3.2:3b-instruct-fp16) in ogx-ai#5746, so the gpt-oss-keyed batch recordings are never consulted in replay. Flagged by the unused-recordings check. Signed-off-by: Charlie Doern <[email protected]>

leseb

one miss?

leseb · 2026-05-13T09:11:27Z

+        limit: int = 20,
+        before_id: str | None = None,
+        after_id: str | None = None,


should be ListMessageBatchesRequest?

…ssageBatchesRequest Addresses review feedback on PR ogx-ai#5577: the GET /messages/batches route now takes a ListMessageBatchesRequest directly via Annotated[..., Query()] instead of unpacking the individual query parameters and reconstructing the request class inside the handler. Signed-off-by: Charlie Doern <[email protected]>

meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Apr 16, 2026

cdoern changed the title ~~feat(messages): add Anthropic Message Batches API~~ feat(messages)!: add Anthropic Message Batches API Apr 16, 2026

cdoern force-pushed the feat/messages-batches-api branch 2 times, most recently from 7f7488d to 0ed9397 Compare April 16, 2026 19:35

cdoern mentioned this pull request Apr 17, 2026

test(messages): add messages-openai suite covering Anthropic↔OpenAI translation #5587

Merged

ogx-ai deleted a comment from nidhishgajjar Apr 22, 2026

cdoern force-pushed the feat/messages-batches-api branch 4 times, most recently from 70091ac to 25dae2f Compare April 22, 2026 19:28

cdoern marked this pull request as ready for review April 22, 2026 19:47

cdoern requested review from ashwinb, bbrowning, ehhuang, franciscojavierarceo, leseb, mattf and raghotham as code owners April 22, 2026 19:47

cdoern force-pushed the feat/messages-batches-api branch from 25dae2f to f9f0570 Compare May 5, 2026 13:39

cdoern force-pushed the feat/messages-batches-api branch from f9f0570 to d63702b Compare May 5, 2026 14:11

cdoern changed the title ~~feat(messages)!: add Anthropic Message Batches API~~ feat!: add Anthropic Message Batches API May 5, 2026

Merge branch 'main' into feat/messages-batches-api

95741dc

leseb requested changes May 7, 2026

View reviewed changes

cdoern added 2 commits May 12, 2026 14:13

Merge branch 'main' into feat/messages-batches-api

d2fbba2

cdoern added 2 commits May 12, 2026 14:38

chore: apply openai-coverage drift from CI

4a31c13

Signed-off-by: Charlie Doern <[email protected]>

cdoern requested review from leseb and removed request for ashwinb and ehhuang May 12, 2026 18:43

leseb approved these changes May 13, 2026

View reviewed changes

leseb requested changes May 13, 2026

View reviewed changes

cdoern requested a review from leseb May 13, 2026 15:11

leseb approved these changes May 13, 2026

View reviewed changes

leseb enabled auto-merge May 13, 2026 15:18

leseb added this pull request to the merge queue May 13, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 13, 2026

leseb added this pull request to the merge queue May 13, 2026

Merged via the queue into ogx-ai:main with commit 479af78 May 13, 2026
66 checks passed

Uh oh!

Conversation

cdoern commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking change

Test plan

Uh oh!

github-actions Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✱ Stainless preview builds

Uh oh!

cdoern commented Apr 22, 2026

Uh oh!

leseb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leseb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cdoern commented Apr 16, 2026 •

edited

Loading

github-actions Bot commented Apr 16, 2026 •

edited

Loading