test(reasoning-effort-grid): cover Claude Opus 4.8 across provider routes#29327
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummaryThis test-only PR extends the
Confidence Score: 4/5Safe to merge — changes are confined to test infrastructure with no production code affected. The Bedrock Opus 4.8 entry sets bedrock_effort_ceiling="xhigh" while the structurally identical Opus 4.7 entry leaves that field unset. This has no clamping effect today, and both entries are xfail, so tests are unaffected. Still, the structural divergence between two entries described as capability-equivalent is worth resolving before either xfail guard is removed. tests/llm_translation/reasoning_effort_grid/grid_spec.py — the Bedrock Opus 4.7 and 4.8 entries differ in bedrock_effort_ceiling despite identical stated capabilities.
|
| Filename | Overview |
|---|---|
| tests/llm_translation/reasoning_effort_grid/grid_spec.py | Adds claude-opus-4-8 entries to four route tuples and renames _CAPS_OPUS_4_7 → _CAPS_XHIGH_MAX; Bedrock Opus 4.8 sets bedrock_effort_ceiling="xhigh" while the Bedrock Opus 4.7 entry (same capabilities) leaves the field unset — a minor inconsistency with no runtime impact since both are xfail. |
| tests/llm_translation/reasoning_effort_grid/test_reasoning_effort_grid.py | Updates cell-count assertion from 21×11 (231) to 25×11 (275) to account for the four new model entries; no other logic changes. |
Reviews (1): Last reviewed commit: "test(reasoning-effort-grid): add claude-..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| BEDROCK_CONVERSE_MODELS: Tuple[ModelEntry, ...] = ( | ||
| ModelEntry( | ||
| alias="bedrock-claude-opus-4-8", | ||
| model="bedrock/converse/us.anthropic.claude-opus-4-8", | ||
| mode="adaptive", | ||
| extra_params=(("aws_region_name", "us-east-1"),), | ||
| required_env=_BEDROCK_REQ, | ||
| caps=_CAPS_XHIGH_MAX, | ||
| bedrock_effort_ceiling="xhigh", | ||
| fail_reason=( | ||
| "claude-opus-4-8 is not entitled on the Bedrock CI account " | ||
| "941277531214 (model access requires an AWS Sales request, not " | ||
| "self-serve); this cell fails on purpose so it stays loud in CI — " | ||
| "remove this fail_reason once access is granted" | ||
| ), | ||
| ), |
There was a problem hiding this comment.
bedrock_effort_ceiling between Opus 4.7 and 4.8
The new Bedrock Opus 4.8 entry sets bedrock_effort_ceiling="xhigh", but the Bedrock Opus 4.7 entry (lines 279–292) leaves the field unset — even though the PR description states they share the exact same Bedrock effort tier. Because _EFFORT_RANK["xhigh"] = 4 is the highest rank in the spec (all other adaptive efforts rank ≤ 3), setting the ceiling to "xhigh" has no clamping effect today; both entries produce identical CellExpectation values. The inconsistency is harmless at runtime (both are xfail), but a future maintainer removing the Opus 4.7 fail_reason would see the two entries differ structurally with no explanation. Consider either adding bedrock_effort_ceiling="xhigh" to the Opus 4.7 entry for symmetry, or adding an inline comment explaining why they differ.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
|
@mateo-berri — could you add a screenshot or short video showing the new reasoning-effort-grid tests running green locally (or pasted output of the test run for Claude Opus 4.8 across providers)? Visual proof really speeds up review. Thanks! |
|
Here is the local run for the new Claude Opus 4.8 grid cells. The Anthropic direct route is exercised against the real Anthropic API (all 11 effort tiers, real spend): The Azure, Vertex, and Bedrock Opus 4.8 cells are guarded with xfail until the model is provisioned on those CI accounts (the Azure Foundry resource returns DeploymentNotFound for claude-opus-4-8 today, Bedrock needs an entitlement request, and Vertex availability is unconfirmed). Each guard has a comment telling maintainers to remove it once the deployment or entitlement lands. Here are the Azure cells xfailing cleanly alongside the structural grid assertions, so nothing goes red: The two passing tests there are test_grid_cell_count (now asserting 25x11 = 275 cells) and test_grid_route_coverage. CircleCI llm_translation_testing on this PR runs the full grid the same way. |
…edaction PR #28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped.
mateo-berri
force-pushed
the
litellm_opus_4_8_effort_grid
branch
from
90eb34d to
15747ad
Compare
mateo-berri
changed the base branch from
litellm_internal_staging
to
claude/focused-allen-QifiT
mateo-berri
changed the base branch from
claude/focused-allen-QifiT
to
litellm_fix_staging_ci
The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions.
mateo-berri
force-pushed
the
litellm_fix_staging_ci
branch
from
96dc86c to
1015f9e
Compare
Adds claude-opus-4-8 to the anthropic, azure, vertex and bedrock-converse routes (275 cells total) so the reasoning-effort wire translation is covered for the new model. The bedrock opus-4-8 and opus-4-7 cells reuse the self-heal path: they run the call and skip only on Bedrock's "is not available for this account" reply, then assert in full once the model is entitled. The azure and vertex opus-4-8 cells stay xfail until a Foundry deployment exists and Vertex availability is confirmed. The shared xhigh+max capability set is renamed to _CAPS_XHIGH_MAX now that more than one model uses it.
mateo-berri
force-pushed
the
litellm_opus_4_8_effort_grid
branch
from
15747ad to
5ce9259
Compare
mateo-berri
changed the base branch from
litellm_fix_staging_ci
to
litellm_internal_staging
…itellm_opus_4_8_effort_grid # Conflicts: # tests/llm_translation/reasoning_effort_grid/grid_spec.py
mateo-berri
merged commit 152b117
into
litellm_internal_staging
…utes (BerriAI#29327) * test(logging): align DB metrics event_metadata assertions with safe redaction PR BerriAI#28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions. * test(reasoning-effort-grid): add claude-opus-4-8 across provider routes Adds claude-opus-4-8 to the anthropic, azure, vertex and bedrock-converse routes (275 cells total) so the reasoning-effort wire translation is covered for the new model. The bedrock opus-4-8 and opus-4-7 cells reuse the self-heal path: they run the call and skip only on Bedrock's "is not available for this account" reply, then assert in full once the model is entitled. The azure and vertex opus-4-8 cells stay xfail until a Foundry deployment exists and Vertex availability is confirmed. The shared xhigh+max capability set is renamed to _CAPS_XHIGH_MAX now that more than one model uses it.
3 participants
Relevant issues
Claude Opus 4.8 shipped with cost-map entries (Anthropic, Bedrock converse and regional, Vertex, Azure AI) and a dedicated config regression test (
tests/test_litellm/test_claude_opus_4_8_config.py), but the real-API reasoning effort grid that runs in thellm_translation_testingCircleCI job still topped out at Opus 4.7. This adds Opus 4.8 to that grid so the new flagship'sreasoning_effortto wire-body translation is exercised the same way every prior generation is.Linear ticket
Pre-Submission checklist
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewChanges
Opus 4.8 was added as the newest flagship to the four routes that track the flagship Opus (anthropic_direct, azure_ai, vertex_ai, bedrock_converse), keeping the existing 4.7 and 4.6 entries. The Bedrock invoke routes were left untouched since they do not track the flagship (they carry no 4.7 entry either).
Opus 4.8 shares Opus 4.7's effort tier exactly: adaptive thinking with native
xhighandmaxsupport. The shared capability constant was renamed from_CAPS_OPUS_4_7to_CAPS_XHIGH_MAXbecause it now spans two generations and a single model-version name was misleading. On Bedrock the new cell carriesbedrock_effort_ceiling="xhigh", matching the model'sbedrock_output_config_effort_ceilingin the cost map.The grid cell-count assertion moved from 21x11 to 25x11 (231 to 275 cells) to reflect the four new model combinations.
Availability guards: the Anthropic direct cells run live. The Bedrock Opus 4.8 cell reuses the self-heal path introduced in the base PR #29344: it runs the call and skips only when Bedrock replies "is not available for this account" (the new model needs an AWS Sales entitlement request), and the moment access lands it asserts the wire body in full with no edit. The Azure and Vertex Opus 4.8 cells stay on an xfail
fail_reason: the Microsoft Foundry resource returnsDeploymentNotFounduntil someone creates theclaude-opus-4-8deployment, and Vertex availability on the CI project is not yet confirmed for this brand-new release. Eachfail_reasondocuments the condition to remove it, and becausexfail_strictis not set these guards never flip CI red.Screenshots / Proof of Fix
The new Anthropic direct Opus 4.8 cells were run against the real Anthropic API (all eleven efforts, real spend):
The Azure Opus 4.8 cells correctly xfail (rather than going red) while the deployment is missing, and the structural assertions pass:
Type
✅ Test