You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b07f181
Browse filesBrowse the repository at this point in the historyBrowse files
authored
feat(providers): adopt GPT-6.1 Sol and Claude Opus 5.5 defaults with compatible structured output (#806)
* feat(providers): support Claude Opus and Sonnet 5.5 structured output on native routes
Claude Opus 5.5 and Sonnet 5.5 answer a forced tool call with HTTP 400 and reject sampling controls. Add both to the shared forced-tool table and a new SAMPLING_REJECTED_MODELS table, so the anthropic and anthropic_proxy providers request the native JSON-schema response format and the bedrock provider binds the schema with toolChoice auto.
Add claude_model_name(), which reads the bare Claude name from bare, namespaced, prefixed, dotted, dated and Bedrock identifiers, and reject_unsupported_controls(), which fails before any request when SKILLSPECTOR_TEMPERATURE is set for a model that rejects it. anthropic, anthropic_proxy and bedrock call it after credentials resolve, so a missing key still returns None and the OpenAI fallback is unchanged.
Register both models for anthropic and anthropic_proxy (1,000,000 context, 128,000 output, json_schema) and the AWS-documented Bedrock profiles as explicit opt-in entries with toolChoice auto: Opus 5.5 on us/eu/au/jp/global and Sonnet 5.5 on us/eu/global. No default changes.
Bind the live OpenAI and Anthropic provider tests through bind_structured_output with room for reasoning tokens, and pin that a refused structured response fails the semantic analyzer instead of reading as a clean scan.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* feat(providers): bind gateway Claude 5.5 IDs without a forced tool call
Gateways namespace or prefix Claude model names (azure/anthropic/claude-opus-5-5, aws/anthropic/bedrock-claude-opus-5-5), so the bare-name rule missed them and LangChain forced a tool call that Claude Opus and Sonnet 5.5 reject with HTTP 400. OpenAI-compatible gateways can also turn a json_schema response format for Claude into a forced tool call.
Read the Claude name with claude_model_name() on every route:
- anthropic and anthropic_proxy request the native JSON-schema format for gateway IDs too (registry entries still win).
- openai gains forced_tool_choice_supported() and structured_output_method(): for those Claude IDs it disables tool_choice and binds the schema as an unforced tool call that the prompt asks for, retrying a prose answer. Other models, including gpt-6.1-sol and earlier Claude IDs, keep LangChain's default.
- openai_compatible applies the same name rule when the registry declares no tool_choice.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* feat(providers): reject controls GPT-6.1 Sol refuses before any request
GPT-6.1 Sol rejects an explicit temperature and the reasoning efforts none
and minimal with HTTP 400, and LangChain only strips temperature for the
gpt-5 family, so SkillSpector would send them and fail mid-scan.
reject_unsupported_controls() now also matches gpt-6.1-sol, bare or behind
a gateway namespace or prefix, with an optional snapshot suffix. Any
explicit SKILLSPECTOR_TEMPERATURE (1.0 included) and any
SKILLSPECTOR_REASONING_EFFORT other than low, medium, high, xhigh or max
raise ValueError with the setting to change. The shared OpenAI-protocol
builder calls it after credentials resolve, so openai, openai_compatible
and the other builder users get the Sol rules and the Claude 5.5
temperature rule; a missing key still returns None. Other models keep
passing both controls through.
SKILLSPECTOR_SEED is still forwarded to Sol and recorded as requested and
retained; the docs call it best-effort.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* fix(providers): repair the Bedrock default to us.anthropic.claude-sonnet-4-6 with a 64K cap
The Bedrock default us.anthropic.claude-sonnet-4-6-20250915-v1:0 is not a
catalogued AWS ID, and its registry entry allowed 128K output tokens while
AWS caps Claude Sonnet 4.6 at 64K on Bedrock.
BEDROCK_DEFAULT_MODEL becomes the suffix-free cross-region profile ID
us.anthropic.claude-sonnet-4-6 that the AWS model card lists, and the
registry key moves with it, capped at 64,000 output tokens. A test pins
that no Claude 5.5 profile is the default; those stay explicit opt-in.
The dated Opus 4.6 and 4.5 keys are unchanged.
A custom SKILLSPECTOR_MODEL_REGISTRY file replaces the bundled registry,
so it has to carry the new key to keep the default's budgets.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* feat(providers): default OpenAI to gpt-6.1-sol
The openai provider defaults every model slot to gpt-6.1-sol instead of
gpt-5.4. The bundled registry already carries its 1,050,000-token context
and 128,000-token output budgets, and it binds structured output through a
strict, tool-free json_schema response format.
The OPENAI_API_KEY credential fallback uses the openai defaults, so it now
runs gpt-6.1-sol for any slot without a model override. An explicit
SKILLSPECTOR_TEMPERATURE, or a reasoning effort other than low, medium,
high, xhigh or max, now fails before the first request on these paths
instead of being sent. Callers who need the previous model set
SKILLSPECTOR_MODEL=gpt-5.4.
The README provider table and the OPENAI_API_KEY row are updated. The row
also records that Claude Opus or Sonnet 5.5 gateway IDs are not supported
through the fallback, which keeps the active provider's structured-output
method.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* feat(providers): default native Anthropic to Opus 5.5 analyzers and Sonnet 5.5 meta
The anthropic provider defaults its analyzer slots to claude-opus-5-5
instead of claude-opus-4-6, and keeps a cheaper meta_analyzer pass with
claude-sonnet-5-5 instead of claude-sonnet-4-6. Both are registered with
1,000,000-token context and 128,000-token output budgets and bind
structured output through the native JSON-schema format, since they
reject a forced tool call.
Effort stays unset, so the API defaults apply: medium for Opus 5.5 and
high for Sonnet 5.5. SKILLSPECTOR_REASONING_EFFORT applies to every slot,
meta included. An explicit SKILLSPECTOR_TEMPERATURE now fails before the
first request with the default models. Slot precedence is unchanged:
SKILLSPECTOR_MODEL replaces the meta default too, and
SKILLSPECTOR_MODEL_META_ANALYZER wins over both. Callers who need the
previous models set SKILLSPECTOR_MODEL=claude-opus-4-6 and
SKILLSPECTOR_MODEL_META_ANALYZER=claude-sonnet-4-6. anthropic_proxy keeps
its default.
The README provider table and effort row are updated, and a
SKILLSPECTOR_MODEL_<SLOT> row documents the per-slot override. A test
pins the meta default against both overrides.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* refactor(providers): read Claude model names through one normalizer on every route
rejects_forced_tool_call() and rejects_sampling_controls() now normalize the
model ID themselves through claude_model_name(), so callers pass the raw ID
instead of wrapping it at each of the six call sites.
Bedrock's forced_tool_choice_supported() uses the same rule, which makes
claude_model_from_bedrock_id() dead; it is removed and its Bedrock ID cases
now pin claude_model_name(). Every Bedrock ID the old parser recognised
still resolves, and the registry tool_choice entry still wins first.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* fix(providers): apply the control guard on every hosted provider
Resolve sampling parameters for a model: resolve_sampling_parameters now takes the model and runs reject_unsupported_controls itself, so azure_openai (whose registry lists gpt-6.1-sol) gets the temperature guard the other providers already had, and no caller can skip it. The guard raises UnsupportedControlError, a ValueError subclass.
GPT-6.1 Sol IDs are read through the same model_name normalizer as Claude IDs (last path segment, lowercased, cut at '@' or ':'), so gpt-6.1-sol:latest, openai/gpt-6.1-sol:nitro and gpt-6.1-sol@2026 are guarded too.
A registry entry can declare 'sampling: rejected'. Bedrock reads it, so an application-inference-profile ARN that serves Claude 5.5 can opt in to the temperature guard, as it can already opt out of forced tool calls with 'tool_choice: auto'.
The registry-then-name forced tool_choice decision moves into structured_output.forced_tool_choice_supported, and the openai, openai_compatible and bedrock providers delegate to it. create_openai_compatible_chat_model takes forced_tool_choice and builds the disabled_params dict that bind_structured_output looks for in one place.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* fix(cli): stop a scan before analysis when a model rejects a requested control
A SKILLSPECTOR_TEMPERATURE (or a GPT-6.1 Sol effort of none/minimal) that the configured model rejects used to surface only while each analyzer built its model: the scan logged one traceback per semantic analyzer, exited 2, and the report showed llm_available: true with a generic telemetry error.
unsupported_control_error builds each distinct slot model the way the analyzers do and returns the UnsupportedControlError text; it does nothing when neither control is set, and leaves other failures, such as missing credentials, to the availability path. The scan and baseline commands call it first and exit 2 with that message. is_llm_available also returns it, so the MCP server and the report carry the reason in llm_error.
The docs now say that the setting stops the scan on every hosted provider, that it worked with the previous gpt-5.4 and claude-opus-4-6 defaults, and how to declare 'sampling: rejected' for a Bedrock application-inference-profile ARN.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* test(semantic): name the OpenAI refusal test after the failed status it pins
test_openai_refusal_is_incomplete_not_clean asserts status == 'failed'; rename it to test_openai_refusal_is_failed_not_clean.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* docs: say an unsupported Sol reasoning effort stops the scan
The README already says an unsupported SKILLSPECTOR_REASONING_EFFORT
for gpt-6.1-sol stops a scan before analysis with exit code 2. Use the
same wording in .env.example and docs/DEVELOPMENT.md.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* fix(providers): address model compatibility and preflight review
Validate only active LLM model slots, keep static CLI and MCP scans out of provider credential initialization, accept native Claude temperature 1.0, and preserve malformed model-registry fallback. Add request, provenance, and credential-process regressions.
Signed-off-by: Narendran Raghavan <[email protected]>
---------
Signed-off-by: Narendran Raghavan <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|`GOOGLE_CLOUD_LOCATION`| Google Cloud location for the `gemini` provider endpoint (e.g. `global`, `us`, `eu`, `us-central1`). Defaults to `global`. | Optional (used when `SKILLSPECTOR_PROVIDER=gemini`) |
725
733
|`GOOGLE_APPLICATION_CREDENTIALS`| Optional path to ADC credential/config file (e.g. Workload or Workforce Identity Federation config; exported service account keys are discouraged). For local development, use `gcloud auth application-default login`; for GKE, use Workload Identity. | Optional (used when `SKILLSPECTOR_PROVIDER=gemini`) |
726
734
|`NVIDIA_INFERENCE_KEY`| Credential for the `nv_build` provider (build.nvidia.com). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=nv_build`|
727
-
|`OPENAI_API_KEY`| Credential for the OpenAI provider (`SKILLSPECTOR_PROVIDER=openai`). Also serves as the tier-2 fallback in the credential waterfall when the active provider returns no credentials. | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=openai`|
735
+
|`OPENAI_API_KEY`| Credential for the OpenAI provider (`SKILLSPECTOR_PROVIDER=openai`). Also serves as the tier-2 fallback in the credential waterfall when the active provider returns no credentials; the fallback then uses the `openai` default model (`gpt-6.1-sol`) for any slot without a model override. Claude Opus or Sonnet 5.5 gateway IDs are not supported through the fallback, because it keeps the active provider's structured-output method; select `SKILLSPECTOR_PROVIDER=openai` for them. | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=openai`|
728
736
|`OPENAI_BASE_URL`| Override the OpenAI endpoint (e.g. point at Ollama). | Optional |
729
-
|`SKILLSPECTOR_REASONING_EFFORT`| Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged. When unset or blank, SkillSpector sends `high` for `nv_build` with `z-ai/glm-5.3`; other provider/model combinations keep their endpoint defaults. | Optional |
737
+
|`SKILLSPECTOR_REASONING_EFFORT`| Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged. When unset or blank, SkillSpector sends `high` for `nv_build` with `z-ai/glm-5.3`; other provider/model combinations keep their endpoint defaults (`medium` for `gpt-6.1-sol` and `claude-opus-5-5`, `high` for `claude-sonnet-5-5`). The value applies to every model slot, so the `anthropic``meta_analyzer` default receives the same effort as the analyzers. `gpt-6.1-sol` accepts only `low`, `medium`, `high`, `xhigh`, or `max`; any other value stops a scan before analysis (exit code 2). | Optional |
730
738
|`SKILLSPECTOR_OUTPUT_LANGUAGE`| Short, single-line language label (letters, numbers, spaces, `_`, or `-`; maximum 64 characters) for human-readable LLM finding text such as messages, explanations, and remediation. Rule IDs, severity values, paths, code, and other machine-readable values remain unchanged. Unset, blank, or invalid values preserve the default output language. | Optional |
731
-
|`SKILLSPECTOR_TEMPERATURE`| Optional sampling temperature from `0` to `1` for hosted providers. Unset or blank preserves the provider default. Lower values can reduce run-to-run variation but do not guarantee identical output. | Optional |
732
-
|`SKILLSPECTOR_SEED`| Optional integer sampling seed for OpenAI-compatible and Azure OpenAI providers. Other hosted providers and CLI providers do not receive it. Provider support remains model-dependent. | Optional |
739
+
|`SKILLSPECTOR_TEMPERATURE`| Optional sampling temperature from `0` to `1` for hosted providers. Unset or blank preserves the provider default. Lower values can reduce run-to-run variation but do not guarantee identical output. `gpt-6.1-sol` rejects every explicit temperature, including `1`. `claude-opus-5-5` and `claude-sonnet-5-5` accept only `1.0` on the native Anthropic endpoint ([API compatibility rule](https://platform.claude.com/docs/en/api/messages/create)); gateways, proxies, and Bedrock still reject every explicit value for these models. Unsupported settings on active LLM slots stop a scan before analysis (exit code 2). Static scans (`--no-llm`) skip provider initialization and control validation. A Bedrock application-inference-profile ARN hides the model name, so declare `sampling: rejected` for it in the model registry. With the previous `gpt-5.4` and `claude-opus-4-6` defaults a temperature was accepted; unset it, or pin the older model with `SKILLSPECTOR_MODEL`. | Optional |
740
+
|`SKILLSPECTOR_SEED`| Optional integer sampling seed for OpenAI-compatible and Azure OpenAI providers. Other hosted providers and CLI providers do not receive it. Provider support remains model-dependent; `gpt-6.1-sol` receives it, but OpenAI treats `seed` as best-effort, so output can still differ between runs. | Optional |
733
741
|`SKILLSPECTOR_COMPACT_PROMPTS`| Opt-in compact line numbering in LLM prompts: numbered lines render as `L1:`, `L2:` instead of zero-padded `L01:`, `L02:`. Accepted truthy values are `1`, `true`, and `yes` (case-insensitive; surrounding whitespace is trimmed). Unset or any other value keeps the default zero-padded format. | Optional |
734
742
|`ANTHROPIC_API_KEY`| Credential for the Anthropic provider (`SKILLSPECTOR_PROVIDER=anthropic`). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=anthropic`|
|`SKILLSPECTOR_COMPAT_API_KEY`| API key for a generic OpenAI-compatible provider. | Required when `SKILLSPECTOR_PROVIDER=openai_compatible`|
748
756
|`SKILLSPECTOR_COMPAT_BASE_URL`| Base URL for a generic OpenAI-compatible provider. | Required when `SKILLSPECTOR_PROVIDER=openai_compatible`|
749
757
|`SKILLSPECTOR_MODEL`| Override the active provider model. For hosted providers, this replaces the bundled default from the LLM Analysis table. For CLI providers, this is forwarded as `--model` instead of using the local runtime fallback. | Optional |
758
+
|`SKILLSPECTOR_MODEL_<SLOT>`| Override the model for one analyzer slot (for example `SKILLSPECTOR_MODEL_META_ANALYZER`). It takes precedence over `SKILLSPECTOR_MODEL`, which in turn replaces provider slot defaults such as the `anthropic``meta_analyzer` default. See [model provenance](docs/INFERENCE_USAGE.md#model-provenance) for the full precedence. | Optional |
750
759
|`SKILLSPECTOR_MODEL_REGISTRY`| Override the bundled per-provider YAML registry (`src/skillspector/providers/<provider>/model_registry.yaml`) with a custom path. | Optional |
|`SKILLSPECTOR_MAX_DEPENDENCY_SOURCE_ANALYSIS_SECONDS`| Ceiling for the dependency-source analysis pass, in seconds. Defaults to `5.0`, which is the historical value. Raise it on slow or heavily loaded machines so the same unchanged tree does not come back partially inspected, which would set `safe_to_install` to false and record `runtime_limit`. The remaining aggregate workflow time (`SKILLSPECTOR_MAX_WORKFLOW_SECONDS`, 600 seconds by default) still caps the effective value; values above that remaining time have no effect. Invalid, zero, negative, infinite, and NaN values keep the 5-second default. The setting is resolved when the module is imported, so a new process is required after changing it. See [analysis resource bounds](docs/ANALYSIS_RESOURCE_BOUNDS.md#configuring-the-dependency-source-deadline). | Optional |
Copy file name to clipboardExpand all lines: docs/DEVELOPMENT.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,7 +38,7 @@ make install-dev
38
38
-**Provider credential**: depends on the active provider. Hosted providers use the matching variables in [.env.example](../.env.example); Ollama and CLI providers do not require an API key. See [providers/](../src/skillspector/providers/).
39
39
-**`OPENAI_BASE_URL`**: Override the OpenAI endpoint (e.g. point at Ollama).
40
40
-**`SKILLSPECTOR_MODEL`**: Override default model; see [constants.py](../src/skillspector/constants.py).
41
-
-**`SKILLSPECTOR_TEMPERATURE`**: Optional hosted-provider sampling temperature from `0` to `1`.
41
+
-**`SKILLSPECTOR_TEMPERATURE`**: Optional hosted-provider sampling temperature from `0` to `1`.`gpt-6.1-sol` rejects every explicit value. `claude-opus-5-5` and `claude-sonnet-5-5` accept only `1.0` on the native Anthropic endpoint; other routes reject every explicit value. Unsupported settings on active LLM slots stop the scan before analysis. Static scans skip provider initialization and control validation.
42
42
-**`SKILLSPECTOR_SEED`**: Optional integer seed for OpenAI-compatible and Azure OpenAI providers.
@@ -300,9 +300,9 @@ Copy [.env.example](../.env.example) to `.env` in the project root and set value
300
300
|`NVIDIA_INFERENCE_KEY`| Credential for `nv_build`. |`nvapi-...`|
301
301
|`OPENAI_API_KEY`| Credential for `SKILLSPECTOR_PROVIDER=openai`. Also tier-2 fallback for non-OpenAI providers. |`sk-...`|
302
302
|`OPENAI_BASE_URL`| Override the OpenAI endpoint (e.g. point at Ollama). |`http://localhost:11434/v1`|
303
-
|`SKILLSPECTOR_REASONING_EFFORT`| Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged. When unset or blank, SkillSpector sends `high` for `nv_build` with `z-ai/glm-5.3`; other provider/model combinations keep their endpoint defaults. |`high`|
303
+
|`SKILLSPECTOR_REASONING_EFFORT`| Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged. When unset or blank, SkillSpector sends `high` for `nv_build` with `z-ai/glm-5.3`; other provider/model combinations keep their endpoint defaults. `gpt-6.1-sol` accepts only `low`, `medium`, `high`, `xhigh`, or `max`; any other value stops a scan before analysis (exit code 2). |`high`|
304
304
|`SKILLSPECTOR_OUTPUT_LANGUAGE`| Optional short, single-line language label (letters, numbers, spaces, `_`, or `-`; maximum 64 characters) for human-readable LLM finding text. Rule IDs, severity values, paths, code, and other machine-readable values remain unchanged. Unset, blank, or invalid values preserve the default output language. |`Japanese`|
305
-
|`SKILLSPECTOR_TEMPERATURE`| Optional sampling temperature from `0` to `1` for hosted providers. Unset or blank preserves provider defaults. Lower values reduce variation but do not guarantee identical output. |`0`|
305
+
|`SKILLSPECTOR_TEMPERATURE`| Optional sampling temperature from `0` to `1` for hosted providers. Unset or blank preserves provider defaults. Lower values reduce variation but do not guarantee identical output. `gpt-6.1-sol` rejects every explicit value. `claude-opus-5-5` and `claude-sonnet-5-5` accept `1.0` on the native Anthropic endpoint only ([API compatibility rule](https://platform.claude.com/docs/en/api/messages/create)). Unsupported settings on active LLM slots stop a scan before analysis (exit code 2); static scans skip provider initialization and these checks. Leave it unset to preserve the default models’ endpoint defaults. |`0`|
306
306
|`SKILLSPECTOR_SEED`| Optional integer sampling seed for OpenAI-compatible and Azure OpenAI providers. Provider/model support is best-effort; CLI providers ignore it. |`42`|
307
307
|`ANTHROPIC_API_KEY`| Credential for `SKILLSPECTOR_PROVIDER=anthropic`. |`sk-ant-...`|
0 commit comments