Skip to content

Commit c66504b

Browse files
rng1995claude
andcommitted
fix(cli): stop a scan before analysis when a model rejects a requested control
A SKILLSPECTOR_TEMPERATURE (or a GPT-6.1 Sol effort of none/minimal) that the configured model rejects used to surface only while each analyzer built its model: the scan logged one traceback per semantic analyzer, exited 2, and the report showed llm_available: true with a generic telemetry error. unsupported_control_error builds each distinct slot model the way the analyzers do and returns the UnsupportedControlError text; it does nothing when neither control is set, and leaves other failures, such as missing credentials, to the availability path. The scan and baseline commands call it first and exit 2 with that message. is_llm_available also returns it, so the MCP server and the report carry the reason in llm_error. The docs now say that the setting stops the scan on every hosted provider, that it worked with the previous gpt-5.4 and claude-opus-4-6 defaults, and how to declare 'sampling: rejected' for a Bedrock application-inference-profile ARN. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Narendran Raghavan <[email protected]>
1 parent a1e6ffc commit c66504b

6 files changed

Lines changed: 168 additions & 21 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,8 @@ SKILLSPECTOR_OUTPUT_LANGUAGE=
4040
# Optional sampling controls. Temperature is supported by hosted providers;
4141
# seed is forwarded only to OpenAI-compatible and Azure OpenAI endpoints.
4242
# Unset or blank values preserve provider defaults. gpt-6.1-sol, claude-opus-5-5
43-
# and claude-sonnet-5-5 reject any explicit temperature before a request.
43+
# and claude-sonnet-5-5 (the openai and anthropic defaults) reject any explicit
44+
# temperature: on every hosted provider the scan stops before analysis.
4445
SKILLSPECTOR_TEMPERATURE= # range: 0..1
4546
SKILLSPECTOR_SEED= # integer
4647

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -727,9 +727,9 @@ Issues (2)
727727
| `NVIDIA_INFERENCE_KEY` | Credential for the `nv_build` provider (build.nvidia.com). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=nv_build` |
728728
| `OPENAI_API_KEY` | Credential for the OpenAI provider (`SKILLSPECTOR_PROVIDER=openai`). Also serves as the tier-2 fallback in the credential waterfall when the active provider returns no credentials; the fallback then uses the `openai` default model (`gpt-6.1-sol`) for any slot without a model override. Claude Opus or Sonnet 5.5 gateway IDs are not supported through the fallback, because it keeps the active provider's structured-output method; select `SKILLSPECTOR_PROVIDER=openai` for them. | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=openai` |
729729
| `OPENAI_BASE_URL` | Override the OpenAI endpoint (e.g. point at Ollama). | Optional |
730-
| `SKILLSPECTOR_REASONING_EFFORT` | Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged. When unset or blank, SkillSpector sends `high` for `nv_build` with `z-ai/glm-5.3`; other provider/model combinations keep their endpoint defaults (`medium` for `gpt-6.1-sol` and `claude-opus-5-5`, `high` for `claude-sonnet-5-5`). The value applies to every model slot, so the `anthropic` `meta_analyzer` default receives the same effort as the analyzers. `gpt-6.1-sol` accepts only `low`, `medium`, `high`, `xhigh`, or `max`; any other value fails before a request. | Optional |
730+
| `SKILLSPECTOR_REASONING_EFFORT` | Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged. When unset or blank, SkillSpector sends `high` for `nv_build` with `z-ai/glm-5.3`; other provider/model combinations keep their endpoint defaults (`medium` for `gpt-6.1-sol` and `claude-opus-5-5`, `high` for `claude-sonnet-5-5`). The value applies to every model slot, so the `anthropic` `meta_analyzer` default receives the same effort as the analyzers. `gpt-6.1-sol` accepts only `low`, `medium`, `high`, `xhigh`, or `max`; any other value stops a scan before analysis (exit code 2). | Optional |
731731
| `SKILLSPECTOR_OUTPUT_LANGUAGE` | Short, single-line language label (letters, numbers, spaces, `_`, or `-`; maximum 64 characters) for human-readable LLM finding text such as messages, explanations, and remediation. Rule IDs, severity values, paths, code, and other machine-readable values remain unchanged. Unset, blank, or invalid values preserve the default output language. | Optional |
732-
| `SKILLSPECTOR_TEMPERATURE` | Optional sampling temperature from `0` to `1` for hosted providers. Unset or blank preserves the provider default. Lower values can reduce run-to-run variation but do not guarantee identical output. `gpt-6.1-sol`, `claude-opus-5-5`, and `claude-sonnet-5-5` reject sampling controls, so any explicit value (`1` included) fails before a request on the `openai`, `openai_compatible`, `anthropic`, `anthropic_proxy`, and `bedrock` providers, including gateway IDs that name these models. | Optional |
732+
| `SKILLSPECTOR_TEMPERATURE` | Optional sampling temperature from `0` to `1` for hosted providers. Unset or blank preserves the provider default. Lower values can reduce run-to-run variation but do not guarantee identical output. `gpt-6.1-sol`, `claude-opus-5-5`, and `claude-sonnet-5-5` reject sampling controls, so any explicit value (`1` included) stops a scan before analysis (exit code 2) on every hosted provider, including gateway IDs that name these models. A Bedrock application-inference-profile ARN hides the model name, so declare `sampling: rejected` for it in the model registry. With the previous `gpt-5.4` and `claude-opus-4-6` defaults a temperature was accepted; unset it, or pin the older model with `SKILLSPECTOR_MODEL`. | Optional |
733733
| `SKILLSPECTOR_SEED` | Optional integer sampling seed for OpenAI-compatible and Azure OpenAI providers. Other hosted providers and CLI providers do not receive it. Provider support remains model-dependent; `gpt-6.1-sol` receives it, but OpenAI treats `seed` as best-effort, so output can still differ between runs. | Optional |
734734
| `SKILLSPECTOR_COMPACT_PROMPTS` | Opt-in compact line numbering in LLM prompts: numbered lines render as `L1:`, `L2:` instead of zero-padded `L01:`, `L02:`. Accepted truthy values are `1`, `true`, and `yes` (case-insensitive; surrounding whitespace is trimmed). Unset or any other value keeps the default zero-padded format. | Optional |
735735
| `ANTHROPIC_API_KEY` | Credential for the Anthropic provider (`SKILLSPECTOR_PROVIDER=anthropic`). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=anthropic` |

‎docs/DEVELOPMENT.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ make install-dev
3838
- **Provider credential**: depends on the active provider. Hosted providers use the matching variables in [.env.example](../.env.example); Ollama and CLI providers do not require an API key. See [providers/](../src/skillspector/providers/).
3939
- **`OPENAI_BASE_URL`**: Override the OpenAI endpoint (e.g. point at Ollama).
4040
- **`SKILLSPECTOR_MODEL`**: Override default model; see [constants.py](../src/skillspector/constants.py).
41-
- **`SKILLSPECTOR_TEMPERATURE`**: Optional hosted-provider sampling temperature from `0` to `1`. `gpt-6.1-sol`, `claude-opus-5-5`, and `claude-sonnet-5-5` reject any explicit value before a request.
41+
- **`SKILLSPECTOR_TEMPERATURE`**: Optional hosted-provider sampling temperature from `0` to `1`. `gpt-6.1-sol`, `claude-opus-5-5`, and `claude-sonnet-5-5` reject any explicit value: on every hosted provider the scan stops before analysis.
4242
- **`SKILLSPECTOR_SEED`**: Optional integer seed for OpenAI-compatible and Azure OpenAI providers.
4343

4444
- **Logging**: Internal/operational logging uses the stdlib `logging` module. User-facing output (report body, errors, progress) uses Rich `console.print()`.
@@ -302,7 +302,7 @@ Copy [.env.example](../.env.example) to `.env` in the project root and set value
302302
| `OPENAI_BASE_URL` | Override the OpenAI endpoint (e.g. point at Ollama). | `http://localhost:11434/v1` |
303303
| `SKILLSPECTOR_REASONING_EFFORT` | Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged. When unset or blank, SkillSpector sends `high` for `nv_build` with `z-ai/glm-5.3`; other provider/model combinations keep their endpoint defaults. `gpt-6.1-sol` accepts only `low`, `medium`, `high`, `xhigh`, or `max`; any other value fails before a request. | `high` |
304304
| `SKILLSPECTOR_OUTPUT_LANGUAGE` | Optional short, single-line language label (letters, numbers, spaces, `_`, or `-`; maximum 64 characters) for human-readable LLM finding text. Rule IDs, severity values, paths, code, and other machine-readable values remain unchanged. Unset, blank, or invalid values preserve the default output language. | `Japanese` |
305-
| `SKILLSPECTOR_TEMPERATURE` | Optional sampling temperature from `0` to `1` for hosted providers. Unset or blank preserves provider defaults. Lower values reduce variation but do not guarantee identical output. Any explicit value fails before a request for `gpt-6.1-sol`, `claude-opus-5-5`, and `claude-sonnet-5-5`. | `0` |
305+
| `SKILLSPECTOR_TEMPERATURE` | Optional sampling temperature from `0` to `1` for hosted providers. Unset or blank preserves provider defaults. Lower values reduce variation but do not guarantee identical output. Any explicit value stops a scan before analysis on every hosted provider for `gpt-6.1-sol`, `claude-opus-5-5`, and `claude-sonnet-5-5`; leave it unset for the default `openai` and `anthropic` models. | `0` |
306306
| `SKILLSPECTOR_SEED` | Optional integer sampling seed for OpenAI-compatible and Azure OpenAI providers. Provider/model support is best-effort; CLI providers ignore it. | `42` |
307307
| `ANTHROPIC_API_KEY` | Credential for `SKILLSPECTOR_PROVIDER=anthropic`. | `sk-ant-...` |
308308
| `OLLAMA_BASE_URL` | Optional Ollama endpoint override. | `http://localhost:11434/v1` |

‎src/skillspector/cli.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@
5757
inspection_work_id,
5858
ledger_event,
5959
)
60+
from skillspector.llm_utils import unsupported_control_error
6061
from skillspector.logging_config import get_logger, set_level
6162
from skillspector.mcp_registry import scan_registry
6263
from skillspector.models import Finding
@@ -371,6 +372,16 @@ def _write_result(
371372
print(report_body)
372373

373374

375+
def _exit_on_unsupported_llm_controls(no_llm: bool) -> None:
376+
"""Exit 2 before any analysis when a configured model rejects a requested control."""
377+
if no_llm:
378+
return
379+
error = unsupported_control_error()
380+
if error is not None:
381+
err_console.print(f"[red]Error:[/red] {error}")
382+
raise typer.Exit(code=2)
383+
384+
374385
def _validate_min_coverage(value: float | None) -> float | None:
375386
"""Accept only finite percentage thresholds in the CLI's supported range."""
376387
if value is not None and (not math.isfinite(value) or not 0 <= value <= 100):
@@ -734,6 +745,7 @@ def scan(
734745
except ValueError as exc:
735746
err_console.print(f"[red]Error:[/red] invalid transitive prefix: {exc}")
736747
raise typer.Exit(code=2) from exc
748+
_exit_on_unsupported_llm_controls(no_llm)
737749
yara_dir = str(yara_rules_dir.resolve()) if yara_rules_dir else None
738750
pre_scan_ledger_events: list[dict[str, object]] = []
739751
discovery_console = (
@@ -3408,6 +3420,7 @@ def baseline(
34083420
skillspector baseline ./my-skill/ -o team-baseline.yaml --no-llm
34093421
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml
34103422
"""
3423+
_exit_on_unsupported_llm_controls(no_llm)
34113424
result = None
34123425
try:
34133426
if verbose:

‎src/skillspector/llm_utils.py‎

Lines changed: 43 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
from langchain_core.language_models.chat_models import BaseChatModel
4949
from langchain_core.runnables import Runnable, RunnableLambda
5050

51+
from skillspector.constants import build_model_config
5152
from skillspector.inference_usage import (
5253
InferenceUsageCollector,
5354
chat_model_controls,
@@ -67,6 +68,7 @@
6768
resolve_chat_model_credentials,
6869
resolve_provider_credentials,
6970
)
71+
from skillspector.providers.chat_models import UnsupportedControlError
7072
from skillspector.providers.gemini import GeminiProvider
7173
from skillspector.providers.openai import OpenAIProvider
7274

@@ -140,7 +142,9 @@ def is_llm_available(timeout: float | None = 120) -> tuple[bool, str | None]:
140142
``opencode_cli``) are checked through their ``is_available()`` method
141143
first. Other providers probe the same native chat-model path used by
142144
:func:`get_chat_model`; unbound HTTP providers keep the
143-
credential-resolution and OpenAI fallback path.
145+
credential-resolution and OpenAI fallback path. A configured model that
146+
rejects a requested control makes the LLM unavailable, with the reason
147+
from :func:`unsupported_control_error`.
144148
"""
145149
try:
146150
provider = get_active_provider()
@@ -149,28 +153,52 @@ def is_llm_available(timeout: float | None = 120) -> tuple[bool, str | None]:
149153
if has_cli_capability(provider):
150154
return provider.is_available() # type: ignore[attr-defined]
151155

152-
if has_provider_binding():
153-
try:
156+
try:
157+
if has_provider_binding():
154158
model = provider.resolve_model()
155159
create_chat_model(
156160
model=model,
157161
max_tokens=get_max_output_tokens(model),
158162
timeout=timeout,
159163
)
160-
except (ValueError, RefreshError, TimeoutError) as exc:
161-
return False, str(exc)
162-
return True, None
163-
if isinstance(provider, GeminiProvider):
164-
try:
164+
elif isinstance(provider, GeminiProvider):
165165
provider.resolve_credentials(timeout=timeout)
166-
except (ValueError, RefreshError, TimeoutError) as exc:
167-
return False, str(exc)
168-
return True, None
169-
try:
170-
_resolve_llm_credentials()
171-
except ValueError as exc:
166+
else:
167+
_resolve_llm_credentials()
168+
except (ValueError, RefreshError, TimeoutError) as exc:
172169
return False, str(exc)
173-
return True, None
170+
control_error = unsupported_control_error(timeout)
171+
return control_error is None, control_error
172+
173+
174+
def unsupported_control_error(timeout: float | None = 120) -> str | None:
175+
"""Return why a configured slot model rejects a requested control, else ``None``.
176+
177+
Only ``SKILLSPECTOR_TEMPERATURE`` and ``SKILLSPECTOR_REASONING_EFFORT`` can
178+
be rejected, so nothing runs when both are unset. Otherwise each distinct
179+
slot model is built the way the analyzers build it, so the provider's own
180+
guard (model name or registry entry) reports an
181+
:class:`~skillspector.providers.chat_models.UnsupportedControlError`
182+
before any analysis. Other failures, such as missing credentials, are
183+
left to the regular availability path.
184+
"""
185+
if not any(
186+
os.environ.get(name, "").strip()
187+
for name in ("SKILLSPECTOR_TEMPERATURE", "SKILLSPECTOR_REASONING_EFFORT")
188+
):
189+
return None
190+
try:
191+
if has_cli_capability(get_active_provider()):
192+
return None
193+
models = dict.fromkeys(build_model_config().values())
194+
for model in models:
195+
get_chat_model(model, timeout=timeout)
196+
except UnsupportedControlError as exc:
197+
return str(exc)
198+
except Exception:
199+
# Not a control rejection; availability checks report it.
200+
return None
201+
return None
174202

175203

176204
def fetch_model_token_limits(model_label: str) -> tuple[int, int]:

‎tests/unit/test_model_capabilities.py‎

Lines changed: 106 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,27 @@
1717

1818
from __future__ import annotations
1919

20+
from pathlib import Path
21+
from unittest.mock import patch
22+
2023
import pytest
2124
from langchain_anthropic import ChatAnthropic
2225
from langchain_core.exceptions import OutputParserException
2326
from langchain_core.messages import AIMessage
2427
from langchain_core.outputs import ChatGeneration, ChatResult
2528
from langchain_openai import ChatOpenAI
2629
from pydantic import BaseModel
30+
from typer.testing import CliRunner
2731

32+
from skillspector.cli import app
33+
from skillspector.constants import _MODEL_SLOTS
2834
from skillspector.inference_usage import chat_model_controls, chat_model_requested_controls
29-
from skillspector.llm_utils import StructuredOutputParseError, bind_structured_output
35+
from skillspector.llm_utils import (
36+
StructuredOutputParseError,
37+
bind_structured_output,
38+
is_llm_available,
39+
unsupported_control_error,
40+
)
3041
from skillspector.providers import registry
3142
from skillspector.providers.anthropic import AnthropicProvider
3243
from skillspector.providers.anthropic_proxy import AnthropicProxyProvider
@@ -77,6 +88,7 @@ def _clean_env(monkeypatch: pytest.MonkeyPatch):
7788
"AZURE_OPENAI_API_KEY",
7889
"AZURE_OPENAI_ENDPOINT",
7990
"AZURE_OPENAI_DEPLOYMENT",
91+
*(f"SKILLSPECTOR_MODEL_{slot.upper()}" for slot in _MODEL_SLOTS),
8092
):
8193
monkeypatch.delenv(name, raising=False)
8294
registry._load.cache_clear()
@@ -531,3 +543,96 @@ def test_other_models_keep_temperature(self, monkeypatch: pytest.MonkeyPatch) ->
531543
monkeypatch.setenv("SKILLSPECTOR_TEMPERATURE", "0.2")
532544
llm = AzureOpenAIProvider().create_chat_model("gpt-4o", max_tokens=1_000)
533545
assert chat_model_requested_controls(llm)["temperature"] == 0.2
546+
547+
548+
_DEFAULT_PROVIDER_ENV = pytest.mark.parametrize(
549+
("provider", "credential", "model"),
550+
[
551+
("openai", "OPENAI_API_KEY", "gpt-6.1-sol"),
552+
("anthropic", "ANTHROPIC_API_KEY", "claude-opus-5-5"),
553+
],
554+
)
555+
556+
557+
class TestScanPreflightControls:
558+
"""A rejected control stops the scan before analysis and names the setting."""
559+
560+
@_DEFAULT_PROVIDER_ENV
561+
def test_scan_exits_before_analysis_with_the_reason(
562+
self,
563+
monkeypatch: pytest.MonkeyPatch,
564+
tmp_path: Path,
565+
provider: str,
566+
credential: str,
567+
model: str,
568+
) -> None:
569+
(tmp_path / "SKILL.md").write_text("---\nname: demo\n---\nHello.\n", encoding="utf-8")
570+
monkeypatch.setenv("SKILLSPECTOR_PROVIDER", provider)
571+
monkeypatch.setenv(credential, "sk-test")
572+
monkeypatch.setenv("SKILLSPECTOR_TEMPERATURE", "0")
573+
574+
with patch("skillspector.cli.graph.invoke") as invoke:
575+
result = CliRunner().invoke(app, ["scan", str(tmp_path), "--format", "json"])
576+
577+
assert result.exit_code == 2
578+
assert f"SKILLSPECTOR_TEMPERATURE is not supported by {model}" in result.output
579+
invoke.assert_not_called()
580+
581+
@_DEFAULT_PROVIDER_ENV
582+
def test_preflight_reports_the_reason_as_llm_error(
583+
self, monkeypatch: pytest.MonkeyPatch, provider: str, credential: str, model: str
584+
) -> None:
585+
monkeypatch.setenv("SKILLSPECTOR_PROVIDER", provider)
586+
monkeypatch.setenv(credential, "sk-test")
587+
monkeypatch.setenv("SKILLSPECTOR_TEMPERATURE", "0")
588+
589+
available, error = is_llm_available()
590+
591+
assert available is False
592+
assert error is not None and f"not supported by {model}; unset it" in error
593+
594+
def test_sol_effort_is_checked_on_every_slot(self, monkeypatch: pytest.MonkeyPatch) -> None:
595+
monkeypatch.setenv("SKILLSPECTOR_PROVIDER", "openai")
596+
monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
597+
monkeypatch.setenv("SKILLSPECTOR_REASONING_EFFORT", "minimal")
598+
monkeypatch.setenv("SKILLSPECTOR_MODEL", "gpt-5.4")
599+
monkeypatch.setenv("SKILLSPECTOR_MODEL_META_ANALYZER", "gpt-6.1-sol")
600+
601+
error = unsupported_control_error()
602+
603+
assert error is not None and "SKILLSPECTOR_REASONING_EFFORT='minimal'" in error
604+
605+
def test_supported_controls_and_missing_credentials_pass(
606+
self, monkeypatch: pytest.MonkeyPatch
607+
) -> None:
608+
monkeypatch.setenv("SKILLSPECTOR_PROVIDER", "openai")
609+
monkeypatch.setenv("SKILLSPECTOR_TEMPERATURE", "0")
610+
# Missing credentials are reported by the availability path, not as a control error.
611+
assert unsupported_control_error() is None
612+
613+
monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
614+
monkeypatch.setenv("SKILLSPECTOR_MODEL", "gpt-5.4")
615+
assert unsupported_control_error() is None
616+
assert is_llm_available() == (True, None)
617+
618+
def test_cli_providers_are_not_checked(self, monkeypatch: pytest.MonkeyPatch) -> None:
619+
monkeypatch.setenv("SKILLSPECTOR_PROVIDER", "claude_cli")
620+
monkeypatch.setenv("SKILLSPECTOR_MODEL", "gpt-6.1-sol")
621+
monkeypatch.setenv("SKILLSPECTOR_TEMPERATURE", "0")
622+
assert unsupported_control_error() is None
623+
624+
def test_no_llm_scan_ignores_rejected_controls(
625+
self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
626+
) -> None:
627+
(tmp_path / "SKILL.md").write_text("---\nname: demo\n---\nHello.\n", encoding="utf-8")
628+
monkeypatch.setenv("SKILLSPECTOR_PROVIDER", "openai")
629+
monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
630+
monkeypatch.setenv("SKILLSPECTOR_TEMPERATURE", "0")
631+
632+
with patch("skillspector.cli.unsupported_control_error") as check:
633+
result = CliRunner().invoke(
634+
app, ["scan", str(tmp_path), "--format", "json", "--no-llm"]
635+
)
636+
637+
check.assert_not_called()
638+
assert "SKILLSPECTOR_TEMPERATURE" not in result.output

0 commit comments

Comments
 (0)