Skip to content

test: raise the pytest floor to 9.2, which fixes the skipped conftest fixtures behind a flaky MCP auth test - #3880

Merged
gutzbenj merged 5 commits into
mainfrom
fix/3859-mcp-auth-test-isolation
Nov 8, 2026
Merged

gutzbenj merged 5 commits into
mainfrom
fix/3859-mcp-auth-test-isolation

Conversation

@gutzbenj

@gutzbenj gutzbenj commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

Closes #3859

Cause

test_mcp_auth_tool_answers_as_the_route_does failed in a selection that names tests/ui files on both sides of a file outside tests/ui. That is a regression of pytest 9.1.0 and 9.1.1, pytest-dev/pytest#14635 ("fixtures not found when a parent directory appears multiple times in a argument list"), fixed by #14645 in 9.2.0.

  • pytest made a second node for tests/ui, which had none of the autouse fixtures of tests/ui/conftest.py. So _forget_auth_probes did not run for test_core.py and test_restapi.py.
  • The probe that test_coverage_and_auth_tell_a_credential_that_could_not_be_checked_as_null leaves in core._auth_probes (a valid: False, kept 60 s by get_auth) then answered the MCP test. The library cache is keyed correctly and is not at fault.
  • It happens on 9.1.0 and 9.1.1 and not on 8.4.2, 9.0.2 or 9.2.0. The same behaviour skipped the autouse fixtures of tests/provider/conftest.py and tests/provider/metno/frost/conftest.py in such selections.

Fix

The floor of the dev group is pytest>=9.2 (one higher minimum, no workaround). uv.lock changes only pytest 9.1.1 to 9.2.0. tests/ui/conftest.py stays as on main. The diff to main is pyproject.toml and uv.lock. pytest is in the dev group, not an extra, so there is no CHANGELOG entry.

Run (pytest 9.2.0)

  • The issue's selection with main's layout, in one process: 1523 passed, 6 skipped. Under -n 4, 3 times: 1523 passed, 6 skipped each time. On 9.1.1 it failed, 3 tests.
  • pytest tests/ui/test_empty_listing.py tests/test_exceptions.py tests/ui/test_core.py --setup-plan: _forget_auth_probes on all 135 tests of test_core.py.
  • pytest tests/provider/metno/frost/test_is_valid.py tests/test_info.py tests/provider/metno/frost/test_values_failures.py tests/provider/dwd/dmo/test_api.py --setup-plan: the tests/provider autouse fixtures (recorded_frames, _timestamp_column_is_utc_microseconds) and the metno/frost one (_no_second_look_at_a_refusal) on every test of those directories.
  • uv run poe format, lint, typecheck: clean. uv run poe "test:offline": 13990 passed, 49 skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_0187kUCUFKYgEkY8mDbH23V8

…lection's order cannot skip it (GH-3859)

`test_mcp_auth_tool_answers_as_the_route_does` failed in a selection that
names `tests/ui` files on both sides of a file outside it. pytest 9 ties the
autouse fixtures of a sub-directory's conftest to the directory node it loaded
the conftest for; the second node it makes for `tests/ui` has none, so
`_forget_auth_probes` of `tests/ui/conftest.py` did not run for `test_core.py`
and `test_restapi.py`. The metno/frost probe that
`test_coverage_and_auth_tell_a_credential_that_could_not_be_checked_as_null`
leaves in the cache (a `valid: False`, kept 60 s by `get_auth`) then answered
the MCP test, which expected the probe it had stubbed.

The fixture moves to `tests/conftest.py`, whose autouse fixtures reach every
test, and clears only where `wetterdienst.ui.core` was imported. A test runs
pytest's `--setup-plan` over the order that failed and asks every test for the
fixture.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0187kUCUFKYgEkY8mDbH23V8
@gutzbenj gutzbenj added bug Something isn't working tests All about the test suite mcp MCP server and its tools (ui/mcp.py) labels Nov 7, 2026
@codecov

codecov Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

❌ 5 Tests Failed:

Tests completed Failed Passed Skipped
15024 5 15019 172
View the top 1 failed test(s) by shortest run time
tests.test_api::test_api_aemet_observation
Stack Traces | 2.4s run time
default_settings = {"cache_disable": false, "cache_dir": ".../pytest-0/popen-gw0/wd-cache-gw0", "fsspec_client_kwargs": {"headers": {"User-Agent": "wetterdienst/0.184.0 (Linux)"}, "timeout": 30}, "auth": {"aemet": "**********", "knmi": "**********", "metno_frost": ["**********", ""], "ceda": ["**********", "**********"]}, "use_certifi": false, "fetch_ahead": 4, "restapi_sql": false, "restapi_max_stations": 100, "restapi_max_rows": 500000, "restapi_max_stream_rows": 10000000, "restapi_timeout": 60.0, "restapi_max_concurrent": 32, "restapi_allowed_hosts": []}

    @pytest.mark.remote
    @pytest.mark.skipif(
        not AemetObservationRequest.is_configured(),
        reason="AEMET credentials not set — provide WD_AUTH__AEMET=<api_key>",
    )
    def test_api_aemet_observation(default_settings: Settings) -> None:
        """Test AEMET observation API."""
        request = AemetObservationRequest(
            parameters=[("daily", "data", "temperature_air_mean_2m")],
            start="2020-01-01",
            end="2020-01-02",
            settings=default_settings,
        ).filter_by_station_id("3195")
        assert not request.df.is_empty()
        assert set(request.df.columns).issuperset(DF_STATIONS_MINIMUM_COLUMNS)
        # AEMET's station inventory doesn't provide start_timestamp/end_timestamp at all.
        assert _is_complete_stations_df(request.df, exclude_columns={"start_timestamp", "end_timestamp"})
        first_date = request.df.get_column("start_timestamp").gather(0).to_list()[0]
        if first_date:
            assert first_date.tzinfo == zoneinfo.ZoneInfo(key="UTC")
>       values = next(request.values().query()).df
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests/test_api.py:1226: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../wetterdienst/model/values.py:477: in query
    yield from self._query_stations(groups, ahead)
.../wetterdienst/model/values.py:528: in _query_stations
    df = self._filter_by_window(self._collect_station_data(station_id, available_datasets))
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../wetterdienst/model/values.py:994: in _collect_station_data
    df = self._process_dataset(station_id, dataset, parameters)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../wetterdienst/model/values.py:1019: in _process_dataset
    df = self._collect_station_parameter_or_dataset(
.../aemet/observation/api.py:486: in _collect_station_parameter_or_dataset
    return self._collect_daily(station_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../aemet/observation/api.py:563: in _collect_daily
    _raise_if_unanswered(payload)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

payload = AemetKeyRejectedError("AEMET answered HTTP 401: 401, message='Unauthorized', url='https://opendata.aemet..../datos/fechaini/2020-01-01T00:00:00UTC/fechafin/2020-01-02T23:59:59UTC/estacion/3195'")

    def _raise_if_unanswered(payload: bytes | Exception) -> None:
        """Raise what `_fetch_datos` returned if it is not an answer to the request, or a refusal of the key.
    
        A connection that could not be made (`NoInternetError`), an AEMET that did not answer
        (`DownloadError`, GH-2965) and an API key that AEMET refuses (`CredentialRejectedError`, GH-3018): a
        request that goes on past any of them reads as a station without data, or as a network without
        stations. What AEMET did answer is left to the caller, which skips the chunk.
        """
        if isinstance(payload, (NoInternetError, DownloadError)):
            raise payload
        if isinstance(payload, AemetKeyRejectedError):
>           raise _key_rejected()
E           wetterdienst.exceptions.CredentialRejectedError: AEMET rejected the request. Check that WD_AUTH__AEMET contains a valid API key.

.../aemet/observation/api.py:305: CredentialRejectedError
View the full list of 4 ❄️ flaky test(s)
tests.provider.metno.frost.test_api::test_metno_frost_values_wind_direction_no_unique_direction_is_null

Flake rate in main: 100.00% (Passed 0 times, Failed 1 times)

Stack Traces | 0.573s run time
@pytest.mark.remote
    def test_metno_frost_values_wind_direction_no_unique_direction_is_null() -> None:
        """The wind direction in weak winds is Frost's -3 and comes back null, the code in the flags (GH-3137).
    
        SN1165900, a WMO station Frost lists since 2026-11-03, has -3 at 2026-11-04 13:00 (beside a speed of 1 to 2 m/s):
        "No unique wind direction is given when winds are weak" in Frost's data clarifications. The range check used to turn
        it into a null as `rejected:range`, which says the reading is out of range.
        """
        df = (
            MetnoFrostRequest(
                parameters=[("hourly", "data", "wind_direction_mean_last_10min")],
                start=dt.datetime(2026, 11, 4, 13, tzinfo=UTC),
                end=dt.datetime(2026, 11, 4, 13, tzinfo=UTC),
                options={"drop_nulls": False},
            )
            .filter_by_station_id("SN1165900")
            .values()
            .all()
            .df
        )
>       assert df["value"].to_list() == [None]
E       assert [] == [None]
E         
E         Right contains one more item: None
E         
E         Full diff: (-: missing in left side, +: extra in left side)
E         + []
E         - [
E         -     None,
E         - ]

.../metno/frost/test_api.py:358: AssertionError
tests.provider.test_station_catalogues_live::test_a_live_catalogue_has_its_data[meteoswiss/observation 10_minutes/data]

Flake rate in main: 100.00% (Passed 0 times, Failed 46 times)

Stack Traces | 2.36s run time
key = 'meteoswiss/observation', resolution = '10_minutes', dataset = 'data'

    @pytest.mark.remote
    @pytest.mark.parametrize(("key", "resolution", "dataset"), _coverage_cases())
    def test_a_live_catalogue_has_its_data(key: str, resolution: str, dataset: str) -> None:
        """Hold the live station list to the data files beside it, naming the provider, station and rule that fails."""
        provider, network = key.split("/")
        request = Wetterdienst.resolve(provider, network)(parameters=[(resolution, dataset)])
        index = INDEXES[key]
>       assert_covered(
            request.all().df,
            index.ids(request.settings, dataset_of(key, resolution, dataset)),
            f"{key} {resolution}/{dataset}",
            accepted=ACCEPTED_LIVE.get(key, ()),
            rules=index.rules,
            fail_on_stale=False,
        )

tests/provider/test_station_catalogues_live.py:255: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/provider/station_catalogue.py:713: in assert_covered
    _assert_none_left(left, stale if fail_on_stale else [], catalogue, limit)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

left = [Violation(catalogue='meteoswiss/observation 10_minutes/data', rule='data_orphan', station_id='MLJ', detail='a data file of 10_minutes/data is named for it and the catalogue does not list it', row={'station_id': 'MLJ', 'resolution': '10_minutes', 'dataset': 'data'})]
stale = [], catalogue = 'meteoswiss/observation 10_minutes/data', limit = 10

    def _assert_none_left(left: list[Violation], stale: list[Accepted], catalogue: str, limit: int) -> None:
        # at most `limit` of each rule, so that a rule broken by thousands of stations does not hide the others
        problems = []
        for rule in dict.fromkeys(violation.rule for violation in left):
            broken = [violation for violation in left if violation.rule == rule]
            problems += [str(violation) for violation in broken[:limit]]
            if len(broken) > limit:
                problems.append(f"{catalogue}: {rule}: ... and {len(broken) - limit} more")
        problems += [
            f"{catalogue}: the exception for {exception.rule} matches no station: {exception.reason}" for exception in stale
        ]
>       assert not problems, "\n".join(problems)
               ^^^^^^^^^^^^
E       AssertionError: meteoswiss/observation 10_minutes/data: data_orphan: station 'MLJ': a data file of 10_minutes/data is named for it and the catalogue does not list it

tests/provider/station_catalogue.py:635: AssertionError
tests.provider.test_station_catalogues_live::test_a_live_catalogue_has_its_data[meteoswiss/observation daily/data]

Flake rate in main: 100.00% (Passed 0 times, Failed 46 times)

Stack Traces | 0.058s run time
key = 'meteoswiss/observation', resolution = 'daily', dataset = 'data'

    @pytest.mark.remote
    @pytest.mark.parametrize(("key", "resolution", "dataset"), _coverage_cases())
    def test_a_live_catalogue_has_its_data(key: str, resolution: str, dataset: str) -> None:
        """Hold the live station list to the data files beside it, naming the provider, station and rule that fails."""
        provider, network = key.split("/")
        request = Wetterdienst.resolve(provider, network)(parameters=[(resolution, dataset)])
        index = INDEXES[key]
>       assert_covered(
            request.all().df,
            index.ids(request.settings, dataset_of(key, resolution, dataset)),
            f"{key} {resolution}/{dataset}",
            accepted=ACCEPTED_LIVE.get(key, ()),
            rules=index.rules,
            fail_on_stale=False,
        )

tests/provider/test_station_catalogues_live.py:255: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/provider/station_catalogue.py:713: in assert_covered
    _assert_none_left(left, stale if fail_on_stale else [], catalogue, limit)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

left = [Violation(catalogue='meteoswiss/observation daily/data', rule='data_orphan', station_id='MLJ', detail='a data file of daily/data is named for it and the catalogue does not list it', row={'station_id': 'MLJ', 'resolution': 'daily', 'dataset': 'data'})]
stale = [], catalogue = 'meteoswiss/observation daily/data', limit = 10

    def _assert_none_left(left: list[Violation], stale: list[Accepted], catalogue: str, limit: int) -> None:
        # at most `limit` of each rule, so that a rule broken by thousands of stations does not hide the others
        problems = []
        for rule in dict.fromkeys(violation.rule for violation in left):
            broken = [violation for violation in left if violation.rule == rule]
            problems += [str(violation) for violation in broken[:limit]]
            if len(broken) > limit:
                problems.append(f"{catalogue}: {rule}: ... and {len(broken) - limit} more")
        problems += [
            f"{catalogue}: the exception for {exception.rule} matches no station: {exception.reason}" for exception in stale
        ]
>       assert not problems, "\n".join(problems)
               ^^^^^^^^^^^^
E       AssertionError: meteoswiss/observation daily/data: data_orphan: station 'MLJ': a data file of daily/data is named for it and the catalogue does not list it

tests/provider/station_catalogue.py:635: AssertionError
tests.provider.test_station_catalogues_live::test_a_live_catalogue_has_its_data[meteoswiss/observation hourly/data]

Flake rate in main: 100.00% (Passed 0 times, Failed 46 times)

Stack Traces | 0.071s run time
key = 'meteoswiss/observation', resolution = 'hourly', dataset = 'data'

    @pytest.mark.remote
    @pytest.mark.parametrize(("key", "resolution", "dataset"), _coverage_cases())
    def test_a_live_catalogue_has_its_data(key: str, resolution: str, dataset: str) -> None:
        """Hold the live station list to the data files beside it, naming the provider, station and rule that fails."""
        provider, network = key.split("/")
        request = Wetterdienst.resolve(provider, network)(parameters=[(resolution, dataset)])
        index = INDEXES[key]
>       assert_covered(
            request.all().df,
            index.ids(request.settings, dataset_of(key, resolution, dataset)),
            f"{key} {resolution}/{dataset}",
            accepted=ACCEPTED_LIVE.get(key, ()),
            rules=index.rules,
            fail_on_stale=False,
        )

tests/provider/test_station_catalogues_live.py:255: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/provider/station_catalogue.py:713: in assert_covered
    _assert_none_left(left, stale if fail_on_stale else [], catalogue, limit)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

left = [Violation(catalogue='meteoswiss/observation hourly/data', rule='data_orphan', station_id='MLJ', detail='a data file of hourly/data is named for it and the catalogue does not list it', row={'station_id': 'MLJ', 'resolution': 'hourly', 'dataset': 'data'})]
stale = [], catalogue = 'meteoswiss/observation hourly/data', limit = 10

    def _assert_none_left(left: list[Violation], stale: list[Accepted], catalogue: str, limit: int) -> None:
        # at most `limit` of each rule, so that a rule broken by thousands of stations does not hide the others
        problems = []
        for rule in dict.fromkeys(violation.rule for violation in left):
            broken = [violation for violation in left if violation.rule == rule]
            problems += [str(violation) for violation in broken[:limit]]
            if len(broken) > limit:
                problems.append(f"{catalogue}: {rule}: ... and {len(broken) - limit} more")
        problems += [
            f"{catalogue}: the exception for {exception.rule} matches no station: {exception.reason}" for exception in stale
        ]
>       assert not problems, "\n".join(problems)
               ^^^^^^^^^^^^
E       AssertionError: meteoswiss/observation hourly/data: data_orphan: station 'MLJ': a data file of hourly/data is named for it and the catalogue does not list it

tests/provider/station_catalogue.py:635: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

gutzbenj and others added 3 commits November 8, 2026 02:52
…back (GH-3859)

`test_mcp_auth_tool_answers_as_the_route_does` failed in a selection that
names `tests/ui` files on both sides of a file outside it. That is a regression
of pytest 9.1.0 and 9.1.1 (pytest-dev/pytest#14635, "fixtures not found when a
parent directory appears multiple times in a argument list"), fixed in 9.2.0
(pytest-dev/pytest#14645): the autouse fixtures of a sub-directory's conftest
did not reach the tests of the second node pytest made for that directory, so
`_forget_auth_probes` of `tests/ui/conftest.py` did not run for `test_core.py`
and `test_restapi.py`, and the probe that an earlier test left in the cache
answered the MCP test.

The floor of the dev group is `pytest>=9.2` instead (uv.lock: pytest 9.1.1 to
9.2.0 only). `tests/ui/conftest.py` and its fixture are as on main, and the
workaround of the first commit of this branch, with its guard test, is gone.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0187kUCUFKYgEkY8mDbH23V8
@gutzbenj gutzbenj changed the title test(ui): reset the credential probes from the root conftest, so a selection's order cannot skip it test: raise the pytest floor to 9.2, which fixes the skipped conftest fixtures behind a flaky MCP auth test Nov 8, 2026
@gutzbenj
gutzbenj merged commit c34f2e7 into main Nov 8, 2026
27 of 39 checks passed
@gutzbenj
gutzbenj deleted the fix/3859-mcp-auth-test-isolation branch November 8, 2026 05:43

This branch had an error being deployed

1 failed deployment
codecov — 1856d500 Deployed Nov 8, 2026 by gutzbenj via coverage #10679
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working mcp MCP server and its tools (ui/mcp.py) tests All about the test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests: test_mcp_auth_tool_answers_as_the_route_does fails depending on test order

1 participant