Skip to content

fix(sudestavenir_fr): parse mixed green-waste schedules - #7079

Merged
markvp merged 2 commits into
mampfes:masterfrom
snowyukitty:fix/sudestavenir-green-waste
Aug 2, 2026
Merged

markvp merged 2 commits into
mampfes:masterfrom
snowyukitty:fix/sudestavenir-green-waste

Conversation

@snowyukitty

Copy link
Copy Markdown
Contributor

Summary

  • parse GPSEA annual precision text containing explicit French dates plus a bounded weekly or fortnightly rule
  • preserve existing month-first precision-only behavior while avoiding next-year extrapolation
  • add deterministic parser and fetch regressions for green waste, precision-only, and fortnightly schedules

Closes #7077.

Type of change

  • New source
  • Bug fix / source fix
  • Documentation update
  • Other

Validation

  • python -m pytest tests/test_sudestavenir_fr.py -q — 8 passed (focused file; not selected by the default pytest.ini)
  • python -m pytest tests/test_source_components.py -q — 34 passed
  • python -m pytest tests/ -q — 38 passed (default-discovered suite; does not include the focused file above)
  • python test_sources.py -s sudestavenir_fr — 277 entries for the committed test case
  • changed-file pre-commit hooks — Ruff, codespell, Bandit, and mypy passed
  • live issue configuration — 20 green-waste entries from 2026-07-29 through 2026-12-09
  • live before/after differential — all 277 serialized entries for the committed test case remained byte-for-byte identical

Checklist

  • python -m pytest tests/test_source_components.py -q passes
  • ruff check --fix and ruff format run on both changed files
  • No generated files in diff
  • Existing source; no new doc/source/<name>.md file required
  • Tests use mocked address data; the public issue example was used only for local live verification

snowyukitty and others added 2 commits July 29, 2026 10:39
The source doc still described dechets vegetaux as a precomputed date list only. Reflect that the precision text can also carry a bounded weekly rule, and note that bounded rules are not extrapolated into the following year.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

@markvp markvp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on this one. The mixed precision-text parsing is exactly the right shape: keeping precisi authoritative, merging the explicit dates with the bounded rule through a set, and clipping to the request window means bounded ranges never leak into next year.

I verified it live both ways. The committed Creteil test case is byte-for-byte identical to master (278 entries, nothing gained or lost), and the Noiseau address from #7077 picks up 19 Déchets végétaux Wednesdays running to 9 December with nothing else changed. The deterministic regression tests are a welcome addition too.

I made one small tweak: the source doc still described green waste as a fixed date list, so I refreshed that paragraph to cover the mixed case. Nothing needed from you.

@markvp
markvp merged commit c3ac5c6 into mampfes:master Aug 2, 2026
3 checks passed
@markvp markvp mentioned this pull request Aug 2, 2026
markvp added a commit that referenced this pull request Aug 2, 2026
* Add Viborg (Revas) to Affaldonline source (#6883)

Viborg Kommune's waste operator Revas uses the same Affaldonline
platform/response format already supported via the "default" parser.

Closes #5681

* Update generated documentation

* feat(source): add CEM Ambiente (cem_ambiente_it) (#6884)

Adds support for CEM Ambiente's public CEM FACILE API, covering ~77
municipalities in the Milano/Monza e Brianza/Lodi area, closing #6871.

* Update generated documentation

* Generalize isontinambiente_it source to cover all Isontina Ambiente municipalities (#6885)

The source only needs address_id to resolve the correct calendar; the
municipality slug in the request URL is cosmetic and does not affect the
result. Update TITLE/DESCRIPTION/EXTRA_INFO/TEST_CASES/HOW_TO text to
reflect that this single source already covers all 27 municipalities
served by Isontina Ambiente, not only Ronchi dei Legionari.

Fixes #5543

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* Add Prince George's County, MD source (#6887)

Implements trash, recycling, bulky trash, and yard trim collection
lookup via the County's public ArcGIS "Once-A-Week Collection" layer,
resolving addresses through the Esri World Geocoder.

Closes #5576

* Update generated documentation

* feat(source): add City of Yarra, VIC, AU (yarracity_vic_gov_au) (#6888)

Co-authored-by: Claude Fable 5 <[email protected]>

* Update generated documentation

* docs: announce the 3.0.0 architecture pre-release (#6892)

Add a short callout to the README and info.md preambles inviting users to test
the opt-in 3.0.0-alpha pipeline pre-release via HACS beta versions, focused on
the benefits (more reuse, easier addition and management of sources, consistent
labels). Forward-ports the announcement shipped in 2.31.1 so it survives into
the next release from master.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* docs(installation): warn that fetch_time/day_switch_time must be quoted (#6891)

PyYAML follows the legacy YAML 1.1 spec, which parses an unquoted
H:MM value (hour 10-23) as a sexagesimal integer instead of a string,
e.g. fetch_time: 10:00 -> 600, breaking HA's time validator. Document
that these values must be quoted (e.g. fetch_time: "10:00") to avoid
the pitfall, since this is a parser characteristic outside the
integration's control.

Refs #5065

* feat(source): add Timrå kommun waste collection (timra_se) (#6894)

Adds support for Timrå kommun, Sweden, by querying the municipality's
public ArcGIS FeatureServer directly by property address. Resolves the
current FeatureServer URL dynamically via the parent Web Map item so
the source keeps working when the date-stamped layer is refreshed for
a new year.

Closes #5541

* Update generated documentation

* Add source for Waikato District Council, New Zealand (#5226) (#6898)

Queries the council's public rubbish-and-recycling address search API
and generates 26 weeks of the combined weekly kerbside collection day.

Closes #5226

* Update generated documentation

* fix(abfallkalender_prezero_network): remove Willich, no longer served by PreZero (#6897)

PreZero's contract for Willich ended (Remondis, then Lankes Entsorgung took over
collection in 2026) and abfallkalender.prezero.network/willich now returns 404.
Willich is already covered by the existing abfallnavi_de source via Kreis Viersen.

* Update generated documentation

* Fix sutherlandshire_nsw_gov_au: council replaced lookup form; query public GIS layer (#6890)

* Fix sutherlandshire_nsw_gov_au: council replaced lookup form; use public GIS layer

The waste-information-booklet page the scraper drove was removed from the
council website (now redirects to a map-based page), and the site also
started rejecting plain library HTTP clients by TLS fingerprint, so every
fetch failed with 403.

The council's new 'When is my bin collected?' map is backed by a public
ArcGIS layer with one feature per property carrying DAY_OF_WEEK, ZONE_NO
and RECYCLE_PICKUP. Query that layer by address instead:

- exact situ1/situ2 attribute match, with a prefix fallback for unit
  numbers and house-number suggestions when the street matches
- schedule generation (weekly garbage, fortnightly recycling/garden by
  zone parity) unchanged from the previous implementation
- properties flagged RECYCLE_PICKUP=Weekly (some unit blocks) now get
  weekly recycling
- second TEST_CASE added

* Restore COUNTRY constant for sutherlandshire_nsw_gov_au

The GIS-layer rewrite dropped the required COUNTRY = "au" constant,
which would orphan the source out of the generated README/sources.json.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

---------

Co-authored-by: Benjamin <[email protected]>
Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* Add source for Gloucester City Council, UK (#6899)

Closes #2909

* Update generated documentation

* feat(portphillip_vic_gov_au): add source for City of Port Phillip, Victoria (#6900)

The council's public IntraMaps GIS widget returns a weekly bin-collection
day per address; general waste, recycling and FOGO are all collected
together on that day.

* Update generated documentation

* feat(ics): add Gemeinde Neuberg, Germany (#6901)

Neuberg publishes a single publicly accessible static ICS calendar
covering the whole municipality; add it to the generic ICS provider
list per issue #3646.

* Update generated documentation

* Add Stadt Tengen (Germany) as a new ICS provider (#6903)

Closes #3634

* Update generated documentation

* Add source for Haugaland Interkommunale Miljøverk (HIM), Norway (#6902)

Implements a scraper for the him.as tømmekalender address-search page,
covering Haugesund and surrounding municipalities. Closes #3654.

* Update generated documentation

* Add Gemeinde Hille (Germany) to ICS shared platform (#6904)

Hille publishes static per-district ICS collection calendars on its
own website; add them as a new entry in doc/ics/yaml so residents can
configure the generic ICS source directly. Closes #3526.

* Update generated documentation

* feat(source): Add SUIBR (Nidwalden, CH) waste collection (#6905)

* Update generated documentation

* Add source for Aarberg, Switzerland (#3655) (#6906)

Aarberg uses the same Weblication CMS as Rapperswil BE, with an
additional zone filter covering Aarberg, Grafenmoos, Leimern,
Mülital, Spins and Zälgli.

* Update generated documentation

* Add source: Předměřice nad Labem (predmerice_nad_labem_cz) (#6907)

* Update generated documentation

* feat(source): add De Afvalapp waste collection schedule (deafvalapp_nl) (#6909)

Adds support for De Afvalapp (deafvalapp.nl), covering Maashorst,
Helmond, Land van Cuijk and Boekel, requested in issue #5067.

* Update generated documentation

* Add source: East Suffolk Council (eastsuffolk_gov_uk) (#6910)

Closes #5183. Adds bin collection schedule support for East Suffolk
Council via their AchieveForms/Bartec-based "Bin collection dates
finder", following the same pattern as other AchieveForms sources
(e.g. tendring_gov_uk, eastcambs_gov_uk).

* Update generated documentation

* fix(communitywastedisposal_com): honor call-to-schedule flag and week-of-month ordinals (#6911)

HHW (and occasionally Bulk/Yard Waste) streams marked "call to schedule"
in CWD's route data were being shown as automatic weekly collections
instead of being omitted, and Monthly/BiMonthly streams with an
unspecified week-of-month silently defaulted to "every week" instead of
being skipped. Also adds a Keller, TX test case (#5072) and switches to
the canonical Icons enum.

* fix(mijnafvalwijzer_nl): scope date/type lookup to each year's own div (#6912)

The fetch loop iterated over each "jaar-XXXX" container but searched
dates/types against the whole page instead of the current year's
element, duplicating every collection date across all years present
on the page and mislabeling half of them with the wrong year
(e.g. around a year boundary when both the current and next year's
calendars are rendered simultaneously).

Fixes #5064

* Add source: GOAP Poznań (goap_poznan_pl) (#6913)

Adds support for GOAP (Związek Międzygminny Gospodarka Odpadami
Aglomeracji Poznańskiej), the waste-management union covering
Murowana Goślina, Czerwonak, Swarzędz and other municipalities in the
Poznań agglomeration, closing #5062.

* Update generated documentation

* feat(fermanaghomagh_gov_uk): add Fermanagh and Omagh District Council source (#6915)

Closes #5162

* Update generated documentation

* feat(source): add Southend-on-Sea City Council waste collection (southend_gov_uk) (#6916)

Closes #4852. Uses the existing Cloud9Client service (already backing
arun_gov_uk, northherts_gov_uk, rugby_gov_uk) against the "southend"
authority, which powers the council's new Southend Waste app.

* Update generated documentation

* feat(source): add Surf Coast Shire, VIC, AU (surf_coast_vic_gov_au) (#6917)

Adds support for Surf Coast Shire's kerbside bin collection via the
same public WhatBinDay API already used by the SCRRApp mobile app,
reusing the shared WhatBinDayService (as used by Kingston and Lismore).

Closes #4971

* Update generated documentation

* Add source: Gemeinde Muttenz (muttenz_ch) (#6918)

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* Add source: City of Tallahassee (talgov_com) (#6919)

Implements weekly Garbage/Recycling and biweekly Red/Blue Bulky
Items/Yard Waste collection lookup for the City of Tallahassee, FL,
using the public JSON endpoints behind talgov.com's address-lookup
widget (closes #5496).

* Update generated documentation

* refactor logan city council to use opencities API (#6920)

* Update generated documentation

* Add source: Gemeinde Kriftel (kriftel_de) (#6923)

Closes #3401

* Update generated documentation

* fix: correct 7 wrong council website domains in impactapps_com_au (#6921)

Six entries had the wrong Australian state suffix (Brisbane pointed
to .nsw instead of .qld; Campbelltown, Canada Bay, Cumberland, Port
Stephens, and Wollongong pointed to .vic instead of .nsw) and Lithgow
pointed to a lithgow.nsw.gov.au domain that doesn't resolve — the
council's real site is council.lithgow.com.

The `url` field (used for the actual waste-info.com.au API calls) was
unaffected; only `website`, the reference link to each council's own
site, was wrong. Found while building an attribution feature in a
downstream app that surfaces this field to end users. Verified every
corrected domain resolves and is the right council before fixing —
cross-referenced against Wikipedia for the ones that didn't respond
directly (network/geo restrictions on my end, not the domains
themselves).

* Update generated documentation

* fix: add Le Cotentin (Cherbourg-en-Cotentin) to publidata_fr source (#6928)

Closes #5129

* Update generated documentation

* fix(ecoharmonogram_pl): disambiguate single/multi-family schedules for multi-id streets (#6924)

Zabrze (and possibly other towns) bundle single-family and multi-family
housing schedules under one street entry with two comma-separated ids
whose `sides` field is identical, so users got both schedules merged
into one calendar. Fall back to the housing-type letter in the
schedule-name prefix and require additional_sides_matcher to pick one
when this ambiguity is detected, while preserving legacy merge
behavior for towns where multiple ids are genuinely meant to combine.

Fixes #6922

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* Add source for Stadt Kerpen (via abfall_io) (#6930)

* Update generated documentation

* feat(omrin_nl): add Omrin source (#6914)

* feat(omrin_nl): add Omrin source

* Address Omrin review feedback

* Address follow-up Omrin review feedback

* Update generated documentation

* Add source: Perth and Kinross Council (pkc_gov_uk) (#6933)

Closes #4529. Implements the AchieveForms-based "Bin collection dates"
lookup at my.pkc.gov.uk, returning general/recycling/paper/food-garden
waste collection dates by UPRN.

* Update generated documentation

* feat: add shared service/OpenCities.py client for MyArea/OpenCities councils (#6934)

37 source files independently hand-roll the same "OpenCities"/"MyArea"
council-CMS flow (api/v1/myarea/search -> ocapi/Public/myarea/wasteservices)
with drifting quality: inconsistent curl_cffi bot-protection, bare
Exception raises, and only 1-2 sources implementing geolocation-id
caching or ambiguous-address-match suggestions.

This is batch 0 of a phased migration (see the execution plan): land the
shared OpenCitiesConfig/OpenCitiesClient client and its unit tests first,
with no source files touched yet, so the abstraction gets reviewed before
33 follow-on sources depend on its shape.

Co-authored-by: Claude Sonnet 5 <[email protected]>

* feat: add extra info for cities in RegioEntsorgung source (#6945)

* Update generated documentation

* refactor: migrate 6 OpenCities sources onto shared service/OpenCities.py (batch 1/5) (#6946)

* refactor: migrate 6 OpenCities sources onto the shared service/OpenCities.py client

Batch 1 of the phased OpenCities/MyArea consolidation (see PR #6934):
kiama_nsw_gov_au, shoalhaven_nsw_gov_au, ballina_nsw_gov_au,
cassowarycoast_qld_gov_au, lanecove_nsw_gov_au, willoughby_nsw_gov_au all
now delegate to OpenCitiesClient instead of hand-rolling the
search/wasteservices flow.

Also extends the shared client based on real-world findings while
migrating these six:
- Some council HTML nests a matching <article> inside a matching
  div.waste-services-result (confirmed live on ballina and
  cassowarycoast), which the client's original selector double-counted.
  Fixed by only keeping the outermost match of a nested pair.
- Added OpenCitiesConfig.require_date_precise for Shoalhaven, which mixes
  precise-date and vague/recurring blocks in the same response and only
  wants the former (a genuine behavioral difference, not just style).
- lanecove and willoughby now 403 under plain requests (confirmed via a
  raw HTTP replay of their pre-migration code) and need curl_cffi, per
  the project's existing "403 from a Cloudflare/Akamai site -> curl_cffi"
  convention; bundled that fix into the migration.

kiama's two TEST_CASES geolocation IDs return "no collection services
found for this address" from the live API regardless of client
implementation (verified against the exact pre-migration HTTP request) --
pre-existing stale test fixture data, not a regression from this change.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* change kiama testcases

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>

* refactor: migrate 6 more OpenCities sources onto the shared client (batch 2/5) (#6951)

hobartcity_com_au, moorabool_vic_gov_au, banyule_vic_gov_au,
cambridge_wa_gov_au, mildura_vic_gov_au, and logan_qld_gov_au now delegate
to OpenCitiesClient. These six were chosen because they collectively
exercise every remaining piece of client-side variance not covered by
batch 1: curl_cffi, the geolocation_id bypass kwarg, a non-default
date_format, and hand-rolled caching/ambiguous-match handling that the
shared client's defaults now subsume.

Client extensions added while migrating, based on real API responses
(not just the plan's sketch):
- OpenCitiesConfig.strip_type_suffixes, used by moorabool to strip the
  "collection" suffix some labels carry.
- OpenCitiesConfig.warm_up_before ("search" | "wasteservices"): banyule
  and cambridge fire their warm-up GET *after* resolving a geolocation id
  (whether via search or the direct bypass) and *before* the wasteservices
  call, not before search as the client originally assumed -- the ordering
  needed to be configurable, not hardcoded.
- banyule and cambridge's search endpoints return XML instead of JSON
  unless an explicit `Accept: application/json` header is sent (confirmed
  via a raw HTTP replay); added that header to both configs rather than
  extending search_response_format, since it's an Accept-header quirk,
  not a deployment that only ever emits XML.

Verified via live test_sources.py -l runs that Cambridge's apparent
duplicate "General Waste" entry for one test address is genuine upstream
duplicate data (confirmed by inspecting the raw wasteservices HTML: the
API itself returns two identical blocks), not a client regression --
pre-migration code would have produced the same duplicate.

Co-authored-by: Claude Sonnet 5 <[email protected]>

* refactor: migrate 7 mechanical-family OpenCities sources (batch 3a/5) (#6953)

mansfield_vic_gov_au, maroondah_vic_gov_au, melton_vic_gov_au,
moyne_vic_gov_au, mrsc_vic_gov_au, nillumbik_vic_gov_au, and
stonnington_vic_gov_au all previously copy-pasted the same
warm-up-GET-then-search-then-wasteservices flow with a bare `raise
Exception(...)` on no results; they now delegate to OpenCitiesClient.

Fixes a real design flaw in the shared client found via live-testing,
not an isolated bug: OpenCitiesConfig.strict_address_matching defaulted
to True, meaning any multi-result search that didn't textually match the
query's AddressSingleLine byte-for-byte (e.g. the server appends a state
abbreviation the user's TEST_CASES address omits, as happened for both
moyne and maroondah) raised a hard "ambiguous" failure instead of using
the search API's own top-ranked (and, in both cases, correct) result.
That default punished the common case -- an ordinary top-hit resolution
-- to guard against a rare one. Flipped the default to False (trust the
top search result, matching how the vast majority of these ~20 councils'
original code already behaved) and made strict matching an explicit
opt-in, set on hobartcity_com_au and moorabool_vic_gov_au (#6951) since
those two are the only sources whose original hand-written code actually
implemented this disambiguation themselves.

Also confirmed two pre-existing, migration-unrelated outages while
live-testing this batch (documented, not fixed here):
- mansfield.vic.gov.au has moved off OpenCities entirely onto a Drupal 11
  site; /api/v1/myarea/search now 404s regardless of implementation.
- melton.vic.gov.au's wasteservices endpoint returns {"success": false}
  even when replaying the exact pre-migration request sequence (address
  search itself succeeds and returns a valid geolocation id).

Co-authored-by: Claude Sonnet 5 <[email protected]>

* refactor: migrate 7 more mechanical-family OpenCities sources (batch 3b/5) (#6955)

wangaratta_vic_gov_au, whittlesea_vic_gov_au, yarra_ranges_vic_gov_au,
mornpen_vic_gov_au, midcoast_nsw_gov_au, goldcoast_qld_gov_au, and
monash_vic_gov_au now delegate to OpenCitiesClient. goldcoast's migration
incidentally fixes a pre-existing "Https://" (capital H) typo in its
wasteservices URL.

Two new client extensions, both found in more than one source in this
batch (not one-off, so added to the shared client rather than kept
per-source):
- OpenCitiesConfig.exclude_types: yarra_ranges and mornpen both mix an
  unrelated "Burning off" permit article into the wasteservices response
  and need it dropped.
- OpenCitiesClient._parse_date now falls back to resolving "Every
  <Weekday>" recurring-service text (no concrete next date given at all)
  to the next occurrence of that weekday, needed by midcoast_nsw_gov_au;
  midcoast also keeps a local post-filter to the known waste types,
  matching its original allowlist behaviour (the wasteservices response
  includes non-waste informational articles too).

Confirmed two more pre-existing, migration-unrelated issues while
live-testing (documented, not fixed):
- mornpen's three TEST_CASES addresses (all in the "<number> <Name> Rd
  <Suburb> VIC <postcode>" format) return zero search results against
  the live API today; replaying the exact same request confirms this is
  independent of implementation. Dropping "VIC <postcode>" and spelling
  out "Road" both return real matches, so this looks like an
  abbreviation-sensitive search index change on the council's end, not a
  broken address.

Co-authored-by: Claude Sonnet 5 <[email protected]>

* refactor: migrate final 2 mechanical-family OpenCities sources (batch 3c/5) (#6957)

lakemac_nsw_gov_au and onkaparingacity_com now delegate to
OpenCitiesClient, completing the mechanical old-style family.

Both previously paired waste-type <h3> tags with next-service dates via
two separate find_all() passes zipped together positionally -- a latent
correctness bug (misaligns type/date if any block is skipped, e.g. a
non-precise date filtered out of one list but not the other) that the
client's per-block extraction doesn't have. Both also had
`if addresses == 0: raise SourceArgumentNotFound(...)` guards that could
never actually trigger (comparing a JSON dict to int 0), meaning an empty
search result would previously crash with an IndexError instead of the
intended error -- the client's real "no items" check fixes this too.

New client extension: OpenCitiesConfig.exclude_type_prefixes, needed by
onkaparingacity to drop "Calendar"-prefixed entries (calendar-download
links, not real collections, with a varying zone-number suffix so an
exact-match exclude_types wouldn't catch them all).

Co-authored-by: Claude Sonnet 5 <[email protected]>

* fix(abfall_io): migrate Entsorgungsbetriebe Essen to v3 GraphQL API (#6961)

EBE Essen's legacy abfall.io key now returns 401; move the provider to
the abfall_io_graphql source with its new v3 key and point the legacy
source's error message at the supported replacement.

Fixes #4775

* Update generated documentation

* feat(source): add Abfallwirtschaft Altenburger Land to AWIDO service map (#6962)

Closes #4794

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* refactor: migrate final 6 OpenCities sources -- 4-field-address/XML family (batch 4/5) (#6960)

* refactor: migrate final 6 OpenCities sources -- the 4-field-address/XML family (batch 4/5)

blacktown_nsw_gov_au, campbelltown_nsw_gov_au, unley_sa_gov_au,
ryde_nsw_gov_au, horowhenua_govt_nz, and hume_vic_gov_au now delegate to
OpenCitiesClient. This is the last and trickiest batch: these all take
4 separate address fields (post_code/suburb/street_name/street_number,
or post_code/town/street_name/street_number for horowhenua) instead of
one address string, concatenated locally before calling
OpenCitiesClient.fetch(address=...); campbelltown/unley exercise the
JSON/XML search-response variance (json_then_xml and xml respectively).

New client capability: OpenCitiesClient.get_waste_services_html(), a
public method exposing the raw wasteservices HTML fragment (previously
only reachable internally via fetch_by_geolocation_id). hume_vic_gov_au's
`predict` feature needs each block's "note" div text (a fortnightly/
weekly hint) to extrapolate future dates, which the shared parser doesn't
expose -- hume still delegates address search/resolution to the client,
but does its own block-by-block parsing against the raw HTML only for
the predict=True path. While preserving that parsing, also fixed a real
crash: the original code referenced `weeks` unconditionally after an
if/elif that could leave it unset, raising UnboundLocalError whenever a
block's "note" text matched neither "fortnight" nor "same day each week";
it now safely falls back to the single resolved date in that case.

Two more curl_cffi/Accept-header fixes found via live-testing (same
"403 -> curl_cffi" and "no Accept header -> XML not JSON" patterns seen
in earlier batches, not new categories): unley_sa_gov_au and
horowhenua_govt_nz both now need curl_cffi, and horowhenua also needs an
explicit `Accept: application/json` header on its search request.

Confirmed one more pre-existing, unrelated outage: blacktown's "Plumpton
Marketplace" test address's wasteservices call returns {"success":
false} even replaying the exact pre-migration request -- the address
search itself succeeds and returns a valid geolocation id, but the
council's service-lookup step fails server-side for that specific
address today.

This completes the OpenCities/MyArea shared-service migration across all
6 batches (0, 1, 2, 3a, 3b, 3c, 4).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* remove non working testcase

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>

* feat(source): add Seuzach (seuzach_ch) waste collection schedule (#6963)

Live-parses both the special-collections table and the linked
regular-collection detail pages on seuzach.ch/abfalldaten, so the
weekly Kehrichtabfuhr/Grünabfuhr weekday and Grünabfuhr season are
discovered at runtime rather than hardcoded.

Closes #4685

* Update generated documentation

* Republic Services: fix holiday schedule adjustments (#6964)

* Republic Services: fix holiday schedule adjustments

The v3 holiday endpoint returns an empty payload, and the delay lookup
never matched the API's description string, so holidays were never applied.
Switch to the v2 holiday endpoint and handle One Day Delay / Not Running /
Service Moved, matching holidays to pickups by line of business.

* Republic Services: run ruff format on holiday-adjustment fix

Co-Authored-By: Claude Sonnet 5 <[email protected]>

---------

Co-authored-by: carefulcomputer <[email protected]>
Co-authored-by: Benjamin <[email protected]>
Co-authored-by: Claude Sonnet 5 <[email protected]>

* fix: remove rotherham_gov_uk source (backend no longer provides structured data) (#6966)

Rotherham council's bin-collection page dropped its per-UPRN table
lookup in favour of generic PDF calendars and a mobile app, so the
source can no longer fetch personalized schedules. Rotherham is
already fully covered by the apps_imactivate_com source, which two
users in the issue confirmed works as a drop-in replacement.

Fixes #4721

* Update generated documentation

* Add Wirtschaftsbetriebe Duisburg (WBD) to abfall.io GraphQL source (#6968)

WBD Duisburg's online waste calendar loads the abfall.io v3 widget
with service key 80acad6c77fe9342ebafad29a8c58bf6, so it's already
covered by the existing abfall_io_graphql source/service map.

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* feat(source): add KELL GmbH (Landkreis Leipzig) to Abfall.IO GraphQL service map (#6969)

Also fix Origin/Referer handling so providers whose abfall.io v3 API key
enforces an Origin allowlist (like KELL) don't get a 401 on the GraphQL
appointments query.

* Update generated documentation

* feat: add Montri (montri.fr) source (#6965)

* feat: add Montri source for French waste collection schedules (#5051)

Implements the Montri platform (montri.fr) used by SICOVAD and other
French intercommunal waste syndicates, via its anonymous-session JSON
API (city lookup, address search, and monthly collection calendar).

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

* fix: add timeout to API requests in Montri source

* Update generated documentation

* feat: add Clackmannanshire Council (clackmannanshire_gov_uk) source (#6967)

* feat: add Clackmannanshire Council (clackmannanshire_gov_uk) source

Closes #4997. Fetches bin collection dates via the council's public
postcode/address lookup and the per-property .ics calendar feed
linked from the property details page.

* fix: add timeout to HTTP requests in Clackmannanshire source

* Update generated documentation

* docs(ics): add Kelkheim (Taunus) to generic ICS source documentation (#6971)

Kelkheim's mod_abfallkalender exposes a public, unauthenticated ICS feed
already supported by the generic ICS source; add the doc/ics/yaml entry
so it is discoverable, closing the gap left after issue #4245's comment.

* Update generated documentation

* feat(source): add Irado (irado_nl) waste collection source (#6973)

* feat: add Irado (irado_nl) source for Vlaardingen and other municipalities

Closes #4344

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

* feat(source): add Popůvky (popuvky_cz) waste collection schedule (#6972)

* feat(source): add Popůvky (popuvky_cz) waste collection schedule (#4441)

Adds support for the Obec Popůvky waste collection calendar, covering
both collection routes (Popůvky village and the cottage/recreational
area), parsed from the site's public AJAX calendar widget.

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

* Update generated documentation

* Add Uddevalla Energi source (#6974)

* Update generated documentation

* feat: add various agents and skills for waste collection management, including cleanup, new-source, release, issue review, and PR review functionalities (#6975)

* Add Älvsbyns Energi source (#6976)

* Update generated documentation

* Add Peoria (IL) to RecycleCoach EXTRA_INFO (#6989)

Peoria, IL already uses the RecycleCoach "My Schedule" widget on its
official site; add it to the shared source's EXTRA_INFO list and a
TEST_CASES entry using the example address from #5019.

* Update generated documentation

* fix(publidata_fr): add Sud Sainte Baume (Saint-Cyr-sur-Mer) to Publidata source (#6990)

Adds the missing EXTRA_INFO/TEST_CASES entry for instance_id 1483. The
"list index out of range" crash originally reported was already fixed
by #5896; this closes out the remaining ask (listing the municipality)
and adds a label/icon mapping for the "jrm" (paper/magazines) waste type
seen in this instance's data.

* Update generated documentation

* fix(montreal_ca): fall back to Green schedule when Food message has no weekday (#6991)

Some Montreal sectors (e.g. PIRO-1, Pierrefonds-Roxboro) report Food
waste collection as merely "included in the collection of organic
waste" with no weekday of its own, causing Food entries to silently
be dropped. Reuse the matching Green (organic) sector's message,
which does carry the collection day, and add a regression test case
for the reported sector.

* Update generated documentation

* fix(carmarthenshire_gov_wales): handle UPRNs with no collection data (#6992)

The council returns a container without a date paragraph when it has
no record for a UPRN, causing an unhandled IndexError ("list index
out of range"). Skip such containers and raise a clear
SourceArgumentNotFound instead.

Fixes #5256

* feat(source): add GLØR (glor_no) waste collection for Innlandet, Norway (#6993)

Implements a source for GLØR, covering Lillehammer, Øyer, Ringebu and
Gausdal municipalities, using the public proaktiv.glor.offcenit.no
search/details JSON API. Closes #4487.

* Update generated documentation

* Update Lichfield source (#6986)

* Update lichfielddc_gov_uk.py

Updating Lichfield to accommodate new web layout and weekly food caddy collections.

* Update lichfielddc_gov_uk.py

Source formatting for test pass

* Update lichfielddc_gov_uk.py

More test formatting pedantry...

* Update lichfielddc_gov_uk.py

Spacing update

* Update lichfielddc_gov_uk.py

* Update lichfielddc_gov_uk.py

ruff'd

* Update lichfielddc_gov_uk.py

spacing

* Update lichfielddc_gov_uk.py

"Sack" to "Bag" to match source.

* Add Gargždų švara source for Klaipėda district, Lithuania (#6994)

Closes #4419

* Update generated documentation

* Add SIAVED (Valenciennes Métropole) to Publidata FR source (#6995)

SIAVED runs on the Publidata platform already supported by
publidata_fr.py; add it as a new EXTRA_INFO entry with a live
test case, closes #4666.

* Update generated documentation

* Add Stadtwerke Aschaffenburg to Gipsprojekt ICS platform; remove stale MyMuell listing (#6996)

Aschaffenburg's MyMuell (Junker) calendar has returned zero collection
dates for every street since 2024 (confirmed live against the MyMuell
API), so drop the dead city listing from jumomind_de.py. The city's
utility, Stadtwerke Aschaffenburg, now publishes the schedule via the
same "GIPS" system already documented as the shared Gipsprojekt ICS
platform (used e.g. by Stadtwerke Speyer), so document it there instead
of writing a new dedicated source.

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* Add City of Raleigh, NC to ReCollect shared-platform list (#6997)

Raleigh's "Raleigh Reuse" waste tool (raleighnc.gov) is powered by
the ReCollect platform (area CityofRaleighNC), already supported
via the generic ICS source. Closes #4265.

* Update generated documentation

* fix(abfall_io_graphql): add Landkreis Göttingen to v3 GraphQL service map (#6998)

Landkreis Göttingen uses the abfall.io v3 API with a flat idHouseNumber
(no kommune/strasse hierarchy), which the legacy abfall_io source
cannot drive. Add it to the GraphQL source's SERVICE_MAP so it is
discoverable via EXTRA_INFO/config wizard, fixing #4125.

* Update generated documentation

* Add AWB Birkenfeld (Landkreis Birkenfeld) source (#7001)

Fetches the embedded street/collection-group/date JSON from the AWB
Birkenfeld waste calendar page to support all municipalities in
Landkreis Birkenfeld, Germany.

Closes #3762

* Update generated documentation

* Add West Kelowna (BC) to recyclecoach_com source (#7003)

West Kelowna shares the same RDCO/Recycle Coach project (502) as the
existing Kelowna (BC) entry, distinguished only by district_id.

Closes #7002

* Update generated documentation

* fix(abfall_io): migrate ASG Nordsachsen to v3 GraphQL API (#7005)

ASG Nordsachsen's legacy abfall.io key now returns 401; move the
provider to the abfall_io_graphql source with its new v3 key found
in the live widget embed on asg-nordsachsen.de.

Fixes #4335

* Update generated documentation

* Add Epsom and Ewell Borough Council to iapp_itouchvision_com source (#4388) (#7006)

* Update generated documentation

* fix(abfall_io_graphql): add Landkreis Böblingen to v3 GraphQL service map (#7007)

Abfallwirtschaft Landkreis Böblingen (awb-bb.de, covering Böblingen and
its Dagersheim district) uses the abfall.io v3 GraphQL widget. Add it to
the GraphQL source's SERVICE_MAP so it is discoverable via
EXTRA_INFO/config wizard, closing #3549.

* Update generated documentation

* Add RIR (Romsdalshalvøya Interkommunale Renovasjonsselskap) source (#7009)

Fetches collection dates from the public address-lookup and
per-month calendar HTML endpoints on rir.no, covering Aukra,
Hustadvika, Gjemnes, Molde and Rauma, Norway.

Closes #4301

* Add Czysty Region source (Opole/Silesia region, Poland) (#7010)

Fetches waste collection schedules for the 15 communes served by
Związek Międzygminny "Czysty Region" via the cascading gmina/rodzaj/
miejscowość lookup at czystyregion.pl/Harmonogramy, parsing the
embedded FullCalendar events JSON from the resulting schedule page.

Closes #7004

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* fix(abfallnavi_de): don't abort fetch when a street name resolves to multiple split segments (#7011)

Some AbfallNavi providers (e.g. Solingen) split long streets into
several entries with the identical name but disjoint house-number
ranges, keyed by city district. get_dates() previously aborted the
whole fetch as soon as the first matched segment didn't contain the
requested house number, even if a later segment did — causing empty
calendars for any address on a split street (#4701).

* Add Gemeinde Mils (Tyrol) source (#7012)

Mils runs the same RiSKommunal platform already used by Fritzens and
other Tyrolean municipalities; reuses the shared RiSKommunalSource
base class for address-based (street + house number) lookup.

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* Add Gemeinde Rödinghausen to Abfall Export (vCal) ICS provider (#7013)

Closes #3547

* Update generated documentation

* Add Gronau (regio iT Abfallkalender) source (#7014)

Closes #3585. Implements the kalender-wml regio iT Abfallkalender
backend for Gronau (Kreis Borken, NRW): looks up the street's numeric
ID from the session-scoped street list, then downloads the ICS feed
for the selected waste fractions.

* Update generated documentation

* Add Gemeinde Kirchlengern (kirchlengern_de) source (#7015)

Resolves the street name to an internal ID via the municipality's
calendar widget, then fetches and parses the per-year vCal/ICS export.
The feed double-encodes non-ASCII waste-type labels; a strict
encode/decode round-trip repairs them without corrupting text that is
already correct.

Closes #3574

* Update generated documentation

* Add Senden to MyMuell (Jumomind) source (#7016)

Senden (Bavaria) is already served by the MyMuell App backend but was
missing from the provider's documented city list.

Closes #4162

* Update generated documentation

* Add millvalleyrefuse source (#7018)

* Add Mill Valley Refuse Service source

Implement source for Mill Valley Refuse Service with ICS calendar integration for waste collection schedules.

* Create documentation for Mill Valley Refuse Service

Added documentation for Mill Valley Refuse Service, including recycling schedules and configuration instructions.

* Update generated documentation

* Add Schwarzwald-Baar-Kreis (Königsfeld) to Abfall.IO / AbfallPlus (GraphQL) (#7031)

Resolves a source request for Königsfeld, Baden-Württemberg, which is
served by the Schwarzwald-Baar-Kreis waste-management office via the
abfall.io v3 widget already supported by the abfall_io_graphql source.

* Update generated documentation

* Fix: North Kesteven, UK — use curl_cffi to avoid Cloudflare blocks (#7032)

North Kesteven's bin lookup endpoint intermittently enables
Cloudflare's bot-check, which blocks plain requests sessions and
causes fetch() to silently return no entries. Switch to curl_cffi
with browser impersonation (the project's documented mitigation for
Cloudflare-protected sites), and make icon lookup match on the
leading colour keyword since the council has changed some bin type
label wording (e.g. "Orange Lidded Caddy") since the source was
first written.

* Add City of Stamford, CT source (stamford_ct_us) (#7033)

Adds a source for Stamford, CT's garbage/recycling pickup-day lookup
(stamfordapps.org), resolving addresses by street name and an optional
house number for streets with multiple pickup-day ranges.

* Update generated documentation

* fix(app_abfallplus_de): stop leaking other municipalities' Giftmobil dates for Landkreis München (#7034)

The per-municipality "Sondermüll" skip heuristic in AppAbfallplusDe only
matched that literal heading text, so apps whose provider labels the same
per-town toggle list "Giftmobil" (e.g. de.k4systems.lkmabfallplus) fell
through to a generic fallback that swept up every municipality's toggle id
indiscriminately. The per-entry name match was also comparing against the
<input>'s (always empty) text instead of the sibling <ion-item> label, so
even the "Sondermüll" case never actually selected the searched town - it
just got lucky by dropping the whole section instead of leaking.

Fixes #7026

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Add San Giovanni Teatino to Junker APP source (#3780) (#7035)

Junker already serves this municipality generically; list it in
SERVICE_PROVIDERS and add live-test coverage for its zone-based
collection schedule.

* Update generated documentation

* Add Nantes Métropole (data.nantesmetropole.fr) source (#7036)

Closes #3779

* Update generated documentation

* Add Marktgemeinde Pernitz (Austria) source (#7037)

Fetches Restmüll (by Rayon 1/2), Biotonne, Gelber Sack, Gelber
Container, and Papier collection dates from pernitz.gv.at.

Closes #3741

* Update generated documentation

* Prevent a stale/renamed source argument from crashing the whole integration (#7038)

Source instantiation failures (e.g. after a source's constructor
arguments changed, a failed config-entry migration, or a `customize`
block accidentally nested under `args`) were raised uncaught out of
SourceShell.create(). For YAML setups this aborted the entire
component (every configured source went down), and for UI config
entries it surfaced as a confusing TypeError instead of a clear
"setup failed" reason.

Fixes #3580

* Add Kansas City, MO source (#7043)

Geocodes the address via the ArcGIS World Geocoder, then queries the
city's "Trash Route Boundary" MapServer layer (TRASHDAY field) for the
weekly collection day. Trash and recycling are collected on the same
day city-wide, so recycling is derived rather than queried separately.
Holiday-shifted dates are out of scope since KCMO's GIS exposes no
holiday data.

Co-authored-by: Claude Sonnet 5 <[email protected]>

* Update generated documentation

* Add siunet it (#7039)

* add siunet_it source

* Ignore codespell on italian strings

* Update generated documentation

* Add sludge (Slam) week-number pickup parsing to edpevent_se (#2355)

Fall back to WastePickupFrequency (e.g. "Vecka 27") when NextWastePickup
is empty and derive a date from the ISO week number. Keep the existing
"v32 Aug 2024" form as an explicit branch and log a warning instead of
crashing on unparseable values.

Rebased onto current master and removed leftover debug prints per review.

Co-authored-by: Benjamin <[email protected]>
Co-authored-by: Claude Sonnet 5 <[email protected]>

* feat(source): add KVA Thurgau (CH) covering 70 Thurgau communities (#6952)

* feat(source): add KVA Thurgau (kvatg_ch) covering 70 Swiss communities

Parses the per-community Entsorgungsplan PDFs from kvatg.ch (canton
Thurgau plus neighbouring SH/SG communities) using pdfminer word
coordinates:

- month-grid date extraction incl. per-village/district zone rows,
  superscript zone markers, glued footnote digits, and year-rollover
  columns (DEZ | JAN '27)
- weekly Kehricht expansion from the stated Abfuhrtag, applying
  explicitly listed holiday replacement dates
- weekly Grüngut season spans ("Von Anfang März bis Ende November" /
  "Vom 6. März bis 27. November wöchentlich" incl. cancellations)
- calendar-week rules (Frauenfeld "KW 2/4/6/8/10, KW 11-52" weekday
  zones)
- inline street-collection dates (Metalle, Häckseldienst, Christbaum)

Optional zone parameter filters zoned communities; without it, zoned
collections are suffixed with their zone name. Verified against all 70
community PDFs (2026 plans).

Closes #3738
Closes #4918

Co-Authored-By: Claude Fable 5 <[email protected]>

* docs(kvatg_ch): correct Kehricht holiday-shift note

The highlights bullet still said holiday shifts are not applied, which
contradicts the parser (explicit replacement dates are now honoured) and
the Known limitations section. Reword for consistency.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* Ignore codespell false positives in kvatg_ch translations

"tre" (Italian "three") and "invalide" (French "invalid") are correct
words in the localised PARAM_DESCRIPTIONS, not typos. Add them to
.codespellignore so the validate check passes, matching the existing
foreign-word entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* Revert "Ignore codespell false positives in kvatg_ch translations"

This reverts commit e56d17a95aeb8a7b6471dabf70be12332da7999a.

* Ignore codespell false positives in kvatg_ch translations

"tre" (Italian "three") and "invalide" (French "invalid") are correct
words in the localised PARAM_DESCRIPTIONS, not typos. Add them to
.codespellignore alongside the existing foreign-word entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Co-authored-by: Claude Fable 5 <[email protected]>
Co-authored-by: Benjamin <[email protected]>
Co-authored-by: Mark van Proctor <[email protected]>

* Update generated documentation

* Test against the supported Home Assistant floor (#6987)

* fix: repair master CI (update-docs Python, kvatg_ch translation placeholders) (#7060)

Two regressions surfaced on master after recent merges:

1. Update Documentation workflow failed: #6987 raised the requirements to
   homeassistant>=2024.4.0 (Python >=3.11) but update-docs.yaml still ran
   Python 3.10, so HA could not be installed. Bump it to 3.12, matching
   validate.yaml.

2. hassfest failed: kvatg_ch's PARAM_DESCRIPTIONS used markdown links
   ([label](url)), which the doc generator turned into HA translation
   placeholders that are not valid identifiers. De-link them to plain
   label + bare URL (bare URLs, as already used in this source's HOWTO,
   do not generate placeholders).

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* Update generated documentation

* fix(kvatg_ch): drop URLs from PARAM_DESCRIPTIONS to avoid config-form placeholders (#7061)

update_docu_links.py extracts any URL (bare or markdown) from a param
description into a {url_...} placeholder. For config-flow data_description
those placeholders are never defined at runtime, so the form would show the
literal {url_...} text (plus a stray paren). #7060 fixed the invalid-identifier
hassfest error but left this cosmetic issue.

Reference the sites by bare domain name (kvatg.ch, kreuzlingen.ch) instead;
the full URLs remain in HOW_TO_GET_ARGUMENTS_DESCRIPTION, which the generator
does not placeholder-ise. Verified extract_urls_from_text now yields no
placeholders for any language/field.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* Update generated documentation

* Test minimum and current Home Assistant runtimes in CI (#6999)

* Test against the supported Home Assistant floor

* Test minimum and current Home Assistant runtimes

* Keep minimum Home Assistant lane importable

* Allow retry after failed source fetch (#6983)

* fix(config_flow): render the sensor selector as a list, not a dropdown (#6627)

The options-flow sensor selector rendered as a dropdown overlay that overflowed the viewport once a config entry had many sensors. That selector has a fixed set of choices (custom_value=False), so SelectSelectorMode.LIST gives a bounded, scrollable list box instead.

The custom-value selectors stay on DROPDOWN: Home Assistant's frontend ignores LIST mode whenever custom_value is set, so free-text entry of an unlisted waste type still works.

* Adding Carleton Place, Ontario (#7062)

* Adding Carleton Place, Ontario

* Add Carleton Place TEST_CASE and icon mappings

Cover the new CARLETON district with a live TEST_CASE using the Town Hall address so the nightly source tests exercise it, and map the waste types Carleton Place returns to the Icons enum.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Mark van Proctor <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

* Update generated documentation

* Fix centralbedfordshire new site (#7063)

* Fix centralbedfordshire_gov_uk: adapt to new LocalGov Drupal site

Central Bedfordshire Council migrated their waste collection page to
the LocalGov Drupal Waste Collection module, which uses a different
URL flow (postcode search -> UPRN lookup -> direct view by UPRN) and
a different page structure than the old scraper expected.

- Replace the old multi-step POST/session flow with two plain GETs
- Parse dates from the <time datetime> attribute instead of text
- Split the combined 'X, Y and Z collections' sentence into
  individual Collection entries per bin type
- Deduplicate bin types per date, since some properties (e.g. those
  with a separate garden waste subscription) list the same date
  across two <li> rows

* Apply ruff formatting

* Fix centralbedfordshire_gov_uk error handling and docs

- Raise SourceArgumentNotFound for an unrecognised postcode instead of a
  page-structure ValueError: the site returns 200 with no address dropdown
  when the postcode is not found, so the old message misattributed the
  most common user error.
- Trim developer-facing instructions out of the remaining parse error.
- Restore the "postcode without space" test coverage the old cases had,
  on the address that also exercises the garden-waste deduplication.
- Update doc/source/centralbedfordshire_gov_uk.md for the new site: the
  old /info/163/ URL now redirects, and the house_name guidance described
  the old option format.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Mark van Proctor <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

* Added geminfo_app.yaml to the ICS YAML folder to support Thannhausen … (#7040)

* Added geminfo_app.yaml to the ICS YAML folder to support Thannhausen in Austria.

* Added the geminfo_app.md file.

* Corrections made according to the review comments from 23.07.2026. YAML modified. Please check and approve the changes.

* Delete .vscode/settings.json

Removed from commit

* fix(7040): correct geminfo_app ICS provider metadata

- add YAML document start marker so the yamlfmt hook passes
- use the `at` country code instead of `Austria`, which would have
  orphaned the provider from README/info/sources.json
- add an extra_info entry so Thannhausen is listed by name
- drop doc/source/geminfo_app.md; ICS providers generate doc/ics/*.md
- restore .vscode/settings.json, deleted unintentionally

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Mark van Proctor <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

* Update generated documentation

* Fix Esch schedule fetch with identity encoding (#7075)

The Esch endpoint's compressed response advertises Transfer-Encoding: chunked twice. Some urllib3 versions reject it as multiple Transfer-Encoding headers, preventing schedule refreshes.

Use the canonical trailing-slash endpoint and request identity encoding. This avoids the malformed compressed response without changing user configuration. Add a regression test covering the request and parsed schedule.

* Add source provider for S.E.S.A. (issue #6870) (#6982)

* sesa_it: add source file boilerplate

* sesa_it: fetch pdf from user argument

* sesa_it/draft: PDF parsing

* sesa_it: working draft of a PDF parser

* sesa_it: add source provider

* sesaeste_it: fix date parsing, TLS chain, icon lookup and metadata

- datetime.date has no strptime; every fetch raised AttributeError
- app.sesaeste.it omits the RapidSSL intermediate, so requests and
  curl_cffi both fail certificate verification; use the same
  verify=False workaround as awn_de and others
- add COUNTRY, PARAM_TRANSLATIONS and HOW_TO_GET_ARGUMENTS_DESCRIPTION
- ICON_MAP.get() so an unmapped waste type cannot break the fetch
- extend the calendar window into next year so it does not run dry
  in December
- SourceArgumentNotFoundWithSuggestions instead of a bare not-found
- add request timeouts, drop noisy info logging, tidy imports
- add the missing doc/source/sesaeste_it.md

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Mark van Proctor <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

* Update generated documentation

* fix(aucklandcouncil_govt_nz): bypass WAF 406 and fail loudly on empty results (#7078)

* fix(aucklandcouncil_govt_nz): bypass WAF 406 and fail loudly on empty results

The source shipped a hard-coded Chrome/58 User-Agent (2017). Auckland
Council's WAF now answers HTTP 406 Not Acceptable to it on the per-address
collection page. fetch() never checked the response status, so the 406 error
page was handed to BeautifulSoup, no acpl-schedule-card was found, and an
empty list was returned - the coordinator logged a successful fetch with no
data and nothing appeared in the log.

The page markup itself is unchanged; only the request was being rejected.

- Use curl_cffi with impersonate="chrome" instead of a hand-written
  User-Agent, so the client does not go stale again and also matches on TLS
  fingerprint if the WAF tightens further.
- Call raise_for_status() so a future block surfaces as an error.
- Raise SourceArgumentNotFound when no collections parse. An unknown area
  number returns HTTP 200 with the cards rendered but every date field blank,
  which previously produced the same silent empty schedule.
- Extract parsing into a module-level parse_page() and move waste-type
  normalisation into a COLLECTION_TYPES map.
- new.aucklandcouncil.govt.nz now redirects to www.aucklandcouncil.govt.nz;
  update URL, the fetch URL and the doc links accordingly.

Fixes #7070

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix(aucklandcouncil_govt_nz): add request timeout and capitalise DESCRIPTION

Maintainer tweaks on top of the WAF fix:

- Pass timeout=30 to the curl_cffi request so a stalled response cannot
  hang an update cycle, matching toogoodtowaste_co_nz.py.
- Capitalise "Auckland Council" in DESCRIPTION, folded in from #7067.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Co-authored-by: Mark van Proctor <[email protected]>

* Update generated documentation

* Adds garden waste to herefordshire (#7073)

* Adds garden waste to herefordshire

* fix(herefordshire_gov_uk): correct ICON_MAP access and skip non-date garden entries

ICON_MAP is a dict, so ICON_MAP.General raised AttributeError on every
fetch and broke both test cases.

The garden waste heading is rendered for all properties, but properties
without a garden subscription get a calendar link instead of dates, which
crashed strptime and took the other two bins down with it. Non-date list
items are now skipped.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Mark van Proctor <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

* fix(sudestavenir_fr): parse mixed green-waste schedules (#7079)

* fix(sudestavenir_fr): parse bounded green-waste schedules

* docs(sudestavenir_fr): describe mixed green-waste schedules

The source doc still described dechets vegetaux as a precomputed date list only. Reflect that the precision text can also carry a bounded weekly rule, and note that bounded rules are not extrapolated into the following year.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: snowyukitty <[email protected]>
Co-authored-by: Mark van Proctor <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

* fix(swale_gov_uk): update form field IDs (#7071)

Swale Borough Council updated the form field IDs on the Check Your Bin Day page.

This updates both POST requests to use the current field names.

* Add Coteau-du-Lac ICS waste schedule (#7081)

* Add Coteau-du-Lac ICS waste schedule

* Fix Coteau-du-Lac ICS provider definition

- Correct the malformed test_cases block that failed
  test_ics_source_has_necessary_parameters
- Collapse the two duplicate test cases into one (the feed is French only)
- Rewrite the en/fr howto against the real portal instead of brossard.ca
- Fix the title capitalisation and drop the trailing slash from url

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

---------

Co-authored-by: Mark van Proctor <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

* Update generated documentation

* Release v2.32.0 (#7086)

Bump manifest to 2.32.0, add the 2.32.0 changelog section and backfill the 2.31.1 section that was tagged but never merged to master.

* test(v3): enforce component reuse in pipeline sources

A pipeline source should compose shared components, not define its own.
Behaviour written inside a source module is invisible to the next
provider on the same platform, so it gets rewritten slightly differently
and the two drift.

Adds test_pipeline_sources_reuse_shared_components, which fails when a
source declares its own Retriever or Parser subclass, with an allowlist
for genuinely single-consumer cases. 242 discoverable pipeline sources
pass today; regioentsorgung_de is the sole allowlisted entry.

Documents the rule in CLAUDE.md and in the anti-patterns section of
doc/contributing_source.md, calling out that it binds conversions as
much as new sources.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* chore(v3): report declarative migration separately from pipeline debt

arch_coverage now headlines sources that are pipeline AND fully declarative, and reports pipeline-with-own-steps as debt with a target of zero. Adds the cassette and step-override gates to the PR template.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix: convert southend_gov_uk to the v3 Cloud9 pipeline components

southend_gov_uk arrived from master (#6916) still on the legacy fetch()
contract, importing Cloud9Client from service/uk_cloud9_apps. v3
componentised that platform into Cloud9Retriever + Cloud9Parser, so the
module failed to import and broke collection of the whole test suite.

Re-implement it as a declarative BaseSource, matching arun_gov_uk (the
other UPRN-or-postcode Cloud9 council):

- retrieve = Cloud9Retriever("southend", ...), parse = Cloud9Parser(),
  transform = RowTransformer. No source-local step overrides.
- PARAMS = alternatives([uprn()], [postcode()]) plus an optional address
  field, replacing the hand-rolled SourceArgumentRequired check.
- ICON_MAP keyword guessing replaced by an explicit type_value_map onto
  canonical waste types. Southend collects mixed recycling (plastic,
  glass, cans) in the pink lid bin and paper and card in the blue lid
  bin, so those map to RECYCLABLES and PAPER respectively.
- RAISE_ON_EMPTY for the address lookup path.

TEST_CASES and the wire parameter names are unchanged, so existing user
configurations keep working. Waste-type labels change, which is the
expected breaking change for a pipeline migration.

Drop the hand-written doc/source/southend_gov_uk.md: doc_generator.py
renders it from the class metadata for pipeline sources.

Record offline fixtures for all three test cases.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 343c6b2028872a3e85bb79fafcdaae546aff2929)

* fix: convert yarracity_vic_gov_au to the v3 pipeline

The source came across from master (#6888) importing get_next_n_dates and
most_recent_weekday from service.ArcGis. Neither survived v3's ArcGIS
componentisation, so the module failed to import and broke collection of
the whole test suite.

Rather than restore those helpers, the source is now declarative on the
shared components:

- retrieve/parse: ArcGisMultiFeatureRetriever + ArcGisMultiFeatureParser
  (one geocode, a spatial query against the glass and waste layers, each
  response tagged with its label)
- preprocess: Compose(RecurrenceExpander, HolidayShift)
- transform: ICSTransformer onto the canonical waste types

The two missing helpers needed no platform work: most_recent_weekday
already exists in recurrence, and get_next_n_dates is Schedule(anchor=True)
via recurrence.recurring_from_anchor. The private weekday dict is gone in
favour of recurrence.weekday().

Adds recurrence.good_friday(), replacing hand-rolled easter-minus-two-days
arithmetic; armadale_wa_gov_au and subiaco_wa_gov_au duplicate the same
calculation and can adopt it later.

Behaviour, TEST_CASES and the codeowner are preserved: 52 weekly Rubbish
and FOGO, 26 fortnightly Recycling, 13 Glass on the layer's own cycle,
with Good Friday and Christmas Day deferred one day. Both test cases
return 143 collections live.

Also drops the hand-written doc page (pipeline sources generate it) and
records offline fixtures for both test cases.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit ae1acc292f145640098d24e22bad28215e2a03b0)

* port(abfall_io): carry master's provider changes onto the v3 pipeline

Ports the behaviour and data of three upstream PRs that landed on master
while abfall_io was being rewritten as a BaseSource pipeline source. The
v3 structure is kept throughout; none of master's legacy code shape is
carried across.

#6961 (Entsorgungsbetriebe Essen migrated to the v3 GraphQL API):
drop the Essen entry from the abfall.io provider registry, which on v3
is the _PROVIDERS list feeding Source.REGIONS rather than the service
module's SERVICE_MAP. Update the legacy 401 message to point users at
the abfall_io_graphql source; on v3 that message lives in the shared
AbfallIoRetriever, so every abfall.io source gets it.

#7005 (ASG Nordsachsen migrated to the v3 GraphQL API): drop the ASG
Nordsachsen entry from the same registry.

#6930 (Stadt Kerpen): convert the source that arrived from master to
this branch's abfall.io API. It now subclasses the pipeline Source and
declares only metadata, REGIONS, TEST_CASES, PARAMS and __init__, the
same shape rh_entsorgung_de uses. No retrieve, parse or transform
override: the shared AbfallIoRetriever, AbfallIoParser and ICSTransformer
do the work. PARAM_TRANSLATIONS, PARAM_DESCRIPTIONS and
HOW_TO_GET_ARGUMENTS_DESCRIPTION give way to field_terms labels on a
cascading_select plus HOWTO. This fixes the failing
test_params_fields_match_init[stadt_kerpen_de].

Kerpen pins the abfall.io key and kommune, so those never appear in the
config flow's selections and the inherited cascade could not resolve the
street or house-number dropdowns. Rather than a bespoke get_choices in
the source, abfall_io.Source gains a PINNED_PARAMS class attribute that
get_choices merges in, so any pinned-provider subclass stays declarative.

The matching GraphQL-side additions for Essen and ASG Nordsachsen belong
to abfall_io_graphql.py and AbfallIOGraphQL.py and are handled separately.

(cherry picked from commit 4c52a4e9c058a68711f5cbdf5a2c5a52b208d0da)

* port(abfall_io_graphql): carry master's provider additions and Origin fix into v3

Ports the behaviour and data of the master commits that landed on
abfall_io_graphql / AbfallIOGraphQL since the v3 branch point, keeping the
v3 pipeline structure.

Providers added to the source's _PROVIDERS registry (and so to REGIONS)
plus a TEST_CASES entry each:

- Entsorgungsbetriebe Essen (#6961)
- Wirtschaftsbetriebe Duisburg (WBD) (#6968)
- KELL Kommunalentsorgung Landkreis Leipzig GmbH (#6969)
- Landkreis Goettingen (#6998)
- ASG Nordsachsen (#7005)
- Abfallwirtschaft Landkreis Boeblingen (#7007)
- Amt fuer Abfallwirtschaft Schwarzwald-Baar-Kreis (#7031)

Behavioural fix from #6969: providers that enforce an Origin allowlist on
their key answer 401 to a GraphQL query with no matching Origin/Referer.
On v3 this lives in the shared AbfallIoGraphQLRetriever, which resolves
the provider's website from the source's REGIONS (the same lookup pattern
service/AbfallnaviDe.py uses), so every abfall.io GraphQL source gets it
and the source module stays declarative.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 3a86e52cc0c4ff783d18cc9f58d72708c37b3d89)

* port(abfallnavi_de): carry master fixes #7011 and #7014 onto the v3 pipeline

Ports the behaviour of upstream PR #7011 ("do not abort fetch when a
street name resolves to multiple split segments") onto the v3 pipeline
path. Master fixed AbfallnaviDe.get_dates(), which no longer exists on
v3: the multi-street-id loop now lives in the shared AbfallnaviRetriever
component, so the fix goes there and every Abfallnavi/regio iT provider
gets it.

Some providers split one street name into several entries with disjoint
house-number ranges keyed by city district (Solingen "Katternberger
Strasse", issue #4701). The retriever previously aborted the whole fetch
on the first segment that did not contain the requested house number. It
now skips that segment and keeps trying the rest, and only raises once
no segment matched, with the combined suggestions from all of them. A
single-segment street still propagates the original error unchanged. The
raised argument name comes from the retriever's house_number field name
rather than a hardcoded string, so it keeps matching the source's
__init__ kwarg.

Adds the Solingen regression test case from #7011 to TEST_CASES.

Upstream PR #7014 (Gronau, regio iT Abfallkalender) needed no port: it
added a standalone gronau_de.py against the kalender-wml backend, which
is a different product from the AbfallNavi REST app, and the merge
carried it across byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 6e79451c65062fad383d50631b6e4ab85ecf81ab)

* port: jumomind_de provider-list changes from master (#7016, #6996)

Port the MyMuell provider-list data from master into the v3 pipelin…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Source Defect]: sudestavenir_fr: missing "Déchets végétaux" (green waste) collection type

2 participants