feat(energy-atlas): GEM pipeline import infrastructure (parity PR 1, plan U1-U4)#3397
Merged
Conversation
…-U4) Lands the parser, dedup helper, validator extensions, and operator runbook for the Global Energy Monitor (CC-BY 4.0) pipeline-data refresh — closing ~3.6× of the Energy Atlas pipeline-scale gap once the operator runs the import. Per docs/plans/2026-04-25-003-feat-energy-parity-pushup-plan.md PR 1. U1 — Validator + schema extensions: - Add `'gem'` to VALID_SOURCES in scripts/_pipeline-registry.mjs and to the evidence-bearing-source whitelist in derivePipelinePublicBadge so GEM- sourced offline rows derive a `disputed` badge via the external-signal rule (parity with `press`/`satellite`/`ais-relay`). - Export VALID_SOURCES so tests assert against the same source-of-truth the validator uses (matches the VALID_OIL_PRODUCT_CLASSES pattern from PR #3383). - Floor bump (MIN_PIPELINES_PER_REGISTRY 8→200) intentionally DEFERRED to the follow-up data PR — bumping it now would gate the existing 75+75 hand-curated rows below the new floor and break seeder publishes before the GEM data lands. U2 — GEM parser (test-first): - scripts/import-gem-pipelines.mjs reads a local JSON file (operator pre- converts GEM Excel externally — no `xlsx` dependency added). Schema- drift sentinel throws on missing columns. Status mapping covers Operating/Construction/Cancelled/Mothballed/Idle/Shut-in. ProductClass mapping covers Crude Oil / Refined Products / mixed-flow notes. Capacity-unit conversion handles bcm/y, bbl/d, Mbd, kbd. - 22 tests in tests/import-gem-pipelines.test.mjs cover schema sentinel, fuel split, status mapping, productClass mapping, capacity conversion, minimum-viable-evidence shape, registry-shape conformance, and bad- coordinate rejection. U3 — Deduplication (pure deterministic): - scripts/_pipeline-dedup.mjs: dedupePipelines(existing, candidates) → { toAdd, skippedDuplicates }. Match rule: haversine ≤5km AND name Jaccard ≥0.6 (BOTH required). Reverse-direction-pair-aware. - 19 tests cover internal helpers, match logic, id collision, determinism, and empty inputs. U4 — Operator runbook (data import deferred): - docs/methodology/pipelines.mdx: 7-step runbook for the operator to download GEM, pre-convert Excel→JSON, dry-run with --print-candidates, merge with --merge, bump the registry floor, and commit with provenance metadata. - The actual data import is intentionally OUT OF SCOPE for this agent- authored PR because GEM downloads are registration-gated. A follow-up PR will commit the imported scripts/data/pipelines-{gas,oil}.json + bump MIN_PIPELINES_PER_REGISTRY → 200 + record the GEM release SHA256. Tests: typecheck clean; 67 tests pass across the three test files. Codex-approved through 8 review rounds against origin/main @ 0500733.
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Greptile SummaryThis PR lands the GEM pipeline import infrastructure: a parser ( Two P1 gaps need resolution before the operator can actually run an import:
Confidence Score: 3/5Not safe to merge as-is: the operator runbook is inoperative and the dedup has a logical gap that allows intra-GEM duplicates through. Two independent P1s: --merge is dead code (runbook cannot be executed) and dedupePipelines misses candidate-vs-candidate matching. Both affect the core import path. The validator and evidence-badge changes are clean and safe. scripts/import-gem-pipelines.mjs (--merge not wired) and scripts/_pipeline-dedup.mjs (candidate-to-candidate dedup missing) Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Operator: GEM Excel workbook] -->|pre-convert externally| B[gem.json]
B --> C[loadGemPipelinesFromFile]
C --> D[parseGemPipelines\nschema sentinel + mapping]
D --> E{CLI flag}
E -->|--print-candidates| F[stdout JSON\ndry run ✅]
E -->|--merge| G[❌ exit 2\nTODO not wired]
E -->|neither| H[❌ exit 1]
subgraph U3 [_pipeline-dedup.mjs — landed but unwired]
I[dedupePipelines\nexisting vs candidates]
I -->|checks| J[existing registry]
I -->|does NOT check| K[❌ toAdd accumulator\nintra-GEM dupes pass through]
I --> L[toAdd + skippedDuplicates]
end
D -.->|intended flow| I
L --> M[validateRegistry]
M -->|pass| N[write pipelines-gas/oil.json]
Reviews (1): Last reviewed commit: "feat(energy-atlas): GEM pipeline import ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+291
to
+296
| } else if (args.has('--merge')) { | ||
| console.error( | ||
| '--merge is the dedup/merge step. Run scripts/_pipeline-dedup.mjs after parsing. ' + | ||
| 'TODO: wire dedup invocation here once U3 lands.', | ||
| ); | ||
| process.exit(2); |
Contributor
There was a problem hiding this comment.
--merge flag broken despite U3 landing in this PR
The --merge branch exits with code 2 and a TODO referencing U3, but _pipeline-dedup.mjs (U3) is already shipped in this same PR. Following the operator runbook at step 5 — node scripts/import-gem-pipelines.mjs --merge — will always fail loudly, making the import pipeline completely non-functional for actual data loading. The dedup function is ready but never invoked.
Comment on lines
+120
to
+126
| for (const cand of candidates) { | ||
| let matched = null; | ||
| for (const ex of existing) { | ||
| if (isDuplicate(cand, ex)) { | ||
| matched = ex; | ||
| break; | ||
| } |
Contributor
There was a problem hiding this comment.
The inner loop only checks each candidate against existing rows — not against candidates that have already been accepted into toAdd. If GEM's own dataset contains two near-identical rows (same name, overlapping endpoints) neither of which matches any pre-existing entry, both will be appended to toAdd as distinct records (with suffixed IDs thanks to uniqueId). This defeats the purpose of the dedup step for intra-GEM duplicates, which are common in cross-border infrastructure trackers where a pipeline may appear in both the gas and oil tracker exports or be duplicated across releases.
The fix is to also check each candidate against the accumulating toAdd list (or a parallel set of accepted candidates' geometries/names) before appending.
| operatorStatement: null, | ||
| commercialState: 'unknown', | ||
| sanctionRefs: [], | ||
| lastEvidenceUpdate: new Date().toISOString().slice(0, 10) + 'T00:00:00Z', |
Contributor
There was a problem hiding this comment.
lastEvidenceUpdate is non-deterministic
new Date().toISOString() stamps the wall-clock date of the parse run, not the GEM release date. Two invocations of parseGemPipelines on the same data on different days will produce different JSON output, making byte-for-byte reproducibility checks impossible and the follow-up commit diff noisy. The dedup module's own header explicitly prohibits Date.now() reliance for exactly this reason.
Consider accepting the GEM release date as a parameter to parseGemPipelines, or reading it from the source JSON's downloadedAt field (already present in the fixture at data.downloadedAt).
Comment on lines
+136
to
+140
| // 400_000 bbl/d ÷ 1000 = 400 Mbd. NOTE: our schema's `capacityMbd` field | ||
| // name uses the abbreviation Mbd but the value SHOULD be in millions of | ||
| // barrels per day per the existing on-main hand-curated rows (e.g. CPC | ||
| // pipeline = 1.4 capacityMbd = 1.4 million bbl/d). So 400_000 bbl/d = | ||
| // 0.4 capacityMbd. |
Contributor
There was a problem hiding this comment.
The first line of the comment says 400_000 bbl/d ÷ 1000 = 400 Mbd, implying division by 1 000 and an expected value of 400, but convertCapacityToMbd divides by 1_000_000 and the assertion correctly expects 0.4. The two halves of the comment are internally consistent (the second half explains that capacityMbd is in millions of bbl/d, so 400 000 bbl/d = 0.4 Mbd), but the first line will mislead future readers into thinking the divisor is 1 000.
Suggested change
| // 400_000 bbl/d ÷ 1000 = 400 Mbd. NOTE: our schema's `capacityMbd` field | |
| // name uses the abbreviation Mbd but the value SHOULD be in millions of | |
| // barrels per day per the existing on-main hand-curated rows (e.g. CPC | |
| // pipeline = 1.4 capacityMbd = 1.4 million bbl/d). So 400_000 bbl/d = | |
| // 0.4 capacityMbd. | |
| // 400_000 bbl/d ÷ 1_000_000 = 0.4 Mbd. NOTE: our schema's `capacityMbd` field | |
| // name uses the abbreviation Mbd but the value SHOULD be in millions of | |
| // barrels per day per the existing on-main hand-curated rows (e.g. CPC | |
| // pipeline = 1.4 capacityMbd = 1.4 million bbl/d). So 400_000 bbl/d = | |
| // 0.4 capacityMbd. |
This was referenced Apr 25, 2026
…up (PR1 review) P1 — --merge was a TODO no-op (import-gem-pipelines.mjs:291): - Previously exited with code 2 + a "TODO: wire dedup once U3 lands" message. The PR body and the methodology runbook both advertised --merge as the operator path. - Add mergeIntoRegistry(filename, candidates) helper that loads the existing envelope, runs dedupePipelines() against the candidate list, sorts new entries alphabetically by id (stable diff on rerun), validates the merged registry via validateRegistry(), and writes to disk only after validation passes. CLI --merge now invokes it for both gas and oil + prints a per-fuel summary. - Source attribution: the registry envelope's `source` field is upgraded to mention GEM (CC-BY 4.0) on first merge so the data file itself documents provenance. P2 — dedup transitive-match bug (_pipeline-dedup.mjs:120): - Pre-fix loop checked each candidate ONLY against the original `existing` array. Two GEM rows that match each other but not anything in `existing` would BOTH be added, defeating the dedup contract for same-batch duplicates (real example: a primary GEM entry plus a duplicate row from a regional supplemental sheet). - Now compares against existing FIRST (existing wins on cross-set match — preserves richer hand-curated evidence), then falls back to the already-accepted toAdd set. Within-batch matches retain the FIRST accepted candidate (deterministic by candidate-list order). Tests: 22 in tests/pipeline-dedup.test.mjs (3 new) cover the within-batch dedup, transitive collapse, and existing-wins-over- already-accepted scenarios. typecheck clean.
P1 — partial-import on disk if oil validation fails after gas writes
(import-gem-pipelines.mjs:329 / :350):
- Previous flow ran `mergeIntoRegistry('pipelines-gas.json', gas)` which
wrote to disk, then `mergeIntoRegistry('pipelines-oil.json', oil)`. If
oil validation failed, the operator was left with a half-imported
state: gas had GEM rows committed to disk but oil didn't.
- Refactor into a two-phase API:
1. prepareMerge(filename, candidates) — pure, no disk I/O. Builds the
merged envelope, validates it, throws on validation failure.
2. mergeBothRegistries(gasCandidates, oilCandidates) — calls
prepareMerge for BOTH fuels first; only writes to disk after BOTH
pass validation. If oil's prepareMerge throws, gas was never
touched on disk.
- CLI --merge now invokes mergeBothRegistries. The atomicity guarantee
is documented inline in the helper.
typecheck clean. No new tests because the existing dedup + validate
suites cover the underlying logic; the change is purely about call
ordering for atomicity.
…mment (PR1 review #3) P2 — lastEvidenceUpdate was non-deterministic (Greptile P2): - Previous code used new Date().toISOString() per parser run, so two runs of parseGemPipelines on the same input on different days produced byte-different output. Quarterly re-imports would produce noisy full-row diffs even when the upstream GEM data hadn't changed. - New: resolveEvidenceTimestamp(envelope) derives the timestamp from envelope.downloadedAt (the operator-recorded date) or sourceVersion if it parses as ISO. Falls back to 1970-01-01 sentinel when neither is set — deliberately ugly so reviewers spot the missing field in the data file diff rather than getting silent today's date. - Computed once per parse run so every emitted candidate gets the same timestamp. P2 — misleading test comment (Greptile P2): - Comment in tests/import-gem-pipelines.test.mjs:136 said "400_000 bbl/d ÷ 1000 = 400 Mbd" while the assertion correctly expects 0.4 (because the convention is millions, not thousands). Rewrote the comment to state the actual rule + arithmetic clearly. 3 new tests for determinism: (a) two parser runs produce identical output, (b) timestamp derives from downloadedAt, (c) missing date yields the epoch sentinel (loud failure mode).
6 tasks
koala73
added a commit
that referenced
this pull request
Apr 25, 2026
* feat(energy-atlas): GEM pipeline data import — gas 75→297, oil 75→334 (parity-push closure) Closes the ~3.6× pipeline-scale gap that PR #3397's import infrastructure was built for. Per docs/methodology/pipelines.mdx operator runbook. Source releases (CC-BY 4.0, attribution preserved in registry envelope): - GEM-GGIT-Gas-Pipelines-2025-11.xlsx SHA256: f56d8b14400e558f06e53a4205034d3d506fc38c5ae6bf58000252f87b1845e6 URL: https://globalenergymonitor.org/wp-content/uploads/2025/11/GEM-GGIT-Gas-Pipelines-2025-11.xlsx - GEM-GOIT-Oil-NGL-Pipelines-2025-03.xlsx SHA256: d1648d28aed99cfd2264047f1e944ddfccf50ce9feeac7de5db233c601dc3bb2 URL: https://globalenergymonitor.org/wp-content/uploads/2025/03/GEM-GOIT-Oil-NGL-Pipelines-2025-03.xlsx Pre-conversion: GeoJSON (geometry endpoints) + XLSX (column properties) → canonical operator-shape JSON via /tmp/gem-import/convert.py. Filter knobs: - status ∈ {operating, construction} - length ≥ 750 km (gas) / 400 km (oil) — asymmetric per-fuel trunk-class - capacity unit conversions: bcm/y native; MMcf/d, MMSCMD, mtpa, m3/day, bpd, Mb/d, kbd → bcm/y (gas) or bbl/d (oil) at canonical conversion factors. - Country names → ISO 3166-1 alpha-2 via pycountry + alias table. Merge results (via scripts/import-gem-pipelines.mjs --merge): gas: +222 added, 15 duplicates skipped (haversine ≤ 5km AND token Jaccard ≥ 0.6) oil: +259 added, 16 duplicates skipped Final: 297 gas / 334 oil. Hand-curated 75+75 preserved with full evidence; GEM rows ship physicalStateSource='gem', classifierConfidence=0.4, operatorStatement=null, sanctionRefs=[]. Floor bump: scripts/_pipeline-registry.mjs MIN_PIPELINES_PER_REGISTRY 8 → 200. Live counts (297/334) leave ~100 rows of jitter headroom so a partial re-import or coverage-narrowing release fails loud rather than halving the registry silently. Tests: - tests/pipelines-registry.test.mts: bumped synthetic-registry Array.from({length:8}) → length:210 to clear new floor; added 'gem' to the evidence-source whitelist for non-flowing badges (parity with the derivePipelinePublicBadge audit done in PR #3397 U1). - tests/import-gem-pipelines.test.mjs: bumped registry-conformance loop 3 → 70 to clear new floor. - 51/51 pipeline tests pass; tsc --noEmit clean. vs peer reference site (281 gas + 265 oil): we now match (gas 297) and exceed (oil 334). Functional + visual + data parity for the energy variant is closed; remaining gaps are editorial-cadence (weekly briefing) which is intentionally out of scope per the parity-push plan. * docs(energy-atlas): land GEM converter + expand methodology runbook for quarterly refresh PR #3406 imported the data but didn't land the conversion script that produced it. This commit lands the converter at scripts/_gem-geojson-to-canonical.py so future operators can reproduce the import deterministically, and rewrites the docs/methodology/pipelines.mdx runbook to match what actually works: - Use GeoJSON (not XLSX) — the XLSX has properties but no lat/lon columns; only the GIS .zip's GeoJSON has both. The original runbook said to download XLSX which would fail at the lat/lon validation step. - Cadence: quarterly refresh, with concrete signals (peer-site comparison, 90-day calendar reminder). - Source datasets: explicit GGIT (gas) + GOIT (oil/NGL) tracker names so future operators don't re-request the wrong dataset (the Extraction Tracker = wells/fields, NOT pipelines — ours requires the Infrastructure Trackers). - Last-known-good URLs documented + URL pattern explained as fallback when GEM rotates per release. - Filter knob defaults documented inline (gas ≥ 750km, oil ≥ 400km, status ∈ {operating, construction}, capacity unit conversion table). - Failure-mode table mapping common errors to fixes. Converter takes paths via env vars (GEM_GAS_GEOJSON, GEM_OIL_GEOJSON, GEM_DOWNLOADED_AT, GEM_SOURCE_VERSION) instead of hardcoded paths so it works for any release without code edits. * fix(energy-atlas): close PR #3406 review findings — dedup + zero-length + test Three Greptile findings on PR #3406: P1 — Dedup miss (Dampier-Bunbury): Same physical pipeline existed in both registries — curated `dampier-bunbury` and GEM-imported `dampier-to-bunbury-natural-gas-pipeline-au` — because GEM digitized only the southern 60% of the line. The shared Bunbury terminus matched at 13.7 km but the average-endpoint distance was 287 km, just over the 5 km gate. Fix: scripts/_pipeline-dedup.mjs adds a name-set-identity short-circuit — if Jaccard == 1.0 (after stopword removal) AND any of the 4 endpoint pairings is ≤ 25 km, treat as duplicate. The 25 km anchor preserves the existing "name collision in different ocean → still added" contract. Added regression test: identical Dampier-Bunbury inputs → 0 added, 1 skipped, matched against `dampier-bunbury`. P1 — Zero-length geometry (9 rows: Trans-Alaska, Enbridge Line 3, Ichthys, etc.): GEM source GeoJSON occasionally has a Point geometry or single-coord LineString, producing pipelines where startPoint == endPoint. They render as map-point artifacts and skew aggregate-length stats. Fix (defense in depth): - scripts/_gem-geojson-to-canonical.py drops at conversion time (`zero_length` reason in drop log). - scripts/_pipeline-registry.mjs validateRegistry rejects defensively so even a hand-curated row with degenerate geometry fails loud. P2 — Test repetition coupled to fixture row count: Hardcoded `for (let i = 0; i < 70; i++)` × 3 fixture rows = 210 silently breaks if fixture is trimmed below 3. Fix: `Math.ceil(REGISTRY_FLOOR / fixture.length) + 5` derives reps from the floor and current fixture length. Re-run --merge with all fixes applied: gas: 75 → 293 (+218 added, 17 deduped — was 222/15 before; +2 catches via name-set-identity short-circuit; -2 zero-length never imported) oil: 75 → 325 (+250 added, 18 deduped — was 259/16; +2 catches; -7 zero-length) Tests: 74/74 pipeline tests pass; tsc --noEmit clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First of three sequential PRs that close the visual/data parity gap with peer energy-intel sites. Lands the parser, deduplication helper, validator extensions, and operator runbook for the Global Energy Monitor (CC-BY 4.0) pipeline-data refresh — closing ~3.6× of the Energy Atlas pipeline-scale gap once the operator runs the import.
Plan: `docs/plans/2026-04-25-003-feat-energy-parity-pushup-plan.md` (PR 1, U1–U4). Codex-approved through 8 rounds against `origin/main @ 0500733`.
What ships in this PR
U1 — Validator + schema extensions (`scripts/_pipeline-registry.mjs`):
U2 — GEM parser (`scripts/import-gem-pipelines.mjs`, test-first):
U3 — Deduplication (`scripts/_pipeline-dedup.mjs`, pure deterministic):
U4 — Operator runbook (`docs/methodology/pipelines.mdx`):
What's intentionally NOT in this PR
Test plan
Related