Repository navigation
Add deterministic JSONL export for incident ingestion - #29
Merged
basitalisandhu merged 4 commits intoOct 28, 2026
Merged
Conversation
basitalisandhu
approved these changes
Oct 28, 2026
Owner
|
Merged, thank you. Testing that a rebuild is byte-identical, and that embedded newlines and non-ASCII text survive, covers what actually breaks line-delimited exports. I added |
basitalisandhu
added a commit
that referenced
this pull request
Oct 28, 2026
…tests; release 1.2.0 (#31) Second deep pass. No record is added or changed; every count below comes from the existing 88 records. **User-visible** - `scripts/digest.py --month YYYY-MM` writes `digest/YYYY-MM.md`: records added that month (by first commit, from git history), counts by type, lens, vector and outcome, vendors named more than once, and a linked list of the records. `digest/2026-10.md` is the first digest (all 88 records were first committed in October 2026). The site front page links every digest. Closes #21. - README Troubleshooting section built from the real messages. **Fuzz findings, fixed** (seeded, one-field-at-a-time damage to real records, every script run on it) - `export_csv.py`, `build_site.py`, `agentic_crosstab.py`, `check_sources.py`: a missing field, a wrong type, an empty `sources` list, a missing mapping list, a value outside the codebook or a malformed date crashed with `KeyError` / `IndexError` / `ValueError`. New `scripts/records.py` checks each record against the types, required fields, enums and patterns in the schema (standard library only) for the fields each script uses, and stops with one line naming the file and the first problem (exit 2). - `validate.py` crashed on non-UTF-8, deeply nested or non-object files and on a non-string `id`; these are now listed errors. - `import_csv.py` crashed on a short row, a bad `adversarial` value, a missing or non-UTF-8 CSV, or an enrichment file that is not an object, and had already written the rows before the bad one. It now exits 2 with file and line and writes nothing until every row is valid. **Performance** (88 real records copied to 880 and 4,400) - 10x: validate 0.07s -> 0.11s, export 0.01s -> 0.03s, crosstab 0.00s -> 0.03s, site 0.02s -> 0.16s, import 0.01s -> 0.07s; 50x: site 0.95s. Linear; nothing to fix. `tests/test_scaling.py` holds all five under a 30x ratio for 10x input. **Tests and CI** - `tests/test_fuzz_scripts.py`, `tests/test_scaling.py`, `tests/test_digest.py`. - `tests/test_quickstart_e2e.py` + new `quickstart` CI job: copies the tracked files, makes a fresh venv, runs the README "Working with the repository" block (everything that needs no network or token) and checks that site/, docs/stats.md and data/incidents.csv come out identical to the committed ones. Release 1.2.0 (CITATION.cff, README, hf/README and the site carry the version; CHANGELOG dated 2026-10-28, also credits #29). Good first issues #3, #7 and #20 stay open. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #4.
The site build now writes
site/incidents.jsonlalongside the existing JSON array and CSV, one UTF-8 JSON object per line in the same id order. The index navigation, dataset download metadata, README and generated llms downloads expose it. The record schema and existing JSON/CSV content are unchanged.Reproduction and tests
Before the change, both new export tests fail because
incidents.jsonldoes not exist. They now check record-by-record equality with the array export, id order, trailing newline, byte-identical rebuilds, Unicode and embedded newlines, and the download links.Checked on Python 3.12:
python -m pytest -q: 479 passed.python scripts/validate.py: all 88 records valid.python scripts/build_site.py: regenerated outputs committed.python scripts/export_csv.py --out /tmp/track08-incidents.csvanddiff -u data/incidents.csv /tmp/track08-incidents.csv: identical.git diff --check: passed.No incident data or mappings were changed.
Also applied the optional README follow-up from #12: the CSV example now writes
to
2025.csv, outside the publisheddata/directory.AI assistance
Implemented and tested with Perplexity Computer acting for this account. No independent human review is claimed.