Skip to content

Add deterministic JSONL export for incident ingestion - #29

Merged
basitalisandhu merged 4 commits into
basitalisandhu:mainfrom
nawaaaaaAaar:fix/jsonl-export-track08
Oct 28, 2026
Merged

basitalisandhu merged 4 commits into
basitalisandhu:mainfrom
nawaaaaaAaar:fix/jsonl-export-track08

Conversation

@nawaaaaaAaar

@nawaaaaaAaar nawaaaaaAaar commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #4.

The site build now writes site/incidents.jsonl alongside the existing JSON array and CSV, one UTF-8 JSON object per line in the same id order. The index navigation, dataset download metadata, README and generated llms downloads expose it. The record schema and existing JSON/CSV content are unchanged.

Reproduction and tests

Before the change, both new export tests fail because incidents.jsonl does not exist. They now check record-by-record equality with the array export, id order, trailing newline, byte-identical rebuilds, Unicode and embedded newlines, and the download links.

Checked on Python 3.12:

  • python -m pytest -q: 479 passed.
  • python scripts/validate.py: all 88 records valid.
  • python scripts/build_site.py: regenerated outputs committed.
  • python scripts/export_csv.py --out /tmp/track08-incidents.csv and diff -u data/incidents.csv /tmp/track08-incidents.csv: identical.
  • git diff --check: passed.

No incident data or mappings were changed.

Also applied the optional README follow-up from #12: the CSV example now writes
to 2025.csv, outside the published data/ directory.

AI assistance

Implemented and tested with Perplexity Computer acting for this account. No independent human review is claimed.

Copilot AI balanced review requested due to automatic review settings October 22, 2026 14:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@basitalisandhu
basitalisandhu merged commit fd4ce10 into basitalisandhu:main Oct 28, 2026
1 check passed
@basitalisandhu

Copy link
Copy Markdown
Owner

Merged, thank you. Testing that a rebuild is byte-identical, and that embedded newlines and non-ASCII text survive, covers what actually breaks line-delimited exports. I added site/incidents.jsonl to LICENSE-DATA on your branch, which was the last point from #4.

basitalisandhu added a commit that referenced this pull request Oct 28, 2026
…tests; release 1.2.0 (#31)

Second deep pass. No record is added or changed; every count below comes
from the existing 88 records.

**User-visible**
- `scripts/digest.py --month YYYY-MM` writes `digest/YYYY-MM.md`:
records added that month (by first commit, from git history), counts by
type, lens, vector and outcome, vendors named more than once, and a
linked list of the records. `digest/2026-10.md` is the first digest (all
88 records were first committed in October 2026). The site front page
links every digest. Closes #21.
- README Troubleshooting section built from the real messages.

**Fuzz findings, fixed** (seeded, one-field-at-a-time damage to real
records, every script run on it)
- `export_csv.py`, `build_site.py`, `agentic_crosstab.py`,
`check_sources.py`: a missing field, a wrong type, an empty `sources`
list, a missing mapping list, a value outside the codebook or a
malformed date crashed with `KeyError` / `IndexError` / `ValueError`.
New `scripts/records.py` checks each record against the types, required
fields, enums and patterns in the schema (standard library only) for the
fields each script uses, and stops with one line naming the file and the
first problem (exit 2).
- `validate.py` crashed on non-UTF-8, deeply nested or non-object files
and on a non-string `id`; these are now listed errors.
- `import_csv.py` crashed on a short row, a bad `adversarial` value, a
missing or non-UTF-8 CSV, or an enrichment file that is not an object,
and had already written the rows before the bad one. It now exits 2 with
file and line and writes nothing until every row is valid.

**Performance** (88 real records copied to 880 and 4,400)
- 10x: validate 0.07s -> 0.11s, export 0.01s -> 0.03s, crosstab 0.00s ->
0.03s, site 0.02s -> 0.16s, import 0.01s -> 0.07s; 50x: site 0.95s.
Linear; nothing to fix. `tests/test_scaling.py` holds all five under a
30x ratio for 10x input.

**Tests and CI**
- `tests/test_fuzz_scripts.py`, `tests/test_scaling.py`,
`tests/test_digest.py`.
- `tests/test_quickstart_e2e.py` + new `quickstart` CI job: copies the
tracked files, makes a fresh venv, runs the README "Working with the
repository" block (everything that needs no network or token) and checks
that site/, docs/stats.md and data/incidents.csv come out identical to
the committed ones.

Release 1.2.0 (CITATION.cff, README, hf/README and the site carry the
version; CHANGELOG dated 2026-10-28, also credits #29). Good first
issues #3, #7 and #20 stay open.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a JSON Lines export

3 participants