Skip to content

Growth P6: Caddy access logs, snippet-dir migration, weekly aggregates export - #26

Merged
kpdagrt22 merged 10 commits into
mainfrom
feat/growth-p6
Nov 7, 2026
Merged

kpdagrt22 merged 10 commits into
mainfrom
feat/growth-p6

Conversation

@kpdagrt22

Copy link
Copy Markdown
Owner

Draft while the architect reviews. M-02 (masked Caddy access log, pinned Caddy, sites/ snippet dir with zero-downtime migration of the thisiskp block) and the aggregates-only weekly export for M-04. Pairs with kpdagrt22/thisiskp feat/caddy-snippet (merge that first).

🤖 Generated with Claude Code

Move the Caddyfile under deploy/caddy, import gitignored site snippets, pin Caddy 2.11.1, and write a masked JSON access log to the new caddy-logs volume with 10 MiB rotation and 30-day retention. The backend reads that volume read-only; next and static assets are skipped. CI validates the config with an empty sites directory and with thisiskp's real block.
Back up the host Caddyfile, extract the marked thisiskp block from origin/main's helper into the snippet directory, restore the tracked file, and pull. Validate the new directory config in a throwaway pinned Caddy before cutover; validate/reload after it; probe thisiskp.in in wait_ready and run_probes; pass GROWTH_EXPORT_TOKEN from the new secret without writing it to disk.
Read the masked Caddy logs for one IST week, aggregate requests, status, page templates, referrer classes, bot share and calendar-feed fetches, and join them with the analytics SQLite product metrics, feedback and ingest health. Require a constant-time bearer token, suppress people cells below five, and expose it through a GET-only Next proxy with no-store.
Add fixture log tests for active and rolled files, bot share, calendar feeds and masked-IP privacy; route tests for token off/401/200, constant-time comparison, small-cell suppression and no raw IDs; and marker extraction with and without a block. Add them to the deploy CI test list.
caddy:2.11.1 has published medium and high advisories; 2.11.7 (2026-10-03) has none and is past the two-week rule. docker-compose.yml is now the only place the tag is written: the CI test-caddy job and the deploy's pre-cutover validation both read it with docker compose config --images, and CI fails unless it is an exact x.y.z tag.
Skip the probe with a warning when the host has no thisiskp.in snippet, so a fresh host can deploy. Otherwise read the status without --fail: 2xx/3xx pass; 502/503/504 warn that thisiskp's upstream is down and pass; any other status, or a TLS/connection error after 12 retries 5 s apart, fails with the existing error. Still part of wait_ready, run_probes and the rollback re-probe.
Move the backup/extract/pull step into scripts/caddy_migrate.sh, taken from origin/main by the deploy like the extraction helper. An EXIT trap armed just before git checkout -- Caddyfile copies the backup back on any exit until the pull succeeds; the extracted snippet stays. The helper now reads with newline='' and writes bytes, so the snippet keeps the block's exact bytes. tests/deploy/test_caddy_migration.sh builds temp repos and checks a rewritten remote and an unreachable remote (Caddyfile byte-identical to the backup), a successful pull and an already-migrated host; CI runs it after the backend tests.
Full Referer URLs can carry search terms or private paths. The filter keeps scheme://host for http(s) referrers and logs anything else as an empty string. Caddy's regexp filter is a replace-all, so the pattern matches every value; ^(https?://[^/?#]+).*$ alone would leave non-matching values such as android-app://...?q= untouched. serverlogs.py already reduces the referrer to a class; a test shows origin-only and empty values classify as before.
…port

Rename subscribers_approx to feed_networks_approx, note in the JSON that it counts distinct masked networks, not people, and apply the same n<5 suppression as the people cells (the helper now lives in serverlogs.py and weekly.py reuses it). Keep each computed week in memory for 10 minutes so repeated calls do not rescan the logs; the token is still checked first and invalid weeks are never cached. week must now be exactly YYYY-Www, so variants such as 2026-W5 cannot create their own cache entry and return empty product cells.
# Conflicts:
#	.github/workflows/deploy-lightsail.yml
@kpdagrt22
kpdagrt22 marked this pull request as ready for review November 7, 2026 05:45
@kpdagrt22
kpdagrt22 merged commit f059843 into main Nov 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant