Repository navigation
Growth P6: Caddy access logs, snippet-dir migration, weekly aggregates export - #26
Merged
Merged
Conversation
Move the Caddyfile under deploy/caddy, import gitignored site snippets, pin Caddy 2.11.1, and write a masked JSON access log to the new caddy-logs volume with 10 MiB rotation and 30-day retention. The backend reads that volume read-only; next and static assets are skipped. CI validates the config with an empty sites directory and with thisiskp's real block.
Back up the host Caddyfile, extract the marked thisiskp block from origin/main's helper into the snippet directory, restore the tracked file, and pull. Validate the new directory config in a throwaway pinned Caddy before cutover; validate/reload after it; probe thisiskp.in in wait_ready and run_probes; pass GROWTH_EXPORT_TOKEN from the new secret without writing it to disk.
Read the masked Caddy logs for one IST week, aggregate requests, status, page templates, referrer classes, bot share and calendar-feed fetches, and join them with the analytics SQLite product metrics, feedback and ingest health. Require a constant-time bearer token, suppress people cells below five, and expose it through a GET-only Next proxy with no-store.
Add fixture log tests for active and rolled files, bot share, calendar feeds and masked-IP privacy; route tests for token off/401/200, constant-time comparison, small-cell suppression and no raw IDs; and marker extraction with and without a block. Add them to the deploy CI test list.
caddy:2.11.1 has published medium and high advisories; 2.11.7 (2026-10-03) has none and is past the two-week rule. docker-compose.yml is now the only place the tag is written: the CI test-caddy job and the deploy's pre-cutover validation both read it with docker compose config --images, and CI fails unless it is an exact x.y.z tag.
Skip the probe with a warning when the host has no thisiskp.in snippet, so a fresh host can deploy. Otherwise read the status without --fail: 2xx/3xx pass; 502/503/504 warn that thisiskp's upstream is down and pass; any other status, or a TLS/connection error after 12 retries 5 s apart, fails with the existing error. Still part of wait_ready, run_probes and the rollback re-probe.
Move the backup/extract/pull step into scripts/caddy_migrate.sh, taken from origin/main by the deploy like the extraction helper. An EXIT trap armed just before git checkout -- Caddyfile copies the backup back on any exit until the pull succeeds; the extracted snippet stays. The helper now reads with newline='' and writes bytes, so the snippet keeps the block's exact bytes. tests/deploy/test_caddy_migration.sh builds temp repos and checks a rewritten remote and an unreachable remote (Caddyfile byte-identical to the backup), a successful pull and an already-migrated host; CI runs it after the backend tests.
Full Referer URLs can carry search terms or private paths. The filter keeps scheme://host for http(s) referrers and logs anything else as an empty string. Caddy's regexp filter is a replace-all, so the pattern matches every value; ^(https?://[^/?#]+).*$ alone would leave non-matching values such as android-app://...?q= untouched. serverlogs.py already reduces the referrer to a class; a test shows origin-only and empty values classify as before.
…port Rename subscribers_approx to feed_networks_approx, note in the JSON that it counts distinct masked networks, not people, and apply the same n<5 suppression as the people cells (the helper now lives in serverlogs.py and weekly.py reuses it). Keep each computed week in memory for 10 minutes so repeated calls do not rescan the logs; the token is still checked first and invalid weeks are never cached. week must now be exactly YYYY-Www, so variants such as 2026-W5 cannot create their own cache entry and return empty product cells.
# Conflicts: # .github/workflows/deploy-lightsail.yml
kpdagrt22
marked this pull request as ready for review
November 7, 2026 05:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft while the architect reviews. M-02 (masked Caddy access log, pinned Caddy, sites/ snippet dir with zero-downtime migration of the thisiskp block) and the aggregates-only weekly export for M-04. Pairs with kpdagrt22/thisiskp feat/caddy-snippet (merge that first).
🤖 Generated with Claude Code