Skip to content

About

Monitor public HTTP endpoints, track incidents and recoveries, and send signed alerts through Hookrelay.

Topics

Resources

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Endpoint Monitor

Endpoint Monitor detects failures on an explicit set of public HTTP and HTTPS URLs and emits incident transitions for downstream alerting. It does not discover targets, assume hostname routing, or remediate failures.

The monitoring core is runtime-neutral JavaScript. The included Cloudflare Workers adapter adds deterministic Cron scheduling, exceptional-state-only D1 persistence, Workers Observability logs, optional Cloudflare edge analytics, and signed CloudEvents delivery through Hookrelay.

Synthetic Endpoint Monitor operator run and incident lifecycle

Why explicit targets

Traffic, DNS inventory, and provider metadata do not define what should be monitored. They can include retired hosts, external fetch destinations, alternate routes, and hostnames whose useful path is not /. Endpoint Monitor therefore treats its explicit configuration as the only enrollment authority. Provider data may corroborate a configured target but cannot add one. On Cloudflare, D1 owns that configuration; local documents are import candidates.

Independent watchdog

Install the optional Unix cron watchdog on hosting outside Cloudflare to detect missing scheduled runs and status failures. It sends Gmail alerts and recovery notifications, retries failed mail, groups flapping failures until stable recovery, and emits a daily healthy heartbeat. Its read-only report retains bounded incident evidence and daily diagnostic aggregates. The observer uses Python's standard library and a dedicated read-only status token.

Installation

Endpoint Monitor ships as one indivisible package: @j-256/endpoint-monitor contains the controller CLI, runtime-neutral monitoring core, Cloudflare Worker, D1 migrations, and deployment templates. Install it into a small operator project rather than globally so the controller always deploys the Worker and migrations from the exact installed version.

GitHub Releases contain an installable j-256-endpoint-monitor-X.Y.Z.tgz archive and SHA256SUMS. Download both assets, verify the checksum, then create an operator project:

shasum -a 256 -c SHA256SUMS
mkdir endpoint-monitor-service
cd endpoint-monitor-service
npm init -y
npm install --save-exact /path/to/j-256-endpoint-monitor-X.Y.Z.tgz
npm exec -- endpoint-monitor init

When npm registry publication is enabled, only the install line changes:

npm install --save-exact @j-256/endpoint-monitor

The remaining init, bootstrap, and deploy commands are identical. The controller and service will not be published separately. See Releases for artifact contents, GNU checksum verification, versioning, and the maintainer procedure.

Quick start

Use Node.js 22 or newer. endpoint-monitor init creates three local files: an editable target document, a mode-0600 operator profile, and .gitignore entries that keep the target document, profile, generated Wrangler configuration, and Wrangler state out of version control.

Edit endpoint-monitor.json, then validate and probe it without durable writes:

npm exec -- endpoint-monitor config validate
npm exec -- endpoint-monitor targets
npm exec -- endpoint-monitor probe

Add --json to config validate, targets, or probe for machine-readable output. A failed target makes probe exit with status 1. Both npm exec -- endpoint-monitor help <command> and npm exec -- endpoint-monitor <command> --help show command help.

From a source checkout, run npm ci and substitute node src/cli.mjs for npm exec -- endpoint-monitor when developing the package itself.

Configuration

{
  "schemaVersion": 2,
  "defaults": {
    "failureThreshold": 2,
    "method": "GET",
    "probeIntervalMinutes": 5,
    "recoveryThreshold": 2,
    "timeoutMilliseconds": 10000
  },
  "targets": [
    {
      "expect": {
        "bodyIncludes": "Example Domain",
        "contentType": "text/html"
      },
      "expectedStatuses": [200],
      "id": "example-home",
      "url": "https://example.com/"
    },
    {
      "expect": {
        "contentType": "application/json",
        "jsonSubset": {
          "ok": true
        }
      },
      "expectedStatuses": [200],
      "id": "example-health",
      "url": "https://status.example.net/health"
    },
    {
      "expect": {
        "location": {
          "url": "https://example.org/docs?probe=1"
        }
      },
      "expectedStatuses": [301],
      "id": "example-redirect",
      "url": "https://old.example.org/docs?probe=1"
    }
  ]
}

id is the stable incident and scheduling identity. It must be a lower-case DNS-style label. url must use a public DNS hostname, may include an exact path and query, and may not include credentials or a fragment. Do not place credentials in query parameters because complete URLs are retained in protected status and incident records.

method is GET or HEAD. Redirects are not followed, so the configured host and route are what the probe verifies. failureThreshold, recoveryThreshold, and timeoutMilliseconds may be overridden per target.

By default, any response below HTTP 500 proves reachability. HTTP 520 through 526 and 530 open an incident immediately; other server responses and network failures use failureThreshold. Supply expectedStatuses only when the endpoint has a narrower application contract. Only successful active probes establish observed recovery; operator dismissal and configuration changes have distinct resolution reasons.

Schema version 1 remains accepted for status-only documents. Schema version 2 adds the optional expect object:

  • contentType matches the media type case-insensitively and ignores parameters such as charset
  • bodyIncludes requires one non-empty text marker of at most 1,024 bytes
  • jsonSubset recursively requires the configured object properties while allowing extra response properties; configured arrays match exactly
  • location resolves the response header against the target URL and compares normalized URL components; set ignoreQuery to true only when query differences are intentionally irrelevant

Location validation requires explicit 3xx expectedStatuses. Body validation requires GET. Status is checked before other expectations. Text markers are searched within at most the first 64 KiB and stop the read as soon as they match; JSON validation requires a complete body within that limit. Read content is discarded immediately and never included in diagnostics. A failed expectation records the observed HTTP status plus a fixed error code and follows the configured failure threshold.

Cloudflare deployment

Online operator applications can use the protected management API for paginated targets, incidents, and action history, plus reviewed configuration edits and triage. It uses dedicated expiring workspace-allowlisted credentials with separate read, configure, and triage capabilities. Its operations share the CLI's configuration authority and triage implementation; no local server is required for management. A read does not trigger probes, and an absent incident is not reported as verified health.

The Cloudflare adapter uses one Cron trigger per minute. A stable hash distributes targets across probeIntervalMinutes, with at most 10 probes and five concurrent outbound connections per invocation. Configuration that cannot satisfy that cadence is rejected rather than silently probed less often. The adapter also enforces a 45-external-subrequest budget and uses manual redirects. Review the Workers limits before deploying on a different plan or after changing these bounds.

Generated deployments additionally cap each invocation at 250 ms CPU and 100 total subrequests. These Workers Standard guardrails preserve headroom above the application limits without retaining Cloudflare's broader Paid defaults. The Cloudflare operation guide records the production observation, Free fallback, and recovery behavior.

Set the account-scoped deployment credentials that Wrangler and the controller already recognize:

export CLOUDFLARE_ACCOUNT_ID="your-32-character-account-id"
export CLOUDFLARE_API_TOKEN="your-account-api-token"

Preview the complete resource plan without local, provider, or durable writes. A safe first configuration enables active monitoring while leaving analytics, status, and delivery disabled:

npm exec -- endpoint-monitor cloudflare bootstrap --enabled --dry-run

Bootstrap creates one D1 database, writes the ignored mode-0600 wrangler.jsonc, records the resource in the operator profile, and applies the migrations bundled with the installed package. Supply --database-id <uuid> to adopt an existing D1 database instead. Rerunning bootstrap reuses the recorded database and preserves selected features.

npm exec -- endpoint-monitor cloudflare bootstrap --enabled

The operator profile remembers the absolute target-document and Wrangler-configuration paths but contains no target values, resource identifiers, or secrets. wrangler.jsonc contains the D1 identifier and feature flags but no secrets or targets.

Check the exact installed Worker bundle, then deploy. A live deploy applies pending migrations, initializes an absent D1 configuration from the validated local candidate, preserves any existing online configuration, publishes the Worker, resolves its account workers.dev hostname, and retries the public /healthz endpoint before succeeding:

npm exec -- endpoint-monitor deploy --dry-run
npm exec -- endpoint-monitor deploy

Routine target changes do not require remembering the D1 identifier or deployment flags:

npm exec -- endpoint-monitor config path
npm exec -- endpoint-monitor config validate
npm exec -- endpoint-monitor targets
npm exec -- endpoint-monitor probe
npm exec -- endpoint-monitor config review
npm exec -- endpoint-monitor config sync --expect-revision <reviewed-revision> --expect-fingerprint <reviewed-candidate-sha256>

config path identifies the local candidate. targets lists its explicit IDs, methods, status and response contracts, and URLs. Profile-backed config validate, targets, and probe use that document automatically; all three accept an explicit target-document argument for ad hoc use. D1 owns the executing configuration, and deployment does not import a local candidate over online edits. After editing, probe locally and run config review; inspect the complete candidate with config show and the executing document with config remote. Import with the exact remote revision and candidate fingerprint reported by the review. A changed revision or candidate fails without overwriting remote state; an unchanged candidate at the reviewed revision writes nothing. No Worker deployment is required for target-only changes. Use --profile <path> with profile-backed commands to select a non-default operator profile.

config remote exports private target URLs and expectations to stdout without touching the local candidate. If preserving the export, use a new file with umask 077 and inspect it before deliberately replacing a local candidate. Do not paste its contents into shared logs. Apply packaged migrations before configuration review or import; older unconditional writers fail closed after the revision-authority migration.

Apply every packaged migration before using incident triage commands. The CLI reads the D1 binding from the same operator profile and authenticates directly to Cloudflare with CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN:

npm exec -- endpoint-monitor incidents list
npm exec -- endpoint-monitor incidents show <incident-id>
npm exec -- endpoint-monitor incidents acknowledge <incident-id> -n "Under investigation"
npm exec -- endpoint-monitor incidents snooze <incident-id> -u 2026-08-28T03:00:00Z -n "Maintenance window"
npm exec -- endpoint-monitor incidents dismiss <incident-id> -n "False positive"

incidents list shows open incidents by default; add -a, --all for resolved history and -l, --limit to bound the result. Acknowledgement records review without changing health or delivery. Snooze delays an undelivered problem transition until its future RFC 3339 deadline but does not stop probes or retract an event already sent. Dismissal records an audited operator-dismissed resolution, clears exceptional state, and emits a resolved transition only when a corresponding problem transition exists. If the target is still failing, it can reopen after its configured threshold; remove or correct the target instead when the monitoring contract itself is obsolete. Operator notes are stored in D1, so keep credentials and private response content out of them.

Install only the secrets required by selected features through concealed Wrangler input:

npm exec -- wrangler secret put CLOUDFLARE_API_TOKEN --config wrangler.jsonc
npm exec -- wrangler secret put ENDPOINT_MONITOR_HOOKRELAY_URL --config wrangler.jsonc
npm exec -- wrangler secret put ENDPOINT_MONITOR_HOOKRELAY_HMAC --config wrangler.jsonc
npm exec -- wrangler secret put ENDPOINT_MONITOR_STATUS_TOKEN --config wrangler.jsonc

Rerun bootstrap to select optional features, then deploy again. Existing selections are retained when their flags are omitted:

npm exec -- endpoint-monitor cloudflare bootstrap --analytics --status
npm exec -- endpoint-monitor deploy

Add --analytics for optional Cloudflare analytics. Add --delivery only after the Hookrelay subscription is live. If Hookrelay is another Worker in the same account, add --hookrelay-service <worker-name> to use a service binding; otherwise delivery uses the public HTTPS route.

The target document is stored in D1 rather than an environment variable because Workers cap individual variables at 5 KB. See Cloudflare operation for shadow verification, Observability fields, and cutover guidance.

Storage behavior

Every enabled invocation reads one configuration row and the small exceptional-state set. A healthy target with no candidate or incident causes no per-target D1 write. Repeated failures for an already-open incident also cause no incident-engine write. Each completed scheduled minute can additionally write one bounded aggregate run-status snapshot, including successful check evidence, into fixed-capacity storage. Other D1 changes cover configuration changes, failure or recovery candidates, incident transitions, explicit operator triage actions, unique provider signals, delivery attempts, suppression after configuration changes, and periodic retention cleanup.

Use endpoint-monitor status for scheduler completion and retained target check evidence, or endpoint-monitor status --json for automation. It reads the executing configuration and bounded run-status ring without contacting targets or reading a local candidate. Passing checks, scheduler freshness, and open incidents are separate facts. See freshness and recovery for configuration matching, retention limits, and missing-run detection.

At one Cron invocation per minute, the Worker receives 1,440 scheduled invocations per day regardless of target count. The actual probe total is controlled by the target interval. Workers Observability carries invocation health, CPU time, wall time, subrequest count, and compact run summaries without an unbounded D1 probe history.

Hookrelay events

Delivery uses structured CloudEvents 1.0 with source urn:endpoint-monitor and these types:

  • urn:endpoint-monitor:problem:v1
  • urn:endpoint-monitor:recovered:v1

The exact serialized body is signed as X-Hookrelay-Signature-256: sha256=<hex> using HMAC-SHA256. An outbox retries failed delivery with bounded exponential backoff and holds a resolved transition until its corresponding problem transition has been delivered. Incidents opened while delivery is disabled are bridged into the outbox when delivery is enabled, so shadow mode does not lose an ongoing problem. Operator-dismissed incidents use the recovered event type with resolutionReason: "operator-dismissed" so downstream consumers can close the existing problem without mistaking the action for a successful probe.

HTTP and diagnostics

GET /healthz returns only service liveness. /api/status is hidden unless status is enabled and requires Authorization: Bearer <ENDPOINT_MONITOR_STATUS_TOKEN>. Its protected response includes target URLs, schedule capacity, exceptional states, incidents, acknowledgement and snooze summaries, and delivery backlog. Incident mutation remains available only through the operator CLI.

Custom logs contain fixed event names, target IDs, statuses, bounded error codes, counts, and incident IDs. They exclude URLs, query strings, response headers, exception messages, Hookrelay paths, signatures, HMACs, API response bodies, and secret values.

Portability

The core configuration, scheduler, probes, reducer, event model, and Hookrelay signing code do not depend on Cloudflare. A different runtime adapter supplies scheduling, exceptional-state persistence, delivery, and an HTTP implementation. See Architecture for the adapter contract.

Development

npm run check

npm run check runs the complete local release gate, including package smoke installation and validation of the committed portfolio cover. Install Playwright Chromium with npx playwright install chromium, then run npm run capture:cover to regenerate docs/screenshots/cover.png from its tracked synthetic HTML scene.

CI renders the cover after the release checks and retains it as an artifact. Successful default-branch runs publish a changed image to docs/screenshots/cover.png with an image-only commit. Publication checks that the source revision is still the branch tip; superseded runs leave publication to the newer build. Pull requests render without publishing. The CI workflow also supports manual refreshes.

See the changelog, release procedure, and contribution guidance. The project is licensed under AGPL-3.0-only.

Cover image density

The project cover is rendered at 4x pixel density while preserving its logical viewport, so enlarged previews retain more detail. Higher density does not increase the displayed text size; use zoom to inspect small labels.

About

Monitor public HTTP endpoints, track incidents and recoveries, and send signed alerts through Hookrelay.

Topics

Resources

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages