Skip to content

Instantly share code, notes, and snippets.

@louzt
Last active May 20, 2026 14:49
Show Gist options
  • Select an option

  • Save louzt/b8349629b602a782ac98d0cbfd0df0c0 to your computer and use it in GitHub Desktop.

Select an option

Save louzt/b8349629b602a782ac98d0cbfd0df0c0 to your computer and use it in GitHub Desktop.
Zero-Overhead Kernel Triage & Remote Runtime Hardening

Zero-Overhead Kernel Triage & Remote Runtime Hardening

Prepared by: LOUST.PRO Infrastructure & Security
Author: David Mireles
Keywords: Linux Kernel, PSI, cgroups v2, inotify, Redis, PM2, systemd, Remote Development, Wayland, DMS

Abstract: This case study documents a redacted infrastructure hardening engagement across a sovereign VPS and a local Linux operator workstation. The work separated transport, runtime, observability, queue orchestration, and desktop notification concerns while preserving live production services. The core lesson is simple: observability and automation must be cheaper than the incident they are meant to control.

Publication Boundary

This document is intentionally public and redacted. It describes methodology, decision models, and representative outcomes without exposing private topology.

Public in this case study Redacted or generalized
Linux primitives: PSI, inotify, cgroups v2, systemd timers exact hostnames, IP addresses, firewall posture, SSH aliases
representative telemetry and before/after deltas client names, application names, project paths, alert callback routes
pseudocode and architecture diagrams proprietary native engine internals and exact policy thresholds
reusable operational patterns private deployment topology and credentials

Where a concrete value could identify an environment, the document uses placeholders such as compute-node-01, remote workspace, optional task orchestrator, or ***.

I. Context: Remote Development As A Production Failure Domain

Remote development tooling is often treated as harmless user-space convenience. On a multi-service VPS, it is a real runtime. A connected workspace can spawn file watchers, extension hosts, shell probes, language servers, terminal hosts, and background scans. Those processes share CPU, memory, file descriptors, and I/O with production services.

The first diagnostic boundary was transport separation:

QUIC/SSH transport healthy  ->  Remote agent connects
Runtime path unhealthy      ->  File explorer, ptyHost, ExtensionHost, or watcher stalls

A successful encrypted tunnel and public-key authentication do not prove the remote workspace is healthy. In the observed deployment, transport was stable while the runtime layer was competing with production workloads and waiting on shell/environment probes.

flowchart LR
    Operator[Operator Workstation] -->|Encrypted SSH/QUIC| Host[Redacted VPS]
    Host --> Agent[Remote Development Agent]
    Agent --> Pty[ptyHost / shell probes]
    Agent --> Watcher[fileWatcher / inotify]
    Agent --> Extensions[ExtensionHost]
    Agent --> Services[Production Services]

    Pty --> Kernel[Linux Kernel]
    Watcher --> Kernel
    Extensions --> Kernel
    Services --> Kernel

    Kernel --> PSI[Pressure Stall Information]
    Kernel --> FDInfo["/proc fdinfo"]
    Kernel --> Scheduler[Scheduler Priority]

    classDef operator fill:#f3f4f6,stroke:#4b5563,color:#111827
    classDef runtime fill:#fff7ed,stroke:#ea580c,color:#431407
    classDef kernel fill:#eff6ff,stroke:#2563eb,color:#172554
    classDef prod fill:#ecfdf5,stroke:#16a34a,color:#052e16
    class Operator operator
    class Agent,Pty,Watcher,Extensions runtime
    class Kernel,PSI,FDInfo,Scheduler kernel
    class Services prod
Loading

II. Metadata Fast-Fail: Making Absence Explicit

Remote shell startup can stall when tooling probes metadata endpoints that do not exist on the current host class. On a plain KVM or bare-metal environment, those probes wait for network timeout instead of failing immediately.

The mitigation is not to remove every possible caller. The safer baseline is to make the absence explicit at the kernel boundary:

# Representative pattern only. Apply through local change control.
ip route replace unreachable <link-local-metadata-address>/32 metric ***

The same principle applies to DNS names used by metadata helpers: resolve absent provider metadata to a fast local failure rather than letting each shell startup discover the absence through a timeout.

This turns an intermittent remote-development failure into a stable, testable baseline.

III. Observer Effect: The Inotify Fork-Bomb Antipattern

The watchdog originally needed to count inotify watches held by remote workspace processes. A naive implementation checks every file descriptor with a separate child process:

# Antipattern: one grep process per descriptor.
for fdinfo in /proc/$pid/fdinfo/*; do
  grep -q '^inotify' "$fdinfo"
done

That approach becomes part of the incident. When the host is already under CPU or I/O pressure, thousands of short-lived helper processes create scheduler churn and inflate the load that the watchdog is trying to measure.

The corrected approach delegates traversal to one compiled tool over selected fdinfo trees:

# Constant child-process overhead; traversal happens inside one grep process.
grep_out=$(grep -rsn '^inotify' /proc/***/fdinfo 2>/dev/null || true)

The optimization is not aesthetic. It changes the monitoring loop from an observer that competes with the system to a bounded probe that can still run during pressure.

IV. Kernel-Aware Signals: PSI Over Free RAM

Free memory is a misleading primary health signal on Linux. A healthy server may show little free memory because page cache is doing useful work. The more important question is whether tasks are waiting.

Pressure Stall Information answers that question:

/proc/pressure/cpu
/proc/pressure/memory
/proc/pressure/io
Signal What it indicates Operational use
cpu.some avg10 at least one runnable task waited for CPU early degradation and priority changes
memory.some avg10 at least one task waited for memory reclaim watch for page-cache or allocation pressure
memory.full avg10 all non-idle tasks stalled on memory critical containment candidate
io.some avg10 tasks waited for I/O avoid adding scans, compression, or broad audits

A bounded watchdog can combine PSI with process CPU, watcher counts, recent logs, and cooldown state. It should avoid broad du, recursive scans over project roots, or repeated process walks unless each command is directly tied to the next decision.

V. Deterministic Policy Cascade

The decision layer is intentionally boring: a mutually exclusive cascade from strongest protection to safest observation. Lower-severity actions cannot override critical actions.

score = I(load_pressure)
      + I(workspace_cpu_pressure)
      + I(watcher_pressure)
      + I(extension_host_pressure)
      + I(pty_or_metadata_signal)
      + I(psi_cpu_signal)
      + I(psi_io_signal)
      + 2 * I(critical_memory_signal)

Where I(condition) returns 1 when true and 0 otherwise. Exact thresholds are environment-specific and are intentionally not published.

flowchart TD
    Metrics[Metric Vector] --> Cascade{Strict Priority Cascade}

    Cascade -->|sustained critical multi-signal pressure| Server[Recycle remote workspace server]
    Cascade -->|sustained watcher pressure| Watcher[Recycle file watcher]
    Cascade -->|sustained extension pressure| Extension[Recycle extension host]
    Cascade -->|early watcher or extension pressure| Degrade[Lower CPU and I/O priority]
    Cascade -->|baseline drift or metadata signal| Baseline[Refresh fast-fail and excludes]
    Cascade -->|healthy path| Observe[Observe and write audit event]

    Server --> Audit[(JSONL audit trail)]
    Watcher --> Audit
    Extension --> Audit
    Degrade --> Audit
    Baseline --> Audit
    Observe --> Audit

    classDef critical fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
    classDef warn fill:#fef3c7,stroke:#d97706,color:#78350f
    classDef safe fill:#ecfdf5,stroke:#16a34a,color:#052e16
    classDef audit fill:#f3f4f6,stroke:#4b5563,color:#1f2937
    class Server critical
    class Watcher,Extension,Degrade warn
    class Baseline,Observe safe
    class Audit audit
Loading

Representative event shape:

{
  "host": "compute-node-01",
  "mode": "run",
  "action": "degrade-workspace",
  "reason": "pressure-without-restart-threshold",
  "metrics": {
    "loadPct": "***",
    "workspaceCpu": "***",
    "psi": { "cpuSomeAvg10": "***", "ioSomeAvg10": "***" },
    "inotify": { "watches": "***", "usagePct": "***" },
    "decisionScore": "***"
  }
}

The point is traceability: every automated action must be explainable from a bounded metric vector.

VI. systemd Guardrails For Bounded Automation

The watchdog is a systemd oneshot timer, not an interactive terminal loop. The service starts, gathers bounded telemetry, applies conservative remediation if needed, writes audit state, and exits.

Important guardrails:

Guardrail Purpose
non-blocking lock overlapping timer runs skip instead of piling up
TimeoutStartSec systemd bounds slow oneshot runs
idle I/O scheduling the observer yields to production work
cooldown state repeated restarts require sustained evidence
latest-state JSON desktop or chat surfaces can consume state without reading logs

A skip-locked event is not a failure by itself. It means a bounded previous run is still active and the next timer tick declined to add more load.

VII. Redis And Queue Orchestration Audit

High Redis client counts are not automatically evidence of a hostile scan. The first checks are exposure and binding:

bind            127.0.0.1 -::1
protected-mode  yes
port            6379

In the redacted deployment, Redis was not listening on a public interface. The high client count came from local queue orchestration: BullMQ workers, queue event listeners, and stream consumers. Blocked commands such as bzpopmin and xread were expected wait states, not necessarily stuck work.

Representative observation from the audit:

Signal Before optional worker shutdown After optional worker shutdown
Redis clients 59 18
Blocked clients 23 0
Evicted keys 0 0
Redis memory low single-digit MB low single-digit MB

This changed the conclusion. The issue was not an exposed Redis socket. It was an optional background orchestrator with many queue connections that did not need to run continuously.

flowchart LR
    PM2[PM2 Process List] --> Prod[Production APIs]
    PM2 --> Dev[Development APIs]
    PM2 --> Worker[Optional Task Orchestrator]

    Worker --> BullMQ[BullMQ Workers]
    Worker --> QueueEvents[QueueEvents]
    BullMQ --> Redis[(Local Redis)]
    QueueEvents --> Redis
    Prod --> Redis

    Dev -. off by default .-> PM2
    Worker -. explicit start only .-> PM2

    classDef prod fill:#ecfdf5,stroke:#16a34a,color:#052e16
    classDef optional fill:#fef3c7,stroke:#d97706,color:#78350f
    classDef redis fill:#eff6ff,stroke:#2563eb,color:#172554
    class Prod prod
    class Dev,Worker,BullMQ,QueueEvents optional
    class Redis redis
Loading

VIII. PM2 Lifecycle Boundaries

pm2 stop is not enough when a later pm2 restart all, deploy script, or resurrected dump can revive a process that should be off by default.

The safer lifecycle model is:

  1. keep production APIs online and named explicitly;
  2. remove development APIs from the live PM2 dump;
  3. keep development activation scripts available for debugging;
  4. rename optional workers according to what they actually do;
  5. start optional workers explicitly only when the workflow needs them.

Representative activation pattern:

# Example only; process name redacted.
pm2 start ecosystem.config.cjs --only optional-i18n-ai-task-orchestrator

This preserves developer ergonomics while preventing idle development services or optional background jobs from consuming Redis connections and CPU indefinitely.

IX. Redis KEYS To SCAN

One runtime hardening issue was a resolver that enumerated namespaced Redis keys with KEYS. KEYS is convenient, but it can block Redis while scanning the keyspace. The low-risk migration preserves the existing key schema and changes only enumeration:

type RedisScanner = {
  scan(
    cursor: string,
    matchKeyword: "MATCH",
    pattern: string,
    countKeyword: "COUNT",
    count: string,
  ): Promise<[string, string[]]>;
};

async function scanRedisKeys(
  redis: RedisScanner,
  pattern: string,
  count = 100,
): Promise<string[]> {
  if (!pattern || pattern.includes("\n") || pattern.includes("\r")) {
    throw new Error("invalid Redis key pattern");
  }

  let cursor = "0";
  const keys = new Set<string>();

  do {
    const [nextCursor, batch] = await redis.scan(
      cursor,
      "MATCH",
      pattern,
      "COUNT",
      String(count),
    );

    cursor = nextCursor;
    for (const key of batch) {
      keys.add(key);
    }
  } while (cursor !== "0");

  return [...keys];
}

Important details:

  • COUNT is a work hint, not a hard page size;
  • SCAN can return duplicate keys, so the helper de-duplicates results;
  • the migration should preserve existing key names first;
  • if this path becomes a hot endpoint over a large keyspace, add an explicit secondary index instead of turning SCAN into a permanent query engine.

This is a good example of production hardening that does not require a schema migration. It removes a blocking command first and leaves deeper indexing decisions for a separate change.

X. Desktop Integration: Pull, Do Not Push

A remote VPS should not attempt to speak directly to a Wayland notification daemon on a laptop. That would require exposing a listener or adding a push path through a network boundary that does not need to exist.

The safer pattern is pull-based:

  1. the VPS writes the latest watchdog event to a local JSON file;
  2. the operator workstation polls that file over the existing encrypted SSH/QUIC path;
  3. the workstation caches it under a local runtime directory;
  4. a notification or desktop widget reacts only when the event signature changes.
sequenceDiagram
    participant VPS as Redacted VPS
    participant JSON as Latest State JSON
    participant SSH as Existing SSH/QUIC Path
    participant Laptop as Operator Workstation
    participant DMS as Local DMS / Notification Surface

    VPS->>JSON: write latest watchdog event atomically
    Laptop->>SSH: pull JSON with BatchMode SSH
    SSH->>JSON: cat latest state
    JSON-->>Laptop: sanitized event payload
    Laptop->>Laptop: cache and compare signature
    Laptop->>DMS: notify only on action/reason change
Loading

This provides desktop visibility with no inbound port on the laptop, no alert callback secret in public configuration, and no dependency on the VPS reaching a local D-Bus session.

A representative local unit shape looks like this. The names are intentionally generic; host aliases, paths, and alert routing stay deployment-specific.

# ~/.config/systemd/user/loust-watchdog-dms-poll.service
[Unit]
Description=LOUST Watchdog Desktop Poll Service

[Service]
Type=oneshot
ExecStart=%h/.local/bin/loust-watchdog-dms-poll
# ~/.config/systemd/user/loust-watchdog-dms-poll.timer
[Unit]
Description=LOUST Watchdog Desktop Poll Timer

[Timer]
OnBootSec=1min
OnUnitActiveSec=10sec
AccuracySec=1sec

[Install]
WantedBy=timers.target
systemctl --user daemon-reload
systemctl --user enable --now loust-watchdog-dms-poll.timer

The poller itself should use non-interactive SSH, a short connection timeout, JSON validation, and signature-based notification suppression. That keeps the desktop bridge asymmetric: the VPS only writes state, while the trusted operator workstation decides when to fetch and display it.

XI. Control Plane vs Data Plane For Migrations

The same failure-domain separation applies to large migrations. Moving source code, manifests, and configuration shape over SSH is appropriate. Pushing multi-gigabyte media archives through a local workstation is often unnecessary.

sequenceDiagram
    participant Operator as Operator Workstation
    participant Source as Source Host
    participant Target as Target Host

    Operator->>Source: inspect deployment shape and asset manifest
    Operator->>Target: sync code, config shape, and manifest over SSH
    Target->>Source: pull heavy static assets over HTTPS edge
    Target->>Target: verify checksums and local routing
    Operator->>Target: run smoke tests before DNS cutover
Loading

The operator controls the migration, but the heavy data plane moves directly between hosts over provider backbones.

XII. Representative Validation

The following values are representative observations from a redacted deployment. They are included to show the validation style, not as universal thresholds.

Area Result
PM2 development services removed from the live dump; still activable through scripts
Optional queue orchestrator renamed by function and left off by default
Redis clients 59 -> 18 after optional orchestrator shutdown
Redis blocked clients 23 -> 0 after optional orchestrator shutdown
Redis key enumeration runtime KEYS usage migrated to SCAN MATCH ... COUNT ...
PM2 log storage hundreds of MB reduced to KB-scale active logs after safe cleanup
Watchdog deployment latest-state JSON emitted for desktop pull integration
Desktop integration local cache populated; notifications gated by signature changes

The most important validation was not a single number. It was the causal chain: identify the owner of resource pressure, remove idle or optional processes from the default runtime, and verify that the host becomes quieter without breaking production services.

XIII. Native Engine Roadmap: Event-Driven PSI

The current reference implementation can remain shell and systemd while the policy is learned from production traces. That is a practical first step: shell is inspectable, deployable, and easy to roll back.

The long-term direction is an event-driven native supervisor that waits on PSI triggers instead of waking on a timer. Linux PSI files are pollable: a process can register a stall threshold and sleep inside epoll_wait(2) until the kernel reports that the threshold was crossed.

// Architectural pseudocode only.
int psi_fd = open("/proc/pressure/memory", O_RDWR | O_NONBLOCK);

// Example threshold. Real policy is workload-specific.
write(psi_fd, "some *** ***", 12);

int epoll_fd = epoll_create1(0);

struct epoll_event event = {};
event.events = EPOLLPRI;
event.data.fd = psi_fd;

epoll_ctl(epoll_fd, EPOLL_CTL_ADD, psi_fd, &event);
epoll_wait(epoll_fd, events, max_events, -1);

dispatch_policy_event();

This section is a roadmap and implementation boundary, not a claim that every reference shim must be rewritten in C++ or Rust. The architectural goal is to align the supervisor with the kernel event model and remove routine polling from the healthy path.

XIV. LOUST.PRO Audit Methodology

The reusable methodology is not a single script. It is a disciplined order of operations:

  1. Separate failure domains: transport, authentication, remote runtime, shell startup, file watching, queues, databases, and notifications are different layers.
  2. Bound the observer: every diagnostic command must be cheaper than the pressure it investigates.
  3. Prefer reversible changes: apply fast-fail routes, excludes, priority changes, and optional-worker shutdowns before broad restarts.
  4. Measure causal deltas: change one owner of pressure at a time and record before/after metrics.
  5. Keep private surfaces private: publish architecture and reasoning, not topology or alert routes.

This is how a VPS stops being a pile of processes and becomes an operable platform.

Implementation Scope & Reference Track

This case study is an open-knowledge artifact. It explains public Linux primitives, diagnostic methodology, and operational reasoning. It is not a full product specification and does not publish private policy tables, deployment topology, or proprietary native engine internals.

Good candidates for public reference shims:

  • bounded inotify descriptor scans;
  • metadata fast-fail checks;
  • JSONL audit event shapes;
  • systemd oneshot/timer validation harnesses;
  • Redis KEYS to SCAN migration helpers.

Implementation-specific systems that require separate review:

  • long-running native supervisors;
  • customer-specific policy thresholds;
  • desktop widgets wired to private host aliases;
  • chat-bot, alert callback, or incident-response bridges;
  • commercial packaging and support boundaries.

Conclusion

Scaling remote infrastructure safely requires treating observability as a first-class resource consumer. The watcher, queue worker, alert bridge, and diagnostic script can all become part of the outage if they are not bounded.

The practical pattern is consistent across remote development, Redis-backed queues, systemd timers, desktop Linux notifications, and migration pipelines: isolate failure domains early, make automated mitigation traceable, and keep the control plane lighter than the workload it protects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment