Prepared by: LOUST.PRO Infrastructure & Security
Author: David Mireles
Keywords: Linux Kernel, PSI, cgroups v2, inotify, Redis, PM2, systemd, Remote Development, Wayland, DMS
Abstract: This case study documents a redacted infrastructure hardening engagement across a sovereign VPS and a local Linux operator workstation. The work separated transport, runtime, observability, queue orchestration, and desktop notification concerns while preserving live production services. The core lesson is simple: observability and automation must be cheaper than the incident they are meant to control.
This document is intentionally public and redacted. It describes methodology, decision models, and representative outcomes without exposing private topology.
| Public in this case study | Redacted or generalized |
|---|---|
| Linux primitives: PSI, inotify, cgroups v2, systemd timers | exact hostnames, IP addresses, firewall posture, SSH aliases |
| representative telemetry and before/after deltas | client names, application names, project paths, alert callback routes |
| pseudocode and architecture diagrams | proprietary native engine internals and exact policy thresholds |
| reusable operational patterns | private deployment topology and credentials |
Where a concrete value could identify an environment, the document uses placeholders such as compute-node-01, remote workspace, optional task orchestrator, or ***.
Remote development tooling is often treated as harmless user-space convenience. On a multi-service VPS, it is a real runtime. A connected workspace can spawn file watchers, extension hosts, shell probes, language servers, terminal hosts, and background scans. Those processes share CPU, memory, file descriptors, and I/O with production services.
The first diagnostic boundary was transport separation:
QUIC/SSH transport healthy -> Remote agent connects
Runtime path unhealthy -> File explorer, ptyHost, ExtensionHost, or watcher stalls
A successful encrypted tunnel and public-key authentication do not prove the remote workspace is healthy. In the observed deployment, transport was stable while the runtime layer was competing with production workloads and waiting on shell/environment probes.
flowchart LR
Operator[Operator Workstation] -->|Encrypted SSH/QUIC| Host[Redacted VPS]
Host --> Agent[Remote Development Agent]
Agent --> Pty[ptyHost / shell probes]
Agent --> Watcher[fileWatcher / inotify]
Agent --> Extensions[ExtensionHost]
Agent --> Services[Production Services]
Pty --> Kernel[Linux Kernel]
Watcher --> Kernel
Extensions --> Kernel
Services --> Kernel
Kernel --> PSI[Pressure Stall Information]
Kernel --> FDInfo["/proc fdinfo"]
Kernel --> Scheduler[Scheduler Priority]
classDef operator fill:#f3f4f6,stroke:#4b5563,color:#111827
classDef runtime fill:#fff7ed,stroke:#ea580c,color:#431407
classDef kernel fill:#eff6ff,stroke:#2563eb,color:#172554
classDef prod fill:#ecfdf5,stroke:#16a34a,color:#052e16
class Operator operator
class Agent,Pty,Watcher,Extensions runtime
class Kernel,PSI,FDInfo,Scheduler kernel
class Services prod
Remote shell startup can stall when tooling probes metadata endpoints that do not exist on the current host class. On a plain KVM or bare-metal environment, those probes wait for network timeout instead of failing immediately.
The mitigation is not to remove every possible caller. The safer baseline is to make the absence explicit at the kernel boundary:
# Representative pattern only. Apply through local change control.
ip route replace unreachable <link-local-metadata-address>/32 metric ***The same principle applies to DNS names used by metadata helpers: resolve absent provider metadata to a fast local failure rather than letting each shell startup discover the absence through a timeout.
This turns an intermittent remote-development failure into a stable, testable baseline.
The watchdog originally needed to count inotify watches held by remote workspace processes. A naive implementation checks every file descriptor with a separate child process:
# Antipattern: one grep process per descriptor.
for fdinfo in /proc/$pid/fdinfo/*; do
grep -q '^inotify' "$fdinfo"
doneThat approach becomes part of the incident. When the host is already under CPU or I/O pressure, thousands of short-lived helper processes create scheduler churn and inflate the load that the watchdog is trying to measure.
The corrected approach delegates traversal to one compiled tool over selected fdinfo trees:
# Constant child-process overhead; traversal happens inside one grep process.
grep_out=$(grep -rsn '^inotify' /proc/***/fdinfo 2>/dev/null || true)The optimization is not aesthetic. It changes the monitoring loop from an observer that competes with the system to a bounded probe that can still run during pressure.
Free memory is a misleading primary health signal on Linux. A healthy server may show little free memory because page cache is doing useful work. The more important question is whether tasks are waiting.
Pressure Stall Information answers that question:
/proc/pressure/cpu
/proc/pressure/memory
/proc/pressure/io
| Signal | What it indicates | Operational use |
|---|---|---|
cpu.some avg10 |
at least one runnable task waited for CPU | early degradation and priority changes |
memory.some avg10 |
at least one task waited for memory reclaim | watch for page-cache or allocation pressure |
memory.full avg10 |
all non-idle tasks stalled on memory | critical containment candidate |
io.some avg10 |
tasks waited for I/O | avoid adding scans, compression, or broad audits |
A bounded watchdog can combine PSI with process CPU, watcher counts, recent logs, and cooldown state. It should avoid broad du, recursive scans over project roots, or repeated process walks unless each command is directly tied to the next decision.
The decision layer is intentionally boring: a mutually exclusive cascade from strongest protection to safest observation. Lower-severity actions cannot override critical actions.
score = I(load_pressure)
+ I(workspace_cpu_pressure)
+ I(watcher_pressure)
+ I(extension_host_pressure)
+ I(pty_or_metadata_signal)
+ I(psi_cpu_signal)
+ I(psi_io_signal)
+ 2 * I(critical_memory_signal)
Where I(condition) returns 1 when true and 0 otherwise. Exact thresholds are environment-specific and are intentionally not published.
flowchart TD
Metrics[Metric Vector] --> Cascade{Strict Priority Cascade}
Cascade -->|sustained critical multi-signal pressure| Server[Recycle remote workspace server]
Cascade -->|sustained watcher pressure| Watcher[Recycle file watcher]
Cascade -->|sustained extension pressure| Extension[Recycle extension host]
Cascade -->|early watcher or extension pressure| Degrade[Lower CPU and I/O priority]
Cascade -->|baseline drift or metadata signal| Baseline[Refresh fast-fail and excludes]
Cascade -->|healthy path| Observe[Observe and write audit event]
Server --> Audit[(JSONL audit trail)]
Watcher --> Audit
Extension --> Audit
Degrade --> Audit
Baseline --> Audit
Observe --> Audit
classDef critical fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
classDef warn fill:#fef3c7,stroke:#d97706,color:#78350f
classDef safe fill:#ecfdf5,stroke:#16a34a,color:#052e16
classDef audit fill:#f3f4f6,stroke:#4b5563,color:#1f2937
class Server critical
class Watcher,Extension,Degrade warn
class Baseline,Observe safe
class Audit audit
Representative event shape:
{
"host": "compute-node-01",
"mode": "run",
"action": "degrade-workspace",
"reason": "pressure-without-restart-threshold",
"metrics": {
"loadPct": "***",
"workspaceCpu": "***",
"psi": { "cpuSomeAvg10": "***", "ioSomeAvg10": "***" },
"inotify": { "watches": "***", "usagePct": "***" },
"decisionScore": "***"
}
}The point is traceability: every automated action must be explainable from a bounded metric vector.
The watchdog is a systemd oneshot timer, not an interactive terminal loop. The service starts, gathers bounded telemetry, applies conservative remediation if needed, writes audit state, and exits.
Important guardrails:
| Guardrail | Purpose |
|---|---|
| non-blocking lock | overlapping timer runs skip instead of piling up |
TimeoutStartSec |
systemd bounds slow oneshot runs |
| idle I/O scheduling | the observer yields to production work |
| cooldown state | repeated restarts require sustained evidence |
| latest-state JSON | desktop or chat surfaces can consume state without reading logs |
A skip-locked event is not a failure by itself. It means a bounded previous run is still active and the next timer tick declined to add more load.
High Redis client counts are not automatically evidence of a hostile scan. The first checks are exposure and binding:
bind 127.0.0.1 -::1
protected-mode yes
port 6379
In the redacted deployment, Redis was not listening on a public interface. The high client count came from local queue orchestration: BullMQ workers, queue event listeners, and stream consumers. Blocked commands such as bzpopmin and xread were expected wait states, not necessarily stuck work.
Representative observation from the audit:
| Signal | Before optional worker shutdown | After optional worker shutdown |
|---|---|---|
| Redis clients | 59 | 18 |
| Blocked clients | 23 | 0 |
| Evicted keys | 0 | 0 |
| Redis memory | low single-digit MB | low single-digit MB |
This changed the conclusion. The issue was not an exposed Redis socket. It was an optional background orchestrator with many queue connections that did not need to run continuously.
flowchart LR
PM2[PM2 Process List] --> Prod[Production APIs]
PM2 --> Dev[Development APIs]
PM2 --> Worker[Optional Task Orchestrator]
Worker --> BullMQ[BullMQ Workers]
Worker --> QueueEvents[QueueEvents]
BullMQ --> Redis[(Local Redis)]
QueueEvents --> Redis
Prod --> Redis
Dev -. off by default .-> PM2
Worker -. explicit start only .-> PM2
classDef prod fill:#ecfdf5,stroke:#16a34a,color:#052e16
classDef optional fill:#fef3c7,stroke:#d97706,color:#78350f
classDef redis fill:#eff6ff,stroke:#2563eb,color:#172554
class Prod prod
class Dev,Worker,BullMQ,QueueEvents optional
class Redis redis
pm2 stop is not enough when a later pm2 restart all, deploy script, or resurrected dump can revive a process that should be off by default.
The safer lifecycle model is:
- keep production APIs online and named explicitly;
- remove development APIs from the live PM2 dump;
- keep development activation scripts available for debugging;
- rename optional workers according to what they actually do;
- start optional workers explicitly only when the workflow needs them.
Representative activation pattern:
# Example only; process name redacted.
pm2 start ecosystem.config.cjs --only optional-i18n-ai-task-orchestratorThis preserves developer ergonomics while preventing idle development services or optional background jobs from consuming Redis connections and CPU indefinitely.
One runtime hardening issue was a resolver that enumerated namespaced Redis keys with KEYS. KEYS is convenient, but it can block Redis while scanning the keyspace. The low-risk migration preserves the existing key schema and changes only enumeration:
type RedisScanner = {
scan(
cursor: string,
matchKeyword: "MATCH",
pattern: string,
countKeyword: "COUNT",
count: string,
): Promise<[string, string[]]>;
};
async function scanRedisKeys(
redis: RedisScanner,
pattern: string,
count = 100,
): Promise<string[]> {
if (!pattern || pattern.includes("\n") || pattern.includes("\r")) {
throw new Error("invalid Redis key pattern");
}
let cursor = "0";
const keys = new Set<string>();
do {
const [nextCursor, batch] = await redis.scan(
cursor,
"MATCH",
pattern,
"COUNT",
String(count),
);
cursor = nextCursor;
for (const key of batch) {
keys.add(key);
}
} while (cursor !== "0");
return [...keys];
}Important details:
COUNTis a work hint, not a hard page size;SCANcan return duplicate keys, so the helper de-duplicates results;- the migration should preserve existing key names first;
- if this path becomes a hot endpoint over a large keyspace, add an explicit secondary index instead of turning
SCANinto a permanent query engine.
This is a good example of production hardening that does not require a schema migration. It removes a blocking command first and leaves deeper indexing decisions for a separate change.
A remote VPS should not attempt to speak directly to a Wayland notification daemon on a laptop. That would require exposing a listener or adding a push path through a network boundary that does not need to exist.
The safer pattern is pull-based:
- the VPS writes the latest watchdog event to a local JSON file;
- the operator workstation polls that file over the existing encrypted SSH/QUIC path;
- the workstation caches it under a local runtime directory;
- a notification or desktop widget reacts only when the event signature changes.
sequenceDiagram
participant VPS as Redacted VPS
participant JSON as Latest State JSON
participant SSH as Existing SSH/QUIC Path
participant Laptop as Operator Workstation
participant DMS as Local DMS / Notification Surface
VPS->>JSON: write latest watchdog event atomically
Laptop->>SSH: pull JSON with BatchMode SSH
SSH->>JSON: cat latest state
JSON-->>Laptop: sanitized event payload
Laptop->>Laptop: cache and compare signature
Laptop->>DMS: notify only on action/reason change
This provides desktop visibility with no inbound port on the laptop, no alert callback secret in public configuration, and no dependency on the VPS reaching a local D-Bus session.
A representative local unit shape looks like this. The names are intentionally generic; host aliases, paths, and alert routing stay deployment-specific.
# ~/.config/systemd/user/loust-watchdog-dms-poll.service
[Unit]
Description=LOUST Watchdog Desktop Poll Service
[Service]
Type=oneshot
ExecStart=%h/.local/bin/loust-watchdog-dms-poll# ~/.config/systemd/user/loust-watchdog-dms-poll.timer
[Unit]
Description=LOUST Watchdog Desktop Poll Timer
[Timer]
OnBootSec=1min
OnUnitActiveSec=10sec
AccuracySec=1sec
[Install]
WantedBy=timers.targetsystemctl --user daemon-reload
systemctl --user enable --now loust-watchdog-dms-poll.timerThe poller itself should use non-interactive SSH, a short connection timeout, JSON validation, and signature-based notification suppression. That keeps the desktop bridge asymmetric: the VPS only writes state, while the trusted operator workstation decides when to fetch and display it.
The same failure-domain separation applies to large migrations. Moving source code, manifests, and configuration shape over SSH is appropriate. Pushing multi-gigabyte media archives through a local workstation is often unnecessary.
sequenceDiagram
participant Operator as Operator Workstation
participant Source as Source Host
participant Target as Target Host
Operator->>Source: inspect deployment shape and asset manifest
Operator->>Target: sync code, config shape, and manifest over SSH
Target->>Source: pull heavy static assets over HTTPS edge
Target->>Target: verify checksums and local routing
Operator->>Target: run smoke tests before DNS cutover
The operator controls the migration, but the heavy data plane moves directly between hosts over provider backbones.
The following values are representative observations from a redacted deployment. They are included to show the validation style, not as universal thresholds.
| Area | Result |
|---|---|
| PM2 development services | removed from the live dump; still activable through scripts |
| Optional queue orchestrator | renamed by function and left off by default |
| Redis clients | 59 -> 18 after optional orchestrator shutdown |
| Redis blocked clients | 23 -> 0 after optional orchestrator shutdown |
| Redis key enumeration | runtime KEYS usage migrated to SCAN MATCH ... COUNT ... |
| PM2 log storage | hundreds of MB reduced to KB-scale active logs after safe cleanup |
| Watchdog deployment | latest-state JSON emitted for desktop pull integration |
| Desktop integration | local cache populated; notifications gated by signature changes |
The most important validation was not a single number. It was the causal chain: identify the owner of resource pressure, remove idle or optional processes from the default runtime, and verify that the host becomes quieter without breaking production services.
The current reference implementation can remain shell and systemd while the policy is learned from production traces. That is a practical first step: shell is inspectable, deployable, and easy to roll back.
The long-term direction is an event-driven native supervisor that waits on PSI triggers instead of waking on a timer. Linux PSI files are pollable: a process can register a stall threshold and sleep inside epoll_wait(2) until the kernel reports that the threshold was crossed.
// Architectural pseudocode only.
int psi_fd = open("/proc/pressure/memory", O_RDWR | O_NONBLOCK);
// Example threshold. Real policy is workload-specific.
write(psi_fd, "some *** ***", 12);
int epoll_fd = epoll_create1(0);
struct epoll_event event = {};
event.events = EPOLLPRI;
event.data.fd = psi_fd;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, psi_fd, &event);
epoll_wait(epoll_fd, events, max_events, -1);
dispatch_policy_event();This section is a roadmap and implementation boundary, not a claim that every reference shim must be rewritten in C++ or Rust. The architectural goal is to align the supervisor with the kernel event model and remove routine polling from the healthy path.
The reusable methodology is not a single script. It is a disciplined order of operations:
- Separate failure domains: transport, authentication, remote runtime, shell startup, file watching, queues, databases, and notifications are different layers.
- Bound the observer: every diagnostic command must be cheaper than the pressure it investigates.
- Prefer reversible changes: apply fast-fail routes, excludes, priority changes, and optional-worker shutdowns before broad restarts.
- Measure causal deltas: change one owner of pressure at a time and record before/after metrics.
- Keep private surfaces private: publish architecture and reasoning, not topology or alert routes.
This is how a VPS stops being a pile of processes and becomes an operable platform.
This case study is an open-knowledge artifact. It explains public Linux primitives, diagnostic methodology, and operational reasoning. It is not a full product specification and does not publish private policy tables, deployment topology, or proprietary native engine internals.
Good candidates for public reference shims:
- bounded inotify descriptor scans;
- metadata fast-fail checks;
- JSONL audit event shapes;
- systemd oneshot/timer validation harnesses;
- Redis
KEYStoSCANmigration helpers.
Implementation-specific systems that require separate review:
- long-running native supervisors;
- customer-specific policy thresholds;
- desktop widgets wired to private host aliases;
- chat-bot, alert callback, or incident-response bridges;
- commercial packaging and support boundaries.
Scaling remote infrastructure safely requires treating observability as a first-class resource consumer. The watcher, queue worker, alert bridge, and diagnostic script can all become part of the outage if they are not bounded.
The practical pattern is consistent across remote development, Redis-backed queues, systemd timers, desktop Linux notifications, and migration pipelines: isolate failure domains early, make automated mitigation traceable, and keep the control plane lighter than the workload it protects.