Changes in v0.3.7

Scheduler Correctness and Observability

• Fixed unbounded timer-queue growth when collectors ran slower than their period: TimerScheduler previously pushed a callback copy every tick, so a collector blocked for 60 s at 20 Hz queued 1200 duplicate callbacks, each re-contending the same collector lock and able to saturate the entire worker pool. Each timer now keeps at most one queued-or-running callback: a tick arriving while the timer is busy increments skipped_runs instead of enqueueing, the ready queue stores timer handles with an upper bound of the number of registered timers, and an overloaded scheduler (deadlines persistently overdue) releases its scheduling lock once per tick so workers, status queries and cancelTimer are not starved (48996ab).
• Added scheduler/collector/job status visibility through new read-only core RPC methods CollectorScheduler/status and CollectorScheduler/collector_status, bridged as the bounded HTTP endpoint GET /joblens/collectors/status: a per-timer view (idle/queued/running, queued/running durations, skipped_runs/completed_runs/error counts, worker index), per-collector operations (init/callback/lookup_job/wait_collector_lock/collect/write with success/error counts, last duration, in-progress entries and job attribution including current_job with stage), and scheduler-level worker/queue statistics. Snapshots copy observational data under short-held locks and render JSON outside them, so the status RPC stays sub-millisecond while callbacks block and remains readable after scheduler destruction (48996ab).

CPUMemCollector Fixes

• Fixed alive-PID baselines being dropped under low-frequency sampling: baselines exceeding the 300 s TTL are now verified against /proc/<pid>/stat and removed only after confirming the process is gone or its starttime changed; baselines are retained on permission or parse failures. Added regression tests covering long-period sampling, PID reuse, and stat read failures (81afb08).

IO Detail Collection

• Fixed IO detail collection blocking on remote filesystems: IOUsageCollector and NewIOUsageCollector no longer call stat() on /proc/<pid>/fd targets, which could hang the entire collect cycle in remote filesystem authentication (for example AFS); file paths are resolved with readlink only and mount information via /proc/<pid>/mountinfo. Two downstream-visible behavior changes: file details are no longer restricted to regular and block devices (directories and deleted-but-open files now appear), and the legacy pos field — which reported file size rather than the current file offset — is now JSON null and text unknown. FileIOInfo is value-initialized to drop the stale st_size dependency (28f811f).

Documentation

• Rewrote doc/collector_scheduler_status.md for the new status API and added an IO file metadata section to doc/configuration.md (e078b32).

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.3.6...ver-0.3.7

Changes in v0.3.6

CPUMemCollector Fixes

• Fixed cpuPercent being inflated for jobs sampled later in the same tick: the system-wide /proc/stat baseline is now cached per PID, so Δproc and Δtotal cover the same interval (8db5461).
• Stale per-PID baselines are now pruned (300 s TTL, swept at most once per minute) and released on deinit(), preventing unbounded growth of pid_state_dict (221a434).

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.3.5...ver-0.3.6

Changes in v0.3.5

Job Discovery Correctness

• Fixed get_ppid_of() reading the pgrp field instead of ppid while parsing /proc/<pid>/stat: HTCondor makes each job process its own process-group leader (pgrp == pid), so starter_pid was recorded as the job's own PID, then filtered out of the cgroup PID list, leaving JobPIDs empty and triggering the job has no running process, delete it path about 200 ms after discovery. The Slurm stepd_pid resolution and the parent-process check in get_slurm_job_pids() were affected the same way (2429a94).

Job Event Drop Observability

• Added a per-CPU job_event_stats map to the job_pid_track BPF program (slot 0 = overflow drops, slot 1 = successful enqueues) incremented in push_event, so events silently discarded when job_event_rb is full are now countable instead of corrupting Job.JobPIDs without any log; JobPidTracker sums the slots across CPUs and exposes pushed_events()/dropped_events() (a2db35e).
• Added the JobRegistry/pid_tracker_stats RPC returning running/pushed_events/dropped_events, and the reconcile cycle now logs a warning when new drops appear, making ringbuf overflow directly observable rather than something to be inferred after the fact (b2af9bf).
• Added GET /joblens/pid_tracker/stats to the Trigger as the HTTP bridge for the RPC above (6ee1413).

Network Usage Collector Correctness

• Extracted SafeRate so TCP counter regressions (reset or wrapped statistics) and zero-length intervals report a rate of 0 instead of an abnormal value (e6cb77a).
• Mixed the socket inode into the connection hash: the previous four-tuple-only key made a recycled four-tuple inherit the stale baseline of the previous connection (e6cb77a).
• Bounded connection_state_dict with a 300 s TTL, swept at most once per 60 s, replacing unbounded growth over long-lived collectors (e6cb77a).

Hardware Inventory Correctness

• Reworked Trigger GPU discovery to match PCI class codes (lspci -nn -d '::0300'/'::0301'/'::0302'/'::0380', with a fallback that filters the full output on [03xx]: for older lspci), replacing the textual grep '(vga|3d|display)' scan that misread 3DNAND in NVMe names and Root Port 3d PCI bridges as GPUs (124a7f6).
• Excluded BMC graphics adapters (vendor IDs 102b/1a03/19e5 and known name keywords) such as Matrox/ASPEED/Huawei iBMC from the compute-GPU list, switched vendor detection to a word-boundary regex so substrings like ati no longer match unrelated names, and strip trailing [vendor:device]/(rev xx) suffixes from the model name (124a7f6).

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.3.4...ver-0.3.5

Changes in v0.3.4

Collector Correctness and CPU Accounting

• Fixed CPUMemCollector reporting astronomical cpuPercent values (up to ~1e15) when a PID was recycled: track per-PID starttime, reset the baseline on PID reuse, skip the delta when utime+stime regresses, and compute the /proc/stat delta once per cycle instead of per PID, preventing unsigned 64-bit underflow across jobs sharing the singleton collector (9c3775e).
• Fixed NewIOUsageCollector deleting still-unseen FDs when a dump was partial or a PID was reused: track dump completeness and compare emitted FDs before removing entries (3a97014).
• Commit rate baselines only after all cancellable work, use RAII for netlink and directory handles, and charge a consumed power sample once per cycle across jobs (5da4536).
• Added cluster_name and clusterTag to persisted job information for both Condor and Slurm (e1695a6).

Collector Performance and Robustness

• Read the system CPU counters once per Job and cache CLK_TCK; use non-blocking netlink with EINTR/EAGAIN retries in the socket-backed collectors (1a393fe).
• Refactored NetUsageCollector to track connection state in an unordered_map, parse /proc/net with string_view, and add socket namespace management and error handling (f8d2160).
• Downgraded high-frequency per-sample debug logs to trace across collectors and writers (7e690b0).

Writer Performance

• Removed redundant std::any/JSON/tuple copies, resolve each collector's parser once per flush through a shared batch parser cache, index Prometheus PIDs while preserving output order, and drain accepted records on explicit shutdown or destruction (d3279a0).

Regression Coverage

• Added a CPUMemCollector regression that fabricates /proc/<pid>/stat to cover PID reuse, counter regression, and normal growth.
• Added NetUsageCollector integration coverage for connection ownership, socket sharing, netlink/namespace races, and writer compatibility, plus a runnable entry point (f8d2160, 1a393fe).
• Extended NewIOUsage cache regression for dump completeness, PID reuse, and same-PID cross-job isolation (3a97014).
• Added PowerCollector cache regression covering per-cycle energy accounting across jobs (5da4536).
• Added writer hot-path regressions for byte batching, buffering, and parser forwarding (d3279a0).

Validation Scope

• Local verification: cmake + Ninja RelWithDebInfo build and cpack -G TGZ succeeded. The CPUMemCollector PID-reuse regression was confirmed to fail against the pre-fix code and pass after the fix.
• Unified RPM/TGZ packaging and cross-distro checks are performed by the Release workflow on the ver-0.3.4 tag; they are not part of this local validation.

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.3.3...ver-0.3.4

Changes in v0.3.4

Collector Correctness and CPU Accounting

• Fixed CPUMemCollector reporting astronomical cpuPercent values (up to ~1e15) when a PID was recycled: track per-PID starttime, reset the baseline on PID reuse, skip the delta when utime+stime regresses, and compute the /proc/stat delta once per cycle instead of per PID, preventing unsigned 64-bit underflow across jobs sharing the singleton collector (9c3775e).
• Fixed NewIOUsageCollector deleting still-unseen FDs when a dump was partial or a PID was reused: track dump completeness and compare emitted FDs before removing entries (3a97014).
• Commit rate baselines only after all cancellable work, use RAII for netlink and directory handles, and charge a consumed power sample once per cycle across jobs (5da4536).
• Added cluster_name and clusterTag to persisted job information for both Condor and Slurm (e1695a6).

Collector Performance and Robustness

• Read the system CPU counters once per Job and cache CLK_TCK; use non-blocking netlink with EINTR/EAGAIN retries in the socket-backed collectors (1a393fe).
• Refactored NetUsageCollector to track connection state in an unordered_map, parse /proc/net with string_view, and add socket namespace management and error handling (f8d2160).
• Downgraded high-frequency per-sample debug logs to trace across collectors and writers (7e690b0).

Writer Performance

• Removed redundant std::any/JSON/tuple copies, resolve each collector's parser once per flush through a shared batch parser cache, index Prometheus PIDs while preserving output order, and drain accepted records on explicit shutdown or destruction (d3279a0).

Regression Coverage

• Added a CPUMemCollector regression that fabricates /proc/<pid>/stat to cover PID reuse, counter regression, and normal growth.
• Added NetUsageCollector integration coverage for connection ownership, socket sharing, netlink/namespace races, and writer compatibility, plus a runnable entry point (f8d2160, 1a393fe).
• Extended NewIOUsage cache regression for dump completeness, PID reuse, and same-PID cross-job isolation (3a97014).
• Added PowerCollector cache regression covering per-cycle energy accounting across jobs (5da4536).
• Added writer hot-path regressions for byte batching, buffering, and parser forwarding (d3279a0).

Validation Scope

• Local verification: cmake + Ninja RelWithDebInfo build and cpack -G TGZ succeeded. The CPUMemCollector PID-reuse regression was confirmed to fail against the pre-fix code and pass after the fix.
• Unified RPM/TGZ packaging and cross-distro checks are performed by the Release workflow on the ver-0.3.4 tag; they are not part of this local validation.

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.3.3...ver-0.3.4

Changes in v0.3.3

Elasticsearch Writer Delivery Hardening

• Reworked ESWriter bulk delivery to build per-document NDJSON entries and chunk requests by a byte budget (max_bulk_bytes, default 1 MiB) in addition to batch_size, keeping payloads within cluster limits when document sizes vary (1300267).
• Added bounded exponential backoff with jitter for retryable failures, configurable via max_retries, retry_initial_backoff_ms, and retry_max_backoff_ms; retryable HTTP statuses (408/429/5xx) and transport errors are retried, while permanent failures are dropped with a warning (1300267).
• Validated every Bulk item against its submitted document (item shape, status range, _id match) so partially failed batches retry only the failed documents and confirmed successes are never resent; ambiguous or inconsistent responses are treated as retryable instead of silently dropped (1300267).
• Added an overridable BaseWriter::on_flush_error() hook invoked outside the queue lock before the buffer is discarded, letting subclasses act on a failed flush without stalling the writer (8eafeeb).

Collector Runtime Observability

• Added CollectorRuntime to track per-collector lifecycle phase, scope/config, job count, timer state, and per-operation (init/deinit/callback/collect/write) success/error counts and durations (7bbd260).
• Exposed CollectorScheduler/status and CollectorScheduler/collector_status JSON-RPC methods returning whole-scheduler or single-collector snapshots; operations are recorded through an RAII guard, including exception paths (7bbd260).
• Fixed jobid_list erase calls that omitted the container end iterator, and initialized collector_state fields to avoid undefined behavior (7bbd260).

Performance and Correctness Fixes

• Eliminated duplicate collector/writer call counting by removing the extra perf counter increment in CollectorRegistry and BaseWriter (b9e497d).
• Indexed the NewIOUsageCollector FD snapshot cache by job_id so collect() and dead-PID cleanup no longer scan the entire cache (c72b44d).
• Initialized connection/process-state structs in NetUsageCollector and default member values in PrometheusExporterWriter to avoid undefined behavior (ce298fc).
• Corrected the addJob log field name from type to native_id (10dfd31).

Regression Coverage and Documentation

• Added ESWriter regression coverage for byte batching, partial failure, permanent errors, timeout, exponential backoff, retry chunking, and invalid configuration, with a runnable entry point (1300267, 6ee0ac7).
• Added collector/writer call-count regression coverage (b9e497d).
• Added NewIOUsage per-job cache index regression covering detail toggles, file details, rate baselines, same-PID cross-job isolation, ephemeral-process cleanup, and empty-cache refresh (c72b44d).
• Added CollectorRuntime status snapshot test coverage (7bbd260).
• Documented the ESWriter byte budget, backoff retry, and failure-handling semantics in the configuration manual (0edc7a9).

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.3.2...ver-0.3.3

Changes in v0.3.2

IO and Filesystem Metadata Fixes

• Fixed lookup_hashmap_batch() discarding valid final-batch entries when libbpf returns -ENOENT, restoring process/file details in NewIOUsageCollector and process details in FSMetadataCollector.
• Fixed HASH map cursor storage and address passing between batches, preventing spurious EFAULT failures and incomplete results for maps larger than one batch.
• Continue successful short batches until the explicit end-of-traversal result rather than assuming a short batch is the final batch.

Trigger Robustness

• Recognize legacy RPC error arrays at the client boundary while preserving legitimate list responses. Unsupported writer info methods now reach HTTP 404 handling, and unavailable performance RPC methods reach HTTP 503 handling instead of response-parsing failures.
• Preserve Flask HTTP exceptions in collector/writer performance routes and retain the existing FIFO fallback when RPC function discovery fails.
• Harden Condor starter discovery with argument-vector ps execution, separate stdout/stderr, exact process-name and static/dynamic slot matching, PID validation, and explicit rejection of ambiguous starters. Preserve Docker container PID discovery.

Diagnostics and Regression Coverage

• Added NewIOUsage configuration, eBPF load, cache, aggregation, cleanup, output, and ES serialization diagnostics, plus shared map batch return/count logging.
• Added 15 Trigger regression cases covering RPC errors, HTTP status handling, slot matching, malformed output, stderr isolation, and Docker container discovery.
• Added real-kernel HASH map regression tools for both collector key/value layouts, covering 0, 1, 1023, 1024, 1025, and 2051 entries with count, uniqueness, and field checks.
• Added live IO/metadata workload and verification scripts. Both test execution nodes passed detail-enabled and detail-disabled checks, including growing aggregates/rates and filesystem error statistics.

Validation Scope

• The fixes were validated on Condor and Slurm test execution nodes using a pre-release 0.3.1 build with build ID 71097aa-fix004. Live checks used manually registered PIDs, serialized document logs, and successful ESWriter HTTP responses; they do not constitute scheduler auto-discovery or browser validation.
• Real Condor Docker-universe end-to-end validation remains blocked by the test pool's no match found scheduling issue. The production source of the reported I0911 parsing input remains unconfirmed.

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.3.1...ver-0.3.2

Changes in v0.3.1

Collector List Rework and Output Detail Control

• Reworked the collector list: removed the deprecated ProcCollector and added cpumem_collector, io_collector, and net_collector (c02fd39). Breaking: ProcCollector is no longer registered, so any configuration enabling it should be updated.
• Added include_process_details configuration to control process/file-level detail output for FSMetadataCollector and NewIOUsageCollector, replacing the previous summary flag (deaab78, 67ae7b4).
• Removed the fs_metadata collector sample section from the minimal example config (c31355a).

Per-Job State Isolation and Aggregation Fixes

• Isolated fs_metadata diff baselines by JobID and added per-job/per-process error-rate statistics, wired to the new Prometheus metric (ba323d5).
• Isolated NewIOUsageCollector ephemeral-process state and diff baselines by JobID, keyed file aggregation by (mount_point, fs_type, path), and merged read/write mean/variance via the new welford_utils (3195a2d).

Unified eBPF Object Resolution

• Added resolve_bpf_obj() to locate .bpf.o objects via a priority probe (JOBLENS_BPF_OBJ_DIR → <executable dir>/bpf_obj/ → install layout), so manually-built binaries can load eBPF objects (e941d12).
• Migrated all collectors, Condor/Slurm watchers, and JobPidTracker to the unified resolver, removing hardcoded bpf_o_path members (36a5bb4, 71804e2, 36c1068, 56d6ea8).

New Common Utilities

• Added mountinfo_utils for unified /proc/<pid>/mountinfo parsing with octal-escape decoding and longest-prefix matching (9974d1a).
• Added welford_utils for online mean/variance merging with saturation and invalid-input handling (42635b7).
• Refactored io_usage onto mountinfo_utils (e1a2b4c).

Trigger Resilience

• Added a /etc/JobLens/trigger/config.yaml → packaged example → built-in fallback chain plus JOBLENS_CONFIG_PATH override (ca77433).
• Made empty etcd configs non-destructive to the local config, and made local config writes atomic (backup → tmp → fsync → replace) with rollback support (0526df2).
• Unified rpc_call() to raise RPCError when the RPC client is unavailable, returning 503 instead of a 500 AttributeError (3ae44a1).

Prometheus

• Exported job_fs_metadata_errors_per_sec error-rate gauges from the Prometheus writer (26b93e7).

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.3.0...ver-0.3.1

Changes in v0.3.0

Kernel-Side Job Attribution (eBPF Shared Pinned Maps)

• Introduced shared pinned-map infrastructure: job_pid_track.bpf.c pins the pid2job/cgroup2job/job_event_rb maps to bpffs (/sys/fs/bpf/joblens) for cross-.bpf.o reuse (1345a60)
• Added kernel-side pid2job/cgroup2job self-maintenance with fork/exit lifecycle tracking, so child-process attribution is propagated and cleaned up in-kernel (9f85614)
• Made JobRegistry own kernel job attribution via a new JobPidTracker that consumes fork/exit ringbuf events and runs a low-frequency reconcile loop (797c54e)
• Converted collectors to shared-map pure readers (PowerCollector no longer writes pid2job itself) with systemd bpffs mount/cleanup guarantees (c7f657c, ee5ac1d)

New Collectors

• FSMetadataCollector: per-job filesystem metadata operation statistics (open/close/getattr/readdir/rename/unlink/xattr/fsync, etc.) with frequency, latency histograms, and error rates via eBPF, plus Prometheus metrics (7af2168, bfa1b96, 34f2809, 1a7ea59)
• NewIOUsageCollector: job/file-centric IO aggregation with VFS syscall-layer semantics, latency histograms, and ephemeral-process handling (399604c); removed the use_ebpf flag and improved speed calculation with floating-point precision (1fc9ef6, 03cd5c8, cea40e4)

Trigger and Runtime

• Removed the service-registration heartbeat interval and added gRPC fork support (GRPC_ENABLE_FORK_SUPPORT + GRPC_POLL_STRATEGY=poll) to prevent gunicorn worker fork core dumps (5e6f348)
• Added configurable log level resolved from the config file and the environment (a3478f8)
• Added initialization/destruction logging to DirectoryWatcher, NetLinkUtils, and TimerScheduler (cf8b8bb)

Configuration and Build

• Added normalize_cgroup_fs_path to canonicalize cgroup filesystem paths (bb09f44)
• Added build metadata support (BUILD_ID/BUILD_TIME) propagated through CMake and the unified RPM build (f319c31)

Bug Fixes

• Fixed FSMetadataCollector failing to load by correcting the fstat tracepoint name to newfstat (fdd9295)
• Fixed cgroup2job map leakage after job completion by caching cgroup_id on Job at add time instead of re-resolving at removal (b90f591)
• Fixed negative variance overflow in NewIO by switching to the correct Welford online algorithm (a4123f7)
• Fixed lookup_hashmap_batch EFAULT during concurrent map deletion by restarting traversal from the head (00caf55)
• Fixed the core service not auto-restarting after RPM upgrade by removing the %preun stop and the redundant %postun restart marker (78fb246)
• Added missing rename/unlink tracepoint hooks (non-*at variants) to FSMetadataCollector so those operations are collected (08e782e)

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.2.3...ver-0.3.0

Changes in v0.2.3

Patch Release - FileWriter Parser V2 and Collector Attribution Fixes

• Added FileWriter plain-text output support and migrated base/extended collectors onto the new FileWriter text parsing path:
  • Introduced the FileWriter write contract for plain-text output (449f1c6)
  • Migrated base and extended collector FileWriter parsing to the text parser flow (0e8d60b, 9da0ddb)
  • Updated deployment, power, and core configuration documentation to describe plain-text FileWriter output (41a473d, 1e05820)
• Added writer parser V2 context and routing support:
  • Added the writer parser V2 context interface (45c8c63)
  • Wired FileWriter and external writer parser V2 invocation paths (7d4ca96, ea03f9f)
  • Documented the writer parser context migration strategy (65759c1)
• Expanded FileWriter/parser regression coverage for output-file management, plain-text assertions, configuration conflict fallback, routing override mode, rotation close behavior, and FileWriter routing/close scenarios (0043b67, 004de52, 71bf1b9, 2e9722d, 68bf981, a02ec7e, 2b14406)
• Improved collector/runtime attribution robustness:
  • Added Docker job support by extracting container names and resolving process PIDs (11725ec)
  • Added tgid fallback for thread-level energy attribution and skipped empty snapshots (73e166e, 0c2ca89)
  • Added debugging context for disappeared processes during collector state handling (0cad7b3)
  • Renamed PowerCollector Elasticsearch fields with sys_, nested job fields, and rapl_ prefixes to distinguish system/IPMI/RAPL metrics (ecf713c, 3b6799b)
• Hardened release/RPM validation:
  • Added Fedora static EPEL dependency support and updated RPM verification/release steps (dd15a04)
  • Added Python ABI dependency validation so RPM artifacts match the build environment (8d6646b)

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.2.2...ver-0.2.3

Changes in v0.2.2

Patch Release - Trigger Gunicorn Fork-Safety Fix

• Fixed intermittent Trigger health-check failures caused by gRPC/etcd fork-safety issues in Gunicorn preload mode:
  • Disabled Gunicorn application preloading for joblens-trigger so etcd3/grpcio resources are initialized inside the worker process instead of being inherited from the master process after fork()
  • Avoided gRPC EventEngine epoll_wait Bad file descriptor errors leaking into /joblens/healthy responses when the health endpoint invokes systemctl via subprocess
  • Reduced risk of Gunicorn worker aborts caused by grpcio EventEngine state inherited across fork boundaries
• Unified runtime version reporting on JobLens -v and passed the packaging version into the Trigger Python package from the RPM spec build flow

Packages

Verify

sha256sum -c SHA256SUMS.txt

Full Changelog: ver-0.2.1...ver-0.2.2