PLATFRM-184 Explore reducing Redis memory usage for non-expiring keys

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Greptile Summary

This PR upgrades BullMQ from ^5.4.2 to ^5.67.3 and migrates all recurring queue jobs from the legacy repeatable jobs API (queue.add with repeat option) to the newer upsertJobScheduler API. This addresses uncapped Redis memory usage caused by the old repeatable jobs implementation, which accumulated Redis keys over time. The PR also adds three new methods to the queue service (upsertJobScheduler, removeJobScheduler, getJobSchedulers), marks the old APIs as @deprecated, and improves type safety by using Partial<Record> for queue/worker containers with optional chaining.

• Migration gap — legacy repeatable jobs not cleaned up: The old code explicitly called stopRepeatableJob before setting up schedules to remove stale repeatable job configs from Redis. These cleanup calls are removed without replacement. Since BullMQ's job schedulers and legacy repeatable jobs are independent data structures in Redis, any existing legacy repeatable jobs will continue to fire alongside the new schedulers after deployment, causing duplicate job executions for every affected queue (~15 queues). A one-time cleanup step should be added.
• Dynamic secret lease queue: Correctly migrated stopRepeatableJobByJobId → stopJobById for delayed (non-repeatable) jobs, and removed a redundant duplicate call.
• Telemetry queue: Previously, stopRepeatableJob was called unconditionally (even when postHog was undefined) to ensure old jobs were always cleaned up. The new code only calls upsertJobScheduler inside the if (postHog) block, so legacy jobs on instances without PostHog will never be cleaned.

Confidence Score: 2/5

• This PR risks duplicate job executions on any deployment upgrading from the previous version with existing Redis state.
• The migration from legacy repeatable jobs to job schedulers is well-structured, but the removal of all stopRepeatableJob cleanup calls without a replacement migration mechanism means existing deployments will have both legacy repeatable jobs AND new scheduler jobs firing simultaneously. This affects ~15 queues including critical ones like certificate rotation, secret rotation, telemetry, and resource cleanup. The fix is straightforward (add a legacy cleanup step), but without it the PR introduces a high-impact regression on upgrades.
• Pay close attention to backend/src/queue/queue-service.ts (missing legacy job cleanup in upsertJobScheduler) and backend/src/services/telemetry/telemetry-queue.ts (conditional cleanup path change).

Important Files Changed

_{Last reviewed commit: 402ece2}