…ation tier

Move the four tests that require a running EmailEngine server (api-test,
sendonly-test, api-routes-smoke-test, ui-routes-smoke-test) and their two
helper modules into test/integration/. The remaining tests form a unit tier
that needs Redis but no server and now runs with default node --test
concurrency (~17s instead of 60s serial).

- Grunt: separate test-unit and test-integration tasks composed into test
- Test globs match *-test.js only so helper modules are never executed
- Replace the fixed 20s grunt-wait startup delay with a /health poll
  (test/helpers/wait-for-server.js) - readiness is typically detected in 2s
- Add npm run test:unit and npm run test:integration scripts
- Remove the now-unused grunt-wait devDependency and regenerate licenses

Split the single test job into independent lint, unit, and integration jobs
so a failed section (usually the non-hermetic integration tier that talks to
live Gmail/MS Graph) can be re-run alone via re-run failed jobs instead of
repeating the whole suite.

- lint: eslint only, no Redis
- unit: npm run test:unit with the Gmail secrets used by the revoke test
  (which skips itself when they are absent)
- integration: npm run test:integration with the full secrets block
- Enable npm caching in setup-node for all jobs to offset the extra installs

…-457f-vx79)

Resolves both open Dependabot alerts for the joi RangeError advisory
(GHSA-q7cg-457f-vx79):
- direct joi 17.13.3 -> 17.13.4 (patched within the 17.x line)
- @postalsys/certs 1.0.14 -> 1.0.15, which moved its joi dependency back
  to 17.13.4, removing the vulnerable transitive joi 18.0.2 copy

joi stays on 17.x because hapi-swagger's peer dependency requires it.
Replaced the blanket joi reject in .ncurc.js with a target function so
joi still receives 17.x security patches while the 18.x major remains
blocked.

Previously the Gmail/Outlook batch fetch path in the export worker only
retried HTTP 429 / rate-limit responses. Any other per-message error -
including transient 5xx/network blips and Outlook's dropped-batch-response
(EMISSING_RESPONSE) - was rethrown immediately, failing the entire export
job (partial file deleted, no resume). A single transient error on one
message could therefore kill an export of hundreds of thousands of messages.

The batch path now retries the same transient classes the IMAP/sequential
path already retries, plus rate limits and dropped batch responses, bounded
by MAX_BATCH_RETRIES with exponential backoff; it still fails only once the
retry budget is exhausted. Extracts the classification into a shared
isRetryableError() helper in lib/export.js (next to isTransientError/
isSkippableError/isFolderMissingError) and adds unit tests.

indexMessages retried every per-folder error 3 times and then swallowed
it, continuing to the next folder. A systemic failure during indexing -
account deleted, OAuth revoked, or the connection down for the whole
indexing window - made every folder fail silently, queued zero messages,
and the export was still marked "completed". The caller received a
successful but silently-incomplete (often empty) export with no failure
signal.

Indexing now retries only transient errors and, on a permanent error or
exhausted retries, throws to fail the whole export (firing an exportFailed
webhook). A folder deleted mid-export is still treated as empty inside
indexFolder, so only real failures abort. This also removes pointless
retry cycles across every folder on systemic failures.

…etes

The BullMQ export Worker was constructed with the default autorun:true, so
it began pulling jobs at construction - concurrently with the startup IIFE
running markInterruptedAsFailed() and cleanup(). A queued export could be
picked up and processed by the worker while recovery concurrently marked
that same export failed and unlinked its file, leading to status flapping,
a freed concurrency slot for a still-running export, or a completed export
whose file had been deleted.

Construct the worker with autorun:false and call run() from the startup
IIFE only after recovery finishes. run() is fire-and-forget (it resolves
only on worker close) and mirrors BullMQ's own autorun error handling:
a fatal run() failure is re-emitted as an 'error' event which, with no
listener, crashes the worker thread so the main process restarts it.

The Document Store (ElasticSearch indexing) is now inactive unless EmailEngine
is started with the documentStore.enabled gate: the --documentStore.enabled CLI
flag, [documentStore] enabled in config, or EENGINE_DOCUMENT_STORE_ENABLED=true.

When the gate is off (the new default):
- the documents worker is not spawned;
- every document-store-only endpoint (/v1/chat/{account}, /v1/unified/search,
  /admin/config/document-store/*) is unregistered and returns a regular 404;
- runtime document-store code takes the existing "disabled" path via the new
  isDocumentStoreEnabled() helper (sync gate AND the documentStoreEnabled
  setting), so behavior is identical to today when the setting was off;
- the account-deletion notification is no longer enqueued to the documents
  queue, avoiding job pileup with no consumer;
- the admin dashboard shows an error alert when the documentStoreEnabled
  setting is still on but the feature is unavailable.

When the gate is on, behavior is exactly as before. Tests keep the feature
enabled via config/test.toml, with a new test asserting the off-state 404s.

Piggyback a compact, anonymized feature snapshot onto the existing daily
license-validation POST so we can tell whether deprecation-candidate features
are still used in the field.

The snapshot (`diag`) carries only enabled-feature codes, mailbox provider
types, coarse magnitude tiers (buckets, not raw counts), exercised-usage
signals, a deprecation watchlist, and runtime context - no PII, URLs, or
secrets. To avoid resending it on every call, only a short digest (`fh`) rides
along each time; the full snapshot is sent only when the digest changes or once
every 30 days. Collection is best-effort and time-boxed, so it can never block,
delay, or alter license validation. Operators can opt out with
EENGINE_BEACON_DISABLED=true.

Coverage is limited to active subscription licenses (the only ones that call
home today). Adds lib/license-beacon.js, a listEnabled() export on the feature
flags module, and unit tests.

undici raises every connection failure as a generic TypeError (e.g. "fetch
failed") with the real DNS/socket error on err.cause. Two problems followed
from this:

- lib/logger.js forwarded any logged TypeError/RangeError to error tracking,
  so transient environmental blips (getaddrinfo EAI_AGAIN reaching Google)
  piled up as useless "fetch failed" Sentry issues. Skip forwarding when the
  cause is a known transient network code; genuine code bugs (no network errno
  cause) are still reported.
- The shared undici RetryAgent did not retry EAI_AGAIN or ETIMEDOUT (not in
  undici's default errorCodes), so a DNS hiccup bubbled straight up instead of
  being retried. Add an explicit errorCodes list (undici defaults + EAI_AGAIN
  + ETIMEDOUT) so transient name-resolution failures are retried away.

Adds test/logger-sentry-filter-test.js covering the forwarding filter.

Refresh dependencies via npm-check-updates and regenerate the lockfile,
SBOM, bundled license list and the gettext template. Notable bumps:
nodemailer 8.0.11 -> 9.0.0 (includes GHSA-55pm-vh62-rwjg: validate TLS
certificates by default when fetching remote content), plus @sentry/node,
bullmq, mailparser, smtp-server, pubface, @postalsys/email-ai-tools and
@postalsys/email-text-tools. Also refresh data/google-crawlers.json.

Pin nodemailer's bundled data files in pkg.assets -- lib/well-known/*.json
(service lookup table) and its package.json (version/User-Agent string) --
so they survive the pkg snapshot, matching how other dependency assets
(bull, swagger-ui-dist, bounce-classifier) are already pinned.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Bump imapflow to 1.4.1, which pins nodemailer 9.0.0. This removes the
last duplicate nodemailer copy (the nested 8.0.10 under imapflow) so the
whole dependency tree now dedupes to a single nodemailer 9.0.0.

Add de/et/fr/ja/nl/pl translations for the OAuth scope error view
(Insufficient Permissions, the missing-permissions guidance, Try Again)
and recompile the .mo files. The remaining diff is source-reference and
line-wrap churn from regenerating the catalogs against the updated .pot.

…er--components--emailengine-app

chore(master): release 2.71.0