Skip to content

Cache DateTime.UtcNow in AAD authenticator hot path#1855

Merged
tiagonapoli merged 52 commits into
mainfrom
tiagonapoli/optimize-aad-isauthorized
Jun 9, 2026
Merged

Cache DateTime.UtcNow in AAD authenticator hot path#1855
tiagonapoli merged 52 commits into
mainfrom
tiagonapoli/optimize-aad-isauthorized

Conversation

@tiagonapoli

@tiagonapoli tiagonapoli commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

Problem

GarnetAadAuthenticator.IsAuthorized() is on the per-command hot path — IsAuthenticated (which forwards to IsAuthorized()) is checked twice per command via AdminCommands.cs:32 and AdminCommands.cs:131.

In a cluster CPU profile under AAD authentication, DateTime.get_UtcNow shows up as the single hottest method at ~62.7% exclusive CPU.

AAD IsAuthorized CPU profile

Fix

Three pieces:

  1. Garnet.common.CoarseTimeProvider — an instance class wrapping any TimeProvider. Exposes UtcNow (DateTime) and UtcNowTicks (long); both read a cached value via a pure memory load. The cache refresh is scheduled via TimeProvider.CreateTimer so a FakeTimeProvider in tests drives refreshes when its clock is advanced — production and tests therefore exercise the same hot path. CoarseTimeProvider.System is the process-wide singleton backed by TimeProvider.System (amortizes the single background Timer across all callers).

  2. Garnet.common.CoarseDateTime — kept as a thin static facade over CoarseTimeProvider.System for back-compat / convenience.

  3. GarnetAadAuthenticator:

    • Constructor takes an optional CoarseTimeProvider (defaults to CoarseTimeProvider.System).
    • AadAuthenticationSettings exposes a public TimeProvider parameter and owns the CoarseTimeProvider lifecycle (reuses .System singleton when no override → zero per-settings Timer cost; owns and disposes a per-settings instance otherwise).
    • Authenticate() precomputes the token validity window in ticks once (_validFromTicks / _validToTicks).
    • IsAuthorized() reads _coarseTime.UtcNowTicks and compares long against long, skipping DateTimes per-call Kind-bit masking.

Microbenchmark

benchmark/BDN.benchmark/Auth/IsAuthorizedBenchmark.cs on Windows 11 24H2, i7-12700K, .NET 10, x64, Server GC, DOTNET_TieredPGO=0:

Method Mean Ratio Notes
IsAuthorized_DateTimeUtcNow 19.62 ns 1.000 baseline
IsAuthorized_TickCount64 1.35 ns 0.069 ~14× — Environment.TickCount64 (per @Tornhoof suggestion)
IsAuthorized_CoarseDateTime 0.32 ns 0.016 ~61× — CoarseDateTime (DateTime variant)
IsAuthorized_CoarseDateTimeTicks 0.12 ns 0.006 ~169× — shipped variant

Environment.TickCount64 is a real win over DateTime.UtcNow (~14×) and dramatically simpler, but on Windows it still reads from KUSER_SHARED_DATA via a method call and is ~10× slower than the L1-cached Volatile.Read of a static long. On Linux the gap compresses (both TickCount64 and DateTime.UtcNow go through vDSO clock_gettime) but CoarseDateTime still wins. For a per-command hot path checked twice per call, CoarseDateTime.UtcNowTicks (0.12 ns ≈ a single mov) is the right choice.

Correctness considerations

  • CoarseTimeProvider granularity is 100ms; AAD token validity windows are measured in hours, so the slop is operationally invisible (smaller than typical inter-host clock skew). A token that just expired may be accepted for up to ~100ms past its true deadline. Under sustained ThreadPool starvation the staleness can exceed 100ms; still harmless for hour-long token windows.
  • GarnetAadAuthenticator is constructed once per RespServerSession (per client connection) and IsAuthorized() / Authenticate() are only ever called on that session''s RESP processing thread, so _validFromTicks / _validToTicks need no cross-thread synchronization.
  • Failed auth in the catch block resets _validFromTicks = long.MaxValue / _validToTicks = long.MinValue so a previously-cached authorized state cannot leak past a failed re-authentication.
  • The process-wide CoarseTimeProvider.System instance is static readonly, so it survives for process lifetime; ~10 callbacks/sec is negligible ThreadPool load. Cached value writes/reads use Volatile.Write / Volatile.Read of a long field — atomic on 64-bit, no JIT reordering.

Testing

  • dotnet build clean.
  • dotnet test test/cluster/Garnet.test.cluster --filter "FullyQualifiedName~Aad":
    • Existing ClusterAadAuthTests (×2) — full server AUTH flow via Authenticate()IsAuthorized() and subsequent commands re-checking via IsAuthenticated. Pass on net8.0 and net10.0.
    • New GarnetAadAuthenticatorTests (×6) — unit-level tests using FakeTimeProvider (Microsoft.Extensions.TimeProvider.Testing) plumbed through CoarseTimeProvider. Covers default fallback to .System, valid window, expiry after Advance(), staying valid across multiple Advance() calls, malformed token, and failed re-auth clearing previously authorized state. Because the authenticator always reads _coarseTime.UtcNowTicks (no test-only branch), advancing the FakeTimeProvider fires the same TimeProvider.CreateTimer refresh callback the production cache uses — tests exercise the production hot path.
  • dotnet format clean on the changed files.

Copilot AI review requested due to automatic review settings June 5, 2026 07:54
@tiagonapoli tiagonapoli marked this pull request as draft June 5, 2026 07:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces per-command overhead in the AAD authentication hot path by avoiding repeated DateTime.UtcNow calls inside GarnetAadAuthenticator.IsAuthorized(). It introduces a process-wide cached UTC tick source and refactors the authenticator to compare precomputed validity ticks against the cached value.

Changes:

  • Added Garnet.common.CachedTime, a static UTC time cache refreshed periodically via System.Threading.Timer.
  • Updated GarnetAadAuthenticator.Authenticate() to precompute _validFromTicks / _validToTicks once per token.
  • Rewrote IsAuthorized() to use a single CachedTime.UtcNowTicks read plus two long comparisons.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
libs/server/Auth/GarnetAadAuthenticator.cs Switch authorization checks from DateTime.UtcNow comparisons to cached-tick comparisons using precomputed token validity window.
libs/common/CachedTime.cs Introduce a process-wide cached UTC tick source to remove wall-clock time queries from hot paths.

Comment thread libs/common/CachedTime.cs Outdated
Comment thread libs/common/CachedTime.cs Outdated
Comment thread libs/common/CachedTime.cs Outdated
Comment thread libs/common/CachedTime.cs Outdated
@tiagonapoli tiagonapoli marked this pull request as ready for review June 5, 2026 08:21
@tiagonapoli tiagonapoli changed the title Cache UTC time in GarnetAadAuthenticator.IsAuthorized to remove syscall from hot path Cache DateTime.UtcNow in AAD authenticator hot path Jun 5, 2026
@tiagonapoli tiagonapoli requested a review from Copilot June 5, 2026 08:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread libs/common/CoarseUtcNow.cs Outdated
Comment thread libs/common/CoarseDateTime.cs Outdated
@Tornhoof

Tornhoof commented Jun 5, 2026

Copy link
Copy Markdown

You could also look at Environment.Tickcount64 for your coarse time. It's likely not as fast as your Timer solution, but should be a lot simpler in code.

@tiagonapoli

Copy link
Copy Markdown
Collaborator Author

I updated the PR description with benchmark for Environment.Tickcount64 - I think would be a viable solution, but on the case of AAD the tokens are valida for many minutes or hours, then we can afford to have a delayed timer with even less impact

Tiago Napoli and others added 15 commits June 8, 2026 11:41
…ll from hot path

IsAuthorized() is checked twice per command (AdminCommands.cs:32 and :131)
via the IsAuthenticated property. Under AAD auth, DateTime.UtcNow showed
~60% exclusive CPU in cluster traces because it issues a clock_gettime
syscall via vDSO on every call.

Introduce Garnet.common.CachedTime, a process-wide UTC time cache refreshed
every 100ms by a background Timer (modeled after Redis's mstime). Reads are
a pure memory load via Volatile.Read of a static long.

GarnetAadAuthenticator now precomputes the token validity window in ticks
once in Authenticate() and compares against CachedTime.UtcNowTicks in
IsAuthorized(). 100ms granularity is operationally invisible against AAD
token windows measured in hours.

Co-authored-by: Copilot <[email protected]>
Redis's server.mstime is refreshed both from serverCron (hz floor, default
100ms) and from beforeSleep (every event loop iteration), so under load it
updates far more often than 100ms. Rephrase as an illustrative example
rather than a precise cadence match.

Co-authored-by: Copilot <[email protected]>
libs/common consistently uses bare camelCase for private fields; align.

Co-authored-by: Copilot <[email protected]>
GarnetAadAuthenticator is per-RespServerSession (single-threaded), so
_validFrom/_validateTo are never accessed cross-thread; there's no
DateTime atomicity concern and DateTime comparison compiles to the same
mask+compare as long comparison. Strip the fields and inline
CachedTime.UtcNow into the original DateTime check.

Co-authored-by: Copilot <[email protected]>
…eUtcNow

Mirrors GarnetAadAuthenticator.IsAuthorized() body verbatim plus the raw
clock primitives for context. Results on i7-12700K / .NET 10:

  IsAuthorized_DateTimeUtcNow : 19.34 ns (baseline)
  IsAuthorized_CoarseUtcNow   :  1.35 ns (0.070x — ~14.4x faster)
  Raw_DateTimeUtcNow          : 18.06 ns
  Raw_CoarseUtcNowValue       :  0.51 ns

Co-authored-by: Copilot <[email protected]>
… in AAD hot path

CoarseDateTime exposes both UtcNow (DateTime) and UtcNowTicks (long) accessors.
GarnetAadAuthenticator now caches token validity as long ticks and compares against
CoarseDateTime.UtcNowTicks, eliminating DateTime's per-call Kind-bit masking on the
hot path. BDN benchmark shows 1.35 ns -> 0.32 ns vs the DateTime variant
(~60x speedup vs DateTime.UtcNow baseline).

Co-authored-by: Copilot <[email protected]>
Tiago Napoli and others added 17 commits June 8, 2026 18:39
…rrides

- Remove the parallel long-ticks Volatile cache (UtcNowTicks). Benchmarks showed the heap-allocated Snapshot<DateTimeOffset> reaches parity with direct ticks under PGO, so the dual cache is unnecessary complexity.

- Drop the four delegating TimeProvider overrides (GetTimestamp, TimestampFrequency, LocalTimeZone, CreateTimer). The base class defaults already match TimeProvider.System, so the overrides are pure noise.

Co-authored-by: Copilot <[email protected]>
The volatile keyword gives reference fields the same acquire/release semantics as Volatile.Read/Write at the call sites, but reads cleaner. The two writes (init + timer tick) and two reads (GetUtcNow + UtcNow) now look like ordinary field access.

Co-authored-by: Copilot <[email protected]>
GarnetAadAuthenticator's ctor already defaults TimeProvider to CoarseTimeProvider.Instance when null, so the settings-level passthrough is unnecessary in production.

Co-authored-by: Copilot <[email protected]>
DateTime.Ticks ignores Kind, so .Ticks on a non-UTC DateTime would silently miscompare against UtcTicks. Construct DateTimeOffset with TimeSpan.Zero so UTC interpretation is explicit at the assignment, and a Local-kind DateTime would throw instead of silently mismatching.

Also trim the verbose XML doc on GarnetAadAuthenticatorTests.

Co-authored-by: Copilot <[email protected]>
Drop the Stopwatch.GetTimestamp variant and a verbose test comment.

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Tiago Napoli and others added 3 commits June 8, 2026 19:36
Singleton via Instance is the only intended use; private ctor prevents accidental allocations.

Co-authored-by: Copilot <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Comment thread libs/server/Auth/GarnetAadAuthenticator.cs
Comment thread libs/common/CoarseTimeProvider.cs
Comment thread test/cluster/Garnet.test.cluster/GarnetAadAuthenticatorTests.cs
Comment thread libs/common/CoarseTimeProvider.cs
Comment thread benchmark/BDN.benchmark/Auth/IsAuthorizedBenchmark.cs
Comment thread benchmark/BDN.benchmark/Auth/IsAuthorizedBenchmark.cs
Comment thread libs/server/Auth/GarnetAadAuthenticator.cs
@tiagonapoli tiagonapoli merged commit 20fec5a into main Jun 9, 2026
145 checks passed
@tiagonapoli tiagonapoli deleted the tiagonapoli/optimize-aad-isauthorized branch June 9, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants