Skip to content

feat: add config.hub_isolation_level for fiber-safe hub storage (Falcon/async)#3018

Merged
sl0thentr0py merged 4 commits into
getsentry:masterfrom
SeanLF:feat/fiber-hub-storage
Jul 14, 2026
Merged

feat: add config.hub_isolation_level for fiber-safe hub storage (Falcon/async)#3018
sl0thentr0py merged 4 commits into
getsentry:masterfrom
SeanLF:feat/fiber-hub-storage

Conversation

@SeanLF

@SeanLF SeanLF commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What

Adds an opt-in config.hub_isolation_level = :thread | :fiber (default :thread) that controls where the SDK stores the current Hub.

  • :thread (default) — thread-local storage, exactly today's behaviour. Nothing changes for anyone who doesn't opt in.
  • :fiber — Ruby 3.2+ Fiber Storage (Fiber[]). Each fiber gets its own hub, and child fibers inherit it, so concurrent requests on a fiber-based server (Falcon, async) are isolated instead of sharing and corrupting one another's scope.

This is the fiber-based option discussed and deferred in #1495.

Why

On a fiber server, concurrent requests run as sibling fibers on one thread, so they share the single thread-local hub. When a request holds a scope across a reactor yield (streaming, IO), a concurrent request's with_scope mutates the same scope stack, and scope/breadcrumbs/user leak between requests.

Reproduced against a real Async::HTTP::Server + the real Sentry::Rack::CaptureExceptions middleware, firing concurrent HTTP requests that each set a user, yield mid-scope, then capture an event:

[:thread] 6 error events | 5 attributed to the WRONG user
[:fiber]  6 error events | 0 attributed to the WRONG user

Under thread-local, 5 of 6 events were reported under the wrong user (cross-user PII bleed). :fiber fixes it.

Why this is safe now, when #1380 moved the other way

#1380 (2021) moved off Ruby's old fiber-local variables (Thread.current[]) because they don't inherit into child fibers, which lost graphql-ruby's context (#1374). Ruby 3.2 added Fiber Storage (Fiber#storage) — a different primitive that copies the parent's storage into a newly created fiber. It gives per-request isolation and child-fiber inheritance at once, so it fixes fiber servers without re-breaking #1374. The added regression tests prove both hold.

This is also how the other SDKs already work (sentry-python contextvars, sentry-javascript AsyncLocalStorage — both isolate per task and inherit into children), and it follows opentelemetry-ruby's 2025 move off thread-local storage (#1807).

Design

  • New Sentry::HubStorage owns get/set/clear of the current hub and switches on the isolation level. THREAD_LOCAL = :sentry_hub is kept as the storage key (user monkey-patches reference the constant).
  • config.isolation_level is applied at Sentry.init. Requesting :fiber on Ruby < 3.2 logs a warning and falls back to :thread (feature-detected, so nothing breaks on 2.7–3.1).
  • Every former thread_variable_* site now routes through HubStorage (core sentry-ruby.rb, plus sentry-rails active_job.rb and core test_helper.rb, which touched THREAD_LOCAL directly and would be wrong under :fiber).

Mirrors Rails' config.active_support.isolation_level ergonomics; uses Fiber[] (not Rails' non-inheriting Fiber.attr_accessor) because the SDK needs the inheritance property.

Sentry.init do |config|
  config.dsn = "..."
  config.hub_isolation_level = :fiber # opt in on Falcon / async
end

Compatibility

  • Default :thread is byte-for-byte the previous behaviour. Puma/Sidekiq/Unicorn/Resque are unaffected.
  • Ruby >= 2.7 still supported; :fiber requires 3.2+ and degrades to :thread with a warning below that.
  • :fiber is safe on threaded servers too (each request re-clones the hub); the difference only matters when child fibers are spawned per request.

Caveat (documented)

Under :fiber, a raw Thread.new spawned mid-request inherits the request's hub object (documented Ruby fiber-storage thread inheritance). Call Sentry.clone_hub_to_current_thread in the spawned thread to isolate it — same as today.

Tests

Performance

benchmark-ips (Ruby 4.0.5): Fiber[] get is 1.24x faster than thread_variable_get; at the level of a full Sentry.add_breadcrumb the two isolation levels are within measurement error. Not a hot-path regression.

Notes for reviewers

  • Two commits: core feature, then a 2-line sentry-rails follow-on so ActiveJob is correct under :fiber. Happy to split the rails commit into a separate PR if you'd prefer to keep this core-only.
  • No CHANGELOG.md edit (per the repo's auto-generated-at-release convention). Suggested entry:
    • Add config.isolation_level = :thread | :fiber for fiber-safe hub storage on fiber-based servers (Falcon/async), fixing cross-request scope contamination. Default :thread keeps existing behaviour.

Refs #1495. /cc @sl0thentr0py @solnic

Changelog Entry

Add Fiber support new config.hub_isolation_level which can be set to :fiber or :thread. Use this in case you use Fiber based servers like Falcon

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5f89c7f. Configure here.

Comment thread sentry-ruby/lib/sentry/configuration.rb
Comment thread sentry-ruby/spec/sentry/configuration_spec.rb
@SeanLF

SeanLF commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Both findings addressed in 3ce928c:

  1. Post-init isolation sync (medium): Configuration#isolation_level= now applies the level to HubStorage when the SDK is already initialized, so a post-init change to Sentry.configuration.isolation_level actually takes effect instead of leaving the SDK on the old backend. The initialize default is assigned to the ivar directly so a throwaway Configuration.new can't clobber the active isolation level (only Sentry.init builds the live config).

  2. Fiber specs vs Ruby < 3.2 fallback (high): the :fiber-asserting specs now stub fiber_storage_available? so they're deterministic across the Ruby matrix, plus an explicit downgrade-to-:thread spec. They previously asserted :fiber unconditionally and would have failed on the < 3.2 cells, where :fiber correctly falls back to :thread. Added a regression test for the post-init sync too.

@sl0thentr0py

Copy link
Copy Markdown
Member

thanks a lot @SeanLF, if you don't mind, I will reduce the code a bit and make it more minimal.

SeanLF and others added 3 commits July 13, 2026 14:14
The current hub is stored in thread-local storage. On a fiber-based
server (Falcon/async) many concurrent requests run as sibling fibers on
one thread, so they share a single hub. When a request holds a scope
across a reactor yield (streaming, IO), a concurrent request's with_scope
mutates the same scope stack and scope/transaction/breadcrumbs/user leak
between requests -- reproduced as 5/6 events attributed to the wrong user,
including their email (cross-user PII bleed).

Add an opt-in config.isolation_level (:thread default, :fiber) that routes
the current hub through Sentry::HubStorage. :fiber uses Ruby 3.2+ Fiber
Storage (Fiber[]), which gives per-request isolation AND child-fiber
inheritance, so it fixes Falcon without re-introducing the graphql-ruby
context loss that getsentry#1374/getsentry#1380 addressed (old fiber-local vars did not
inherit; Fiber Storage does). Requesting :fiber on Ruby < 3.2 warns and
falls back to :thread. Default :thread is byte-for-byte the previous
behaviour. Mirrors Rails' config.active_support.isolation_level.

Refs getsentry#1495.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The ActiveJob integration saved and restored the surrounding hub with
Thread.current.thread_variable_get/set(Sentry::THREAD_LOCAL) directly,
which is wrong under config.isolation_level = :fiber (it would read/write
thread storage while the active hub lives in fiber storage). Route both
sites through Sentry::HubStorage so the save/restore follows the
configured isolation level. Behaviour is byte-for-byte unchanged under
the default :thread level.

Verified with a real ActiveJob perform_now: under :thread the outer hub
is restored and the job's user does not leak; under :fiber, concurrent
jobs run as sibling fibers each keep their own user.

Refs getsentry#1495.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…pecs

Two follow-ups from review:

- Configuration#isolation_level= now applies the level to HubStorage when
  the SDK is already initialized, so changing Sentry.configuration.isolation_level
  after Sentry.init takes effect instead of leaving the SDK on the old backend.
  The default is assigned to the ivar directly in initialize so a throwaway
  Configuration.new cannot clobber the active isolation level (only Sentry.init
  builds the live config).

- Make the :fiber isolation_level specs deterministic across Ruby versions by
  stubbing fiber_storage_available?, and add an explicit downgrade-to-:thread
  spec. Previously these asserted :fiber unconditionally and would fail on the
  Ruby < 3.2 CI cells, where :fiber correctly falls back to :thread.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@sl0thentr0py
sl0thentr0py force-pushed the feat/fiber-hub-storage branch from 3ce928c to ab54490 Compare July 13, 2026 12:14
@sl0thentr0py sl0thentr0py changed the title feat: add config.isolation_level for fiber-safe hub storage (Falcon/async) feat: add config.hub_isolation_level for fiber-safe hub storage (Falcon/async) Jul 13, 2026
@sl0thentr0py
sl0thentr0py force-pushed the feat/fiber-hub-storage branch from 7c2e7f7 to de42598 Compare July 13, 2026 14:56

@sl0thentr0py sl0thentr0py left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • renamed config to hub_isolation_level for more clarity
  • moved logic all inline to Configuration instead of HubStorage

@solnic solnic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this 🙏🏻 Left some nits for your consideration 😄

if level == :fiber && !fiber_storage_available?
log_warn("hub_isolation_level :fiber requires Ruby 3.2+ Fiber Storage; falling back to :thread on Ruby #{RUBY_VERSION}.")
level = :thread
end

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could completely avoid that and define writer method dynamically depending on the availability, which would simplify runtime production code path.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we still want to have the warning message if they try to set it, so this needs to stay

# CaptureExceptions, or a stale hub left by a recycled thread-pool
# thread) so the outer context continues working correctly after
# the job finishes.
original_hub = Thread.current.thread_variable_get(Sentry::THREAD_LOCAL)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this comment inaccurate and that's why it got removed?

@sl0thentr0py sl0thentr0py Jul 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should generally not have this clanker generated comment bloat, so I will remove it wherever I see it unless it really adds any value.

end

def hub_isolation_level=(level)
level = level.to_sym if level.respond_to?(:to_sym)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for allowing non-symbols over here?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just syntax sugar

attr_reader :hub_isolation_level

# Isolation levels the SDK understands for hub storage.
ISOLATION_LEVELS = %i[thread fiber].freeze

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit and could be a follow-up mini-refactor, but: we have a validate macro - we could extend it to support included_in: [...] and then overriding the writer would only be needed for the situation where the Fiber storage API is not available.

@sl0thentr0py
sl0thentr0py merged commit 7ab412f into getsentry:master Jul 14, 2026
295 of 298 checks passed
sl0thentr0py added a commit to getsentry/sentry-docs that referenced this pull request Jul 14, 2026
@trevorturk

Copy link
Copy Markdown

Hello! I ran into this PR because I was having an issue in an app I'm running with Falcon that was leaking Sentry tags across requests where it should have been impossible. I ran this all through Claude Fable, so please accept my apologies, I'll try to keep the AI slop to a minimum, but I think we might have room to improve. I think the tl;dr is a bit strange because of my currently-mixed workload where I'm running Falcon for web, but SolidQueue for jobs, and SolidQueue doesn't ship with an Async/Fiber mode yet (but hopefully will soon: rails/solid_queue#728)

So... Claude slop follows, edited a bit to keep it short, I hope this is helpful, but please accept my apologies for not entirely working through this myself in detail -- this should be taken with a grain of salt, but I thought it worth sharing:


A caveat worth considering before this ships: Fiber storage is inherited by Thread.new, sharing one hub across threads.

While evaluating the :fiber approach (we prototyped equivalent Fiber-storage hub accessors on 6.6.2 while waiting for the release), we hit a caveat worth raising before hub_isolation_level ships: Ruby's fiber storage is inherited by Thread.new, not just Fiber.new, and the inherited values are the same objects — so a hub stored via Fiber[] in one fiber is shared (not copied) into any thread that fiber spawns.

Why this bites in practice

In a typical Rails app, hubs get seeded early and implicitly — e.g. Sentry::Rails::Breadcrumb::ActiveSupportLogger calls Sentry.add_breadcrumb → get_current_hub on every ActiveSupport notification, so the first SQL query in a process seeds the current fiber's hub. Any thread pool created after that point (a job processor's workers, an app's own worker threads) inherits that one hub into every thread. In :fiber mode:

  • Concurrent jobs in a threaded processor (we reproduced the shape with a Solid Queue-style pool) all read and write one shared scope: 4 workers → 1 hub, each seeing another job's tags from inside its own Sentry.with_scope.
  • The shared Hub#@stack Array is then pushed/popped from multiple OS threads without synchronization.

So a mixed web+jobs app that enables :fiber globally (the natural reading of a config option) trades cross-request contamination on the web side for cross-job contamination on the worker side.

Repro sketch (Ruby 3.4, semantics-level)

Fiber[:hub] = Object.new
threads = Array.new(4) { Thread.new { Fiber[:hub] } }
threads.map(&:value).map(&:object_id).uniq.size # => 1 — all threads share the fiber's object

With hub accessors reading Fiber[KEY] || clone, the || clone fallback never fires in those threads — they inherit a non-nil hub — so the per-context isolation the fallback was designed to provide is silently bypassed.

Possible mitigation

Tagging the stored hub with its owning thread restores per-thread cloning at thread boundaries while keeping fiber inheritance within a thread:

def get_current_hub
  thread, hub = Fiber[KEY]
  thread.equal?(Thread.current) ? hub : clone_hub_to_current_thread
end

def clone_hub_to_current_thread
  hub = get_main_hub.clone
  Fiber[KEY] = [Thread.current, hub]
  hub
end

@SeanLF

SeanLF commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

@trevorturk thanks for digging into this! My own claude managed to reproduce it on master

expected actual
4 worker threads 4 hubs 1 hub
4 concurrent with_scope jobs own tags each read another job's tags
3 sibling request fibers 3 hubs 1 hub

Fiber.new inherits storage the same way Thread.new does, so :fiber never fixed the Falcon case it was built for either. The shared @stack is also pushed and popped from several threads with no synchronization.

The existing regression spec missed it because it calls Sentry.clone_hub_to_current_thread inside each fiber, which overwrites the inherited entry. Real request fibers never call that.

Two notes on your mitigation. Tagging with Thread.current fixes the worker-thread case but leaves sibling fibers sharing one hub, since they run on the same thread; tagging with the owning fiber covers both. And clone_hub_to_current_thread returns whatever the setter returns, so Fiber[k] = [thread, hub] hands callers the Array.

Cloning the hub is also not enough on its own. Hub#clone deep-dups the scope's span, which builds a fresh SpanRecorder, so child-fiber spans land on a detached transaction copy and are never sent. The fork has to keep the live span.

PR incoming. This is unreleased, so it can land before the next release

@trevorturk

Copy link
Copy Markdown

Excellent! Thank you for digging into that. Please link the PR here so we can keep track and I'll QA IRL when everything is settled.

@SeanLF

SeanLF commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

PR is up: #3036

It tags each stored entry with its owning fiber, so a context that inherited one takes its own copy. The copy keeps the live span, otherwise child spans land on a detached transaction copy and are never sent.

One thing that turned up while testing across the matrix, relevant to your Solid Queue setup: the fiber-storage gem polyfills Fiber[] on Ruby < 3.2 and on JRuby, so :fiber is live there too. That polyfill inherits into child fibers but not into Thread.new, so on those Rubies only the sibling-fiber half of the bug exists; worker threads already start from the main hub. On 3.2+ with native fiber storage you get both halves, which matches what you saw.

Happy to have it QA'd IRL once it lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants