FFE docs: add in-memory provider testing guidance per language#36198
Merged
leoromanovsky merged 35 commits intoApr 30, 2026
Conversation
Introduces a 'Testing with in-memory providers' section on both the server-side and client-side feature flag landing pages. Explains that the Datadog provider talks to Datadog's backend and is not appropriate for unit tests, and directs readers to per-language pages for concrete examples.
Teaches customers to swap DatadogProvider for OpenFeature's memprovider in unit tests. Uses named clients via SetNamedProviderAndWait so t.Parallel() is safe, and shows the ContextEvaluator pointer-to-func pattern.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider (Flag builder form) in JUnit 5 tests. Emphasizes OpenFeatureAPI singleton reset via shutdown() in @AfterEach and mentions Spring Boot test setup.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider in pytest. Uses a function-scoped fixture with api.shutdown() teardown to prevent global API singleton leakage between tests.
Teaches customers to swap DatadogProvider for OpenFeature's TypedInMemoryProvider in Vitest tests. Shows OpenFeature.close() teardown and per-test putConfiguration() to keep tests isolated from each other.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider in xUnit via IAsyncLifetime. Notes cross-framework applicability (NUnit, MSTest) and the WebApplicationFactory integration path for ASP.NET Core.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider in RSpec via an around hook that captures and restores the prior provider. Notes the Ruby SDK's plain-value flag hash (no variants) and the add_flag mutation API.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider from @openfeature/web-sdk in Vitest. Calls out the required flag shape (variants, defaultVariant, disabled) and the sync evaluation model.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider in tests. Keeps the section minimal per the client-testing style: short paragraph plus a 10-line snippet, no RTL or component harness.
Kotlin/Android OpenFeature SDK does not ship an InMemoryProvider, so the section includes a ~30-line FakeProvider stub and walks through setting it up under runTest from kotlinx-coroutines-test.
OpenFeature Swift SDK ships no InMemoryProvider and has no contrib package, so the section includes a ~50-line InMemoryTestProvider FeatureProvider stub. Uses async setProviderAndWait and clearProviderAndWait for teardown.
Contributor
Corrections from per-language expert review against current upstream SDKs: - Go: ContextEvaluator is already *func(...); drop the extra cast so the snippet compiles (the prior form produced a **func(...) conversion error). - Python: context_evaluator returns a FlagResolutionDetails, not a bare variant name. Fix prose wording. - .NET: InMemoryProvider's mutation method is UpdateFlagsAsync, not UpdateFlags. Fix prose wording. - JavaScript (client) and React: swap the deprecated InMemoryProvider for TypedInMemoryProvider (upstream marks the former @deprecated) and mention OpenFeatureTestProvider as the idiomatic React component-test wrapper.
…e-provider-swift Points readers at the Datadog OpenFeature bridge (dd-openfeature-provider-swift) as the production path for OpenFeature on iOS. Tests swap the bridge's DatadogProvider for a small custom FeatureProvider because the upstream OpenFeature Swift SDK does not ship an in-memory provider.
Verification caught six compile errors in the InMemoryTestProvider stub:
- initialize/onContextSet are 'async throws', not Future<Void, Never>
- observe() returns AnyPublisher<ProviderEvent?, Never> (optional event)
- ProviderEvent.ready has no associated value; use .ready not .ready(nil)
- Reason case is .staticReason, not .static ('static' is reserved)
- OpenFeatureAPI.shared.clearProvider() is sync and does not exist as
clearProviderAndWait() in 0.3.0
Also update the prose callout to match.
…g's CDN Client-side SDKs reach Datadog's CDN to retrieve flag assignments, not the Remote Configuration backend used by server-side SDKs. Updates the testing-section callouts on the shared client _index and on each per-platform client page so the language is accurate.
…tion In-memory providers are one testing approach; running the real Datadog provider against a dedicated test environment is equally valid. Updates the shared _index sections and each per-language page so they present both options up front, then proceed with the in-memory example as 'this section covers...' rather than 'do not use the Datadog provider'.
Adds a new 'Use via OpenFeature' section covering install of dd-openfeature-provider-swift, OpenFeatureAPI setup, evaluation context, typed flag evaluation, and flag details. Existing FlagsClient-based sections are unchanged. Includes a 'not yet production-ready' callout since the bridge package is still in development.
leoromanovsky
marked this pull request as ready for review
April 22, 2026 16:11
leoromanovsky
requested review from
a team,
sameerank and
typotter
and removed request for
a team
April 22, 2026 16:12
Contributor
|
Created DOCS-14128 for review. |
sameerank
approved these changes
Apr 22, 2026
Comment on lines
+316
to
+321
| if err := openfeature.SetNamedProviderAndWait(name, provider); err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| t.Cleanup(func() { | ||
| _ = openfeature.SetNamedProviderAndWait(name, openfeature.NoopProvider{}) | ||
| }) |
Contributor
There was a problem hiding this comment.
Good idea to include guidance for named providers in testing
typotter
approved these changes
Apr 22, 2026
| Datadog supports two testing approaches: | ||
|
|
||
| - **Integration tests**: Point `DatadogProvider` at a dedicated test environment and control flag values from the Datadog UI. This exercises the real provider end-to-end, including the CDN-delivered flag assignments. | ||
| - **Unit tests**: Swap `DatadogProvider` for OpenFeature's standard `InMemoryProvider` (or an equivalent test stub, where no in-memory provider is available in the language) and set flag values directly in test code. This keeps tests hermetic, fast, and offline. |
Co-authored-by: Tyler Potter <[email protected]>
Address review from @typotter: - Remove events.emit(ProviderReady) from FakeProvider.initialize; the SDK emits ProviderReady after initialize returns. - Soften the InMemoryProvider wording and link to open-feature/kotlin-sdk#226, where upstream in-memory provider support is tracked.
Apply @typotter's step 3 rewrite from client/_index.md to server/_index.md for consistency: convey that units under test get an injected OpenFeature client backed by the InMemoryProvider.
joepeeples
reviewed
Apr 29, 2026
joepeeples
left a comment
Contributor
There was a problem hiding this comment.
Thanks for preparing this content, @leoromanovsky! Added some style suggestions, mostly line edits. Let me know when ready for a re-review.
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
joepeeples
approved these changes
Apr 29, 2026
joepeeples
approved these changes
Apr 29, 2026
…docs-testing-in-memory # Conflicts: # content/en/feature_flags/client/android.md
estherk15
pushed a commit
that referenced
this pull request
May 4, 2026
* feature_flags: add shared testing section to server and client _index Introduces a 'Testing with in-memory providers' section on both the server-side and client-side feature flag landing pages. Explains that the Datadog provider talks to Datadog's backend and is not appropriate for unit tests, and directs readers to per-language pages for concrete examples. * feature_flags: add Testing section to Go server-side page Teaches customers to swap DatadogProvider for OpenFeature's memprovider in unit tests. Uses named clients via SetNamedProviderAndWait so t.Parallel() is safe, and shows the ContextEvaluator pointer-to-func pattern. * feature_flags: add Testing section to Java server-side page Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider (Flag builder form) in JUnit 5 tests. Emphasizes OpenFeatureAPI singleton reset via shutdown() in @AfterEach and mentions Spring Boot test setup. * feature_flags: add Testing section to Python server-side page Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider in pytest. Uses a function-scoped fixture with api.shutdown() teardown to prevent global API singleton leakage between tests. * feature_flags: add Testing section to Node.js server-side page Teaches customers to swap DatadogProvider for OpenFeature's TypedInMemoryProvider in Vitest tests. Shows OpenFeature.close() teardown and per-test putConfiguration() to keep tests isolated from each other. * feature_flags: add Testing section to .NET server-side page Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider in xUnit via IAsyncLifetime. Notes cross-framework applicability (NUnit, MSTest) and the WebApplicationFactory integration path for ASP.NET Core. * feature_flags: add Testing section to Ruby server-side page Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider in RSpec via an around hook that captures and restores the prior provider. Notes the Ruby SDK's plain-value flag hash (no variants) and the add_flag mutation API. * feature_flags: add Testing section to JavaScript client-side page Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider from @openfeature/web-sdk in Vitest. Calls out the required flag shape (variants, defaultVariant, disabled) and the sync evaluation model. * feature_flags: add Testing section to React client-side page Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider in tests. Keeps the section minimal per the client-testing style: short paragraph plus a 10-line snippet, no RTL or component harness. * feature_flags: add Testing section to Android client-side page Kotlin/Android OpenFeature SDK does not ship an InMemoryProvider, so the section includes a ~30-line FakeProvider stub and walks through setting it up under runTest from kotlinx-coroutines-test. * feature_flags: add Testing section to iOS client-side page OpenFeature Swift SDK ships no InMemoryProvider and has no contrib package, so the section includes a ~50-line InMemoryTestProvider FeatureProvider stub. Uses async setProviderAndWait and clearProviderAndWait for teardown. * feature_flags: apply upstream OpenFeature SDK verification fixes Corrections from per-language expert review against current upstream SDKs: - Go: ContextEvaluator is already *func(...); drop the extra cast so the snippet compiles (the prior form produced a **func(...) conversion error). - Python: context_evaluator returns a FlagResolutionDetails, not a bare variant name. Fix prose wording. - .NET: InMemoryProvider's mutation method is UpdateFlagsAsync, not UpdateFlags. Fix prose wording. - JavaScript (client) and React: swap the deprecated InMemoryProvider for TypedInMemoryProvider (upstream marks the former @deprecated) and mention OpenFeatureTestProvider as the idiomatic React component-test wrapper. * feature_flags: frame iOS testing around OpenFeature and dd-openfeature-provider-swift Points readers at the Datadog OpenFeature bridge (dd-openfeature-provider-swift) as the production path for OpenFeature on iOS. Tests swap the bridge's DatadogProvider for a small custom FeatureProvider because the upstream OpenFeature Swift SDK does not ship an in-memory provider. * feature_flags: fix iOS test snippet against OpenFeature Swift SDK 0.3.0 Verification caught six compile errors in the InMemoryTestProvider stub: - initialize/onContextSet are 'async throws', not Future<Void, Never> - observe() returns AnyPublisher<ProviderEvent?, Never> (optional event) - ProviderEvent.ready has no associated value; use .ready not .ready(nil) - Reason case is .staticReason, not .static ('static' is reserved) - OpenFeatureAPI.shared.clearProvider() is sync and does not exist as clearProviderAndWait() in 0.3.0 Also update the prose callout to match. * feature_flags: clarify client-side provider fetches flags from Datadog's CDN Client-side SDKs reach Datadog's CDN to retrieve flag assignments, not the Remote Configuration backend used by server-side SDKs. Updates the testing-section callouts on the shared client _index and on each per-platform client page so the language is accurate. * feature_flags: soften testing guidance to present in-memory as one option In-memory providers are one testing approach; running the real Datadog provider against a dedicated test environment is equally valid. Updates the shared _index sections and each per-language page so they present both options up front, then proceed with the in-memory example as 'this section covers...' rather than 'do not use the Datadog provider'. * feature_flags: add OpenFeature usage section to iOS page Adds a new 'Use via OpenFeature' section covering install of dd-openfeature-provider-swift, OpenFeatureAPI setup, evaluation context, typed flag evaluation, and flag details. Existing FlagsClient-based sections are unchanged. Includes a 'not yet production-ready' callout since the bridge package is still in development. * feature_flags: note getIntegerValue returns Int64 in iOS OpenFeature section * Update content/en/feature_flags/client/_index.md Co-authored-by: Tyler Potter <[email protected]> * feature_flags/android: drop ProviderReady emit, note kotlin-sdk#226 Address review from @typotter: - Remove events.emit(ProviderReady) from FakeProvider.initialize; the SDK emits ProviderReady after initialize returns. - Soften the InMemoryProvider wording and link to open-feature/kotlin-sdk#226, where upstream in-memory provider support is tracked. * feature_flags/server: mirror client step 3 rewording Apply @typotter's step 3 rewrite from client/_index.md to server/_index.md for consistency: convey that units under test get an injected OpenFeature client backed by the InMemoryProvider. * 1 picker only Co-authored-by: Joe Peeples <[email protected]> * remove dup from server Co-authored-by: Joe Peeples <[email protected]> * split sentence Co-authored-by: Joe Peeples <[email protected]> * action verb Co-authored-by: Joe Peeples <[email protected]> * with of Co-authored-by: Joe Peeples <[email protected]> * action Co-authored-by: Joe Peeples <[email protected]> * comma dotnet Co-authored-by: Joe Peeples <[email protected]> * comma nodejs Co-authored-by: Joe Peeples <[email protected]> * fast no promises Co-authored-by: Joe Peeples <[email protected]> * fast server no promises Co-authored-by: Joe Peeples <[email protected]> * ios bridge Co-authored-by: Joe Peeples <[email protected]> * these Co-authored-by: Joe Peeples <[email protected]> * Apply suggestions from code review Co-authored-by: Joe Peeples <[email protected]> --------- Co-authored-by: Tyler Potter <[email protected]> Co-authored-by: Joe Peeples <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Customers integrating Datadog Feature Flags via the OpenFeature provider have no documented path for writing unit tests. The Datadog provider requires a running Agent plus Remote Configuration on the server side, or network access to Datadog's CDN on the client side; either way, readers hitting the feature-flags docs today have to infer the testing story on their own. That gap surfaces on every technical evaluation and slows adoption.
This PR adds a shared "Testing with in-memory providers" section on the server-side and client-side landing pages, plus a concrete
## Testingsection on every language page. Each language example uses OpenFeature's standardInMemoryProvider(or a small hand-rolled stub where no in-memory provider exists), preserving the swappable-provider guarantee of the OpenFeature spec.The PR also adds a full Use via OpenFeature section to the iOS page, since the rest of that page only documents the
FlagsClientAPI. This section is additive — existingFlagsClientcontent is unchanged.Changes
feature_flags/server/_index.md,feature_flags/client/_index.md— two-approach explainer (test environment vs in-memory), 4-step pattern, per-language link out. The client_indexclarifies the client-side provider fetches flags from Datadog's CDN.go.md,java.md,python.md,nodejs.md,dotnet.md,ruby.md— concrete test example per language with the idiomatic runner.javascript.md,react.md,android.md,ios.md— same, with Android and iOS shipping a copy-pasteableFeatureProviderstub because neither SDK ecosystem ships an in-memory provider.dd-openfeature-provider-swiftinstall,DatadogProvider+OpenFeatureAPI.sharedsetup, evaluation context, typed evaluation, and flag-evaluation details. Includes a "currently in development" callout since the bridge is pre-1.0.One commit per page so reviewers can read the diffs in isolation. Follow-up commits apply (a) fixes from the verification pass, (b) a scope-softening pass so in-memory is presented as one option alongside exercising the real provider against a test environment, and (c) a CDN terminology pass on client pages.
Verification process
Every code snippet in this PR was certified by a parallel pass of Claude Opus 4.7 subagents, each prompted to act as a staff engineer specialized in one target language and to verify the snippet against the current upstream OpenFeature SDK. This is automated verification, not human review. Reviewers on this PR should still sanity-check the agent findings before merge.
Each subagent:
## Testingblock (and, for iOS, the new## Use via OpenFeatureblock) from the PR.~/dd/openfeature-*-sdk/).Current status: all snippets pass
No open issues remain. Every agent now reports the current snippets as compiling and behaving correctly against the corresponding upstream SDK. Two snippets failed on first pass and were fixed in follow-up commits; several agents flagged small prose inaccuracies that were also patched.
ContextEvaluator(the SDK type is already*func(...), so the original expression double-wrapped it)context_evaluatordescription: it returns aFlagResolutionDetails, not a bare variant nameTypedInMemoryProvideris the current non-deprecated export)UpdateFlagsAsync, notUpdateFlagsInMemoryProviderforTypedInMemoryProvider(live Vitest run passed before and after)OpenFeatureTestProviderfor component testsinitialize/onContextSetareasync throwsnotFuture<Void, Never>;observe()returnsAnyPublisher<ProviderEvent?, Never>;.readytakes no associated value;Reasoncase is.staticReason;clearProvider()(sync) replaces the nonexistentclearProviderAndWait()getIntegerValuereturnsInt64for reader clarityDecisions
testing, Java JUnit 5, Python pytest, Node Vitest, .NET xUnit, Ruby RSpec, JS Vitest, Android JUnit 4 +kotlinx-coroutines-test, iOS XCTest with async methods.TypedInMemoryProvider— the non-genericInMemoryProviderexport from@openfeature/server-sdkis deprecated upstream.SetNamedProviderAndWait+NewClient(name)rather than the default global provider sot.Parallel()is safe and flag state cannot leak between tests.aroundhook (notbefore(:each)) to capture and restore the prior provider on the global singleton.FeatureProviderstubs inline (~30–50 LOC). The upstream Kotlin SDK, the Swift SDK, and any contrib repo for either language do not expose an in-memory provider. An inline stub is cleaner than asking readers to pull a third-party dependency that does not exist.Use via OpenFeaturemain-body section since the rest of the iOS page documents only theFlagsClientAPI. The new section is additive — readers followingFlagsClientare unaffected. Readers who prefer the standard API get a complete walkthrough ofdd-openfeature-provider-swift+OpenFeatureAPI.shared, including setup, context, typed evaluation, and flag-evaluation details. A callout notes the bridge is pre-1.0 and currently in development.com.datadoghq:dd-sdk-android-flags-openfeature; the Testing section uses the sameOpenFeatureAPIthe production path uses. The separatedd-openfeature-provider-kotlinrepo is empty — the Android provider lives in thedd-sdk-androidmonorepo.InMemoryProviderin the PHP SDK today); Unity has no OpenFeature SDK at all.