Skip to content

FFE docs: add in-memory provider testing guidance per language#36198

Merged
leoromanovsky merged 35 commits into
masterfrom
leo.romanovsky/ffe-docs-testing-in-memory
Apr 30, 2026
Merged

FFE docs: add in-memory provider testing guidance per language#36198
leoromanovsky merged 35 commits into
masterfrom
leo.romanovsky/ffe-docs-testing-in-memory

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

Motivation

Customers integrating Datadog Feature Flags via the OpenFeature provider have no documented path for writing unit tests. The Datadog provider requires a running Agent plus Remote Configuration on the server side, or network access to Datadog's CDN on the client side; either way, readers hitting the feature-flags docs today have to infer the testing story on their own. That gap surfaces on every technical evaluation and slows adoption.

This PR adds a shared "Testing with in-memory providers" section on the server-side and client-side landing pages, plus a concrete ## Testing section on every language page. Each language example uses OpenFeature's standard InMemoryProvider (or a small hand-rolled stub where no in-memory provider exists), preserving the swappable-provider guarantee of the OpenFeature spec.

The PR also adds a full Use via OpenFeature section to the iOS page, since the rest of that page only documents the FlagsClient API. This section is additive — existing FlagsClient content is unchanged.

Changes

  • Shared: feature_flags/server/_index.md, feature_flags/client/_index.md — two-approach explainer (test environment vs in-memory), 4-step pattern, per-language link out. The client _index clarifies the client-side provider fetches flags from Datadog's CDN.
  • Server (6): go.md, java.md, python.md, nodejs.md, dotnet.md, ruby.md — concrete test example per language with the idiomatic runner.
  • Client (4): javascript.md, react.md, android.md, ios.md — same, with Android and iOS shipping a copy-pasteable FeatureProvider stub because neither SDK ecosystem ships an in-memory provider.
  • iOS Use via OpenFeature: new main-body section covering dd-openfeature-provider-swift install, DatadogProvider + OpenFeatureAPI.shared setup, evaluation context, typed evaluation, and flag-evaluation details. Includes a "currently in development" callout since the bridge is pre-1.0.

One commit per page so reviewers can read the diffs in isolation. Follow-up commits apply (a) fixes from the verification pass, (b) a scope-softening pass so in-memory is presented as one option alongside exercising the real provider against a test environment, and (c) a CDN terminology pass on client pages.

Verification process

Every code snippet in this PR was certified by a parallel pass of Claude Opus 4.7 subagents, each prompted to act as a staff engineer specialized in one target language and to verify the snippet against the current upstream OpenFeature SDK. This is automated verification, not human review. Reviewers on this PR should still sanity-check the agent findings before merge.

Each subagent:

  1. Read the proposed ## Testing block (and, for iOS, the new ## Use via OpenFeature block) from the PR.
  2. Pulled the current upstream OpenFeature SDK for its language (from GitHub, language-native package manager, or a local checkout under ~/dd/openfeature-*-sdk/).
  3. Verified every API call against the SDK source — imports, method signatures, constructor arguments, async/suspend/throws annotations, protocol/interface conformance.
  4. Where possible, executed the snippet (or a minimal equivalent) against the installed SDK to confirm runtime behavior (Node.js and JS-browser ran live Vitest suites; Ruby and Java were verified against installed gems / local Maven source).
  5. Returned a PASS or FAIL verdict with before/after diffs for any issues found.

Current status: all snippets pass

No open issues remain. Every agent now reports the current snippets as compiling and behaving correctly against the corresponding upstream SDK. Two snippets failed on first pass and were fixed in follow-up commits; several agents flagged small prose inaccuracies that were also patched.

Language / section First pass Correction applied Current status
Go FAIL Removed the extra cast around ContextEvaluator (the SDK type is already *func(...), so the original expression double-wrapped it) PASS
Java PASS PASS
Python PASS (prose nit) Reworded the context_evaluator description: it returns a FlagResolutionDetails, not a bare variant name PASS
Node.js PASS — (live Vitest run passed; TypedInMemoryProvider is the current non-deprecated export) PASS
.NET PASS (prose nit) Corrected the mutation method name: UpdateFlagsAsync, not UpdateFlags PASS
Ruby PASS PASS
JavaScript (client) PASS Swapped the deprecated InMemoryProvider for TypedInMemoryProvider (live Vitest run passed before and after) PASS
React PASS Same deprecation swap; added a pointer to OpenFeatureTestProvider for component tests PASS
iOS (Testing) FAIL Six compile fixes against OpenFeature Swift SDK 0.3.0: initialize/onContextSet are async throws not Future<Void, Never>; observe() returns AnyPublisher<ProviderEvent?, Never>; .ready takes no associated value; Reason case is .staticReason; clearProvider() (sync) replaces the nonexistent clearProviderAndWait() PASS
iOS (Use via OpenFeature) PASS Added a one-liner noting getIntegerValue returns Int64 for reader clarity PASS
Android PASS PASS

Decisions

  • Testing is presented as two approaches, not a single directive. Teams can exercise the real Datadog provider against a dedicated test environment (where flag values are controlled from the Datadog UI) or swap in OpenFeature's in-memory provider (where flag values are controlled from test code). The in-memory path, covered in this PR, is the hermetic-and-offline option.
  • Per-language runner defaults: Go testing, Java JUnit 5, Python pytest, Node Vitest, .NET xUnit, Ruby RSpec, JS Vitest, Android JUnit 4 + kotlinx-coroutines-test, iOS XCTest with async methods.
  • Node.js uses TypedInMemoryProvider — the non-generic InMemoryProvider export from @openfeature/server-sdk is deprecated upstream.
  • Go teaches SetNamedProviderAndWait + NewClient(name) rather than the default global provider so t.Parallel() is safe and flag state cannot leak between tests.
  • Ruby uses an around hook (not before(:each)) to capture and restore the prior provider on the global singleton.
  • Java skips build-system tabs inside the Testing section — tabs already exist in the Installation section above, and re-tabbing every subsequent section turns the page into a maze.
  • Android and iOS ship custom FeatureProvider stubs inline (~30–50 LOC). The upstream Kotlin SDK, the Swift SDK, and any contrib repo for either language do not expose an in-memory provider. An inline stub is cleaner than asking readers to pull a third-party dependency that does not exist.
  • iOS gets a full Use via OpenFeature main-body section since the rest of the iOS page documents only the FlagsClient API. The new section is additive — readers following FlagsClient are unaffected. Readers who prefer the standard API get a complete walkthrough of dd-openfeature-provider-swift + OpenFeatureAPI.shared, including setup, context, typed evaluation, and flag-evaluation details. A callout notes the bridge is pre-1.0 and currently in development.
  • Android already documents OpenFeature as the recommended API via com.datadoghq:dd-sdk-android-flags-openfeature; the Testing section uses the same OpenFeatureAPI the production path uses. The separate dd-openfeature-provider-kotlin repo is empty — the Android provider lives in the dd-sdk-android monorepo.
  • PHP and Unity are out of scope for this PR. PHP's OpenFeature provider is still in development (no stable InMemoryProvider in the PHP SDK today); Unity has no OpenFeature SDK at all.
  • React is deliberately minimal — one paragraph plus a 10-line snippet, no RTL harness, no component test tree. Matches the rest of the React page's terse style.

Introduces a 'Testing with in-memory providers' section on both the
server-side and client-side feature flag landing pages. Explains that the
Datadog provider talks to Datadog's backend and is not appropriate for
unit tests, and directs readers to per-language pages for concrete
examples.
Teaches customers to swap DatadogProvider for OpenFeature's memprovider
in unit tests. Uses named clients via SetNamedProviderAndWait so t.Parallel()
is safe, and shows the ContextEvaluator pointer-to-func pattern.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
(Flag builder form) in JUnit 5 tests. Emphasizes OpenFeatureAPI singleton
reset via shutdown() in @AfterEach and mentions Spring Boot test setup.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
in pytest. Uses a function-scoped fixture with api.shutdown() teardown to
prevent global API singleton leakage between tests.
Teaches customers to swap DatadogProvider for OpenFeature's
TypedInMemoryProvider in Vitest tests. Shows OpenFeature.close() teardown
and per-test putConfiguration() to keep tests isolated from each other.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
in xUnit via IAsyncLifetime. Notes cross-framework applicability (NUnit,
MSTest) and the WebApplicationFactory integration path for ASP.NET Core.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
in RSpec via an around hook that captures and restores the prior provider.
Notes the Ruby SDK's plain-value flag hash (no variants) and the add_flag
mutation API.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
from @openfeature/web-sdk in Vitest. Calls out the required flag shape
(variants, defaultVariant, disabled) and the sync evaluation model.
Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
in tests. Keeps the section minimal per the client-testing style: short
paragraph plus a 10-line snippet, no RTL or component harness.
Kotlin/Android OpenFeature SDK does not ship an InMemoryProvider, so the
section includes a ~30-line FakeProvider stub and walks through setting
it up under runTest from kotlinx-coroutines-test.
OpenFeature Swift SDK ships no InMemoryProvider and has no contrib package,
so the section includes a ~50-line InMemoryTestProvider FeatureProvider
stub. Uses async setProviderAndWait and clearProviderAndWait for teardown.
@github-actions

Copy link
Copy Markdown
Contributor

Preview links (active after the build_preview check completes)

Modified Files

Corrections from per-language expert review against current upstream SDKs:

- Go: ContextEvaluator is already *func(...); drop the extra cast so the
  snippet compiles (the prior form produced a **func(...) conversion error).
- Python: context_evaluator returns a FlagResolutionDetails, not a bare
  variant name. Fix prose wording.
- .NET: InMemoryProvider's mutation method is UpdateFlagsAsync, not
  UpdateFlags. Fix prose wording.
- JavaScript (client) and React: swap the deprecated InMemoryProvider for
  TypedInMemoryProvider (upstream marks the former @deprecated) and
  mention OpenFeatureTestProvider as the idiomatic React component-test
  wrapper.
…e-provider-swift

Points readers at the Datadog OpenFeature bridge (dd-openfeature-provider-swift)
as the production path for OpenFeature on iOS. Tests swap the bridge's
DatadogProvider for a small custom FeatureProvider because the upstream
OpenFeature Swift SDK does not ship an in-memory provider.
Verification caught six compile errors in the InMemoryTestProvider stub:

- initialize/onContextSet are 'async throws', not Future<Void, Never>
- observe() returns AnyPublisher<ProviderEvent?, Never> (optional event)
- ProviderEvent.ready has no associated value; use .ready not .ready(nil)
- Reason case is .staticReason, not .static ('static' is reserved)
- OpenFeatureAPI.shared.clearProvider() is sync and does not exist as
  clearProviderAndWait() in 0.3.0

Also update the prose callout to match.
…g's CDN

Client-side SDKs reach Datadog's CDN to retrieve flag assignments, not
the Remote Configuration backend used by server-side SDKs. Updates the
testing-section callouts on the shared client _index and on each
per-platform client page so the language is accurate.
…tion

In-memory providers are one testing approach; running the real Datadog
provider against a dedicated test environment is equally valid. Updates
the shared _index sections and each per-language page so they present
both options up front, then proceed with the in-memory example as
'this section covers...' rather than 'do not use the Datadog provider'.
Adds a new 'Use via OpenFeature' section covering install of
dd-openfeature-provider-swift, OpenFeatureAPI setup, evaluation context,
typed flag evaluation, and flag details. Existing FlagsClient-based
sections are unchanged. Includes a 'not yet production-ready' callout
since the bridge package is still in development.
@leoromanovsky
leoromanovsky marked this pull request as ready for review April 22, 2026 16:11
@leoromanovsky
leoromanovsky requested a review from a team as a code owner April 22, 2026 16:11
@leoromanovsky
leoromanovsky requested review from a team, sameerank and typotter and removed request for a team April 22, 2026 16:12
@maycmlee maycmlee self-assigned this Apr 22, 2026
@maycmlee maycmlee added the editorial review Waiting on a more in-depth review label Apr 22, 2026
@maycmlee maycmlee removed their assignment Apr 22, 2026
@maycmlee

Copy link
Copy Markdown
Contributor

Created DOCS-14128 for review.

Comment on lines +316 to +321
if err := openfeature.SetNamedProviderAndWait(name, provider); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_ = openfeature.SetNamedProviderAndWait(name, openfeature.NoopProvider{})
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to include guidance for named providers in testing

Datadog supports two testing approaches:

- **Integration tests**: Point `DatadogProvider` at a dedicated test environment and control flag values from the Datadog UI. This exercises the real provider end-to-end, including the CDN-delivered flag assignments.
- **Unit tests**: Swap `DatadogProvider` for OpenFeature's standard `InMemoryProvider` (or an equivalent test stub, where no in-memory provider is available in the language) and set flag values directly in test code. This keeps tests hermetic, fast, and offline.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment thread content/en/feature_flags/client/_index.md Outdated
Comment thread content/en/feature_flags/client/android.md Outdated
Comment thread content/en/feature_flags/client/android.md Outdated
leoromanovsky and others added 3 commits April 22, 2026 18:27
Address review from @typotter:
- Remove events.emit(ProviderReady) from FakeProvider.initialize; the SDK
  emits ProviderReady after initialize returns.
- Soften the InMemoryProvider wording and link to open-feature/kotlin-sdk#226,
  where upstream in-memory provider support is tracked.
Apply @typotter's step 3 rewrite from client/_index.md to server/_index.md
for consistency: convey that units under test get an injected OpenFeature
client backed by the InMemoryProvider.

@joepeeples joepeeples left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for preparing this content, @leoromanovsky! Added some style suggestions, mostly line edits. Let me know when ready for a re-review.

Comment thread content/en/feature_flags/client/_index.md Outdated
Comment thread content/en/feature_flags/server/_index.md Outdated
Comment thread content/en/feature_flags/client/_index.md Outdated
Comment thread content/en/feature_flags/server/_index.md Outdated
Comment thread content/en/feature_flags/client/ios.md Outdated
Comment thread content/en/feature_flags/server/dotnet.md Outdated
Comment thread content/en/feature_flags/server/java.md Outdated
Comment thread content/en/feature_flags/server/nodejs.md Outdated
Comment thread content/en/feature_flags/client/_index.md Outdated
Comment thread content/en/feature_flags/server/_index.md Outdated
leoromanovsky and others added 13 commits April 29, 2026 12:38
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
…docs-testing-in-memory

# Conflicts:
#	content/en/feature_flags/client/android.md
@leoromanovsky
leoromanovsky merged commit c7e4010 into master Apr 30, 2026
15 checks passed
@leoromanovsky
leoromanovsky deleted the leo.romanovsky/ffe-docs-testing-in-memory branch April 30, 2026 02:36
estherk15 pushed a commit that referenced this pull request May 4, 2026
* feature_flags: add shared testing section to server and client _index

Introduces a 'Testing with in-memory providers' section on both the
server-side and client-side feature flag landing pages. Explains that the
Datadog provider talks to Datadog's backend and is not appropriate for
unit tests, and directs readers to per-language pages for concrete
examples.

* feature_flags: add Testing section to Go server-side page

Teaches customers to swap DatadogProvider for OpenFeature's memprovider
in unit tests. Uses named clients via SetNamedProviderAndWait so t.Parallel()
is safe, and shows the ContextEvaluator pointer-to-func pattern.

* feature_flags: add Testing section to Java server-side page

Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
(Flag builder form) in JUnit 5 tests. Emphasizes OpenFeatureAPI singleton
reset via shutdown() in @AfterEach and mentions Spring Boot test setup.

* feature_flags: add Testing section to Python server-side page

Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
in pytest. Uses a function-scoped fixture with api.shutdown() teardown to
prevent global API singleton leakage between tests.

* feature_flags: add Testing section to Node.js server-side page

Teaches customers to swap DatadogProvider for OpenFeature's
TypedInMemoryProvider in Vitest tests. Shows OpenFeature.close() teardown
and per-test putConfiguration() to keep tests isolated from each other.

* feature_flags: add Testing section to .NET server-side page

Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
in xUnit via IAsyncLifetime. Notes cross-framework applicability (NUnit,
MSTest) and the WebApplicationFactory integration path for ASP.NET Core.

* feature_flags: add Testing section to Ruby server-side page

Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
in RSpec via an around hook that captures and restores the prior provider.
Notes the Ruby SDK's plain-value flag hash (no variants) and the add_flag
mutation API.

* feature_flags: add Testing section to JavaScript client-side page

Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
from @openfeature/web-sdk in Vitest. Calls out the required flag shape
(variants, defaultVariant, disabled) and the sync evaluation model.

* feature_flags: add Testing section to React client-side page

Teaches customers to swap DatadogProvider for OpenFeature's InMemoryProvider
in tests. Keeps the section minimal per the client-testing style: short
paragraph plus a 10-line snippet, no RTL or component harness.

* feature_flags: add Testing section to Android client-side page

Kotlin/Android OpenFeature SDK does not ship an InMemoryProvider, so the
section includes a ~30-line FakeProvider stub and walks through setting
it up under runTest from kotlinx-coroutines-test.

* feature_flags: add Testing section to iOS client-side page

OpenFeature Swift SDK ships no InMemoryProvider and has no contrib package,
so the section includes a ~50-line InMemoryTestProvider FeatureProvider
stub. Uses async setProviderAndWait and clearProviderAndWait for teardown.

* feature_flags: apply upstream OpenFeature SDK verification fixes

Corrections from per-language expert review against current upstream SDKs:

- Go: ContextEvaluator is already *func(...); drop the extra cast so the
  snippet compiles (the prior form produced a **func(...) conversion error).
- Python: context_evaluator returns a FlagResolutionDetails, not a bare
  variant name. Fix prose wording.
- .NET: InMemoryProvider's mutation method is UpdateFlagsAsync, not
  UpdateFlags. Fix prose wording.
- JavaScript (client) and React: swap the deprecated InMemoryProvider for
  TypedInMemoryProvider (upstream marks the former @deprecated) and
  mention OpenFeatureTestProvider as the idiomatic React component-test
  wrapper.

* feature_flags: frame iOS testing around OpenFeature and dd-openfeature-provider-swift

Points readers at the Datadog OpenFeature bridge (dd-openfeature-provider-swift)
as the production path for OpenFeature on iOS. Tests swap the bridge's
DatadogProvider for a small custom FeatureProvider because the upstream
OpenFeature Swift SDK does not ship an in-memory provider.

* feature_flags: fix iOS test snippet against OpenFeature Swift SDK 0.3.0

Verification caught six compile errors in the InMemoryTestProvider stub:

- initialize/onContextSet are 'async throws', not Future<Void, Never>
- observe() returns AnyPublisher<ProviderEvent?, Never> (optional event)
- ProviderEvent.ready has no associated value; use .ready not .ready(nil)
- Reason case is .staticReason, not .static ('static' is reserved)
- OpenFeatureAPI.shared.clearProvider() is sync and does not exist as
  clearProviderAndWait() in 0.3.0

Also update the prose callout to match.

* feature_flags: clarify client-side provider fetches flags from Datadog's CDN

Client-side SDKs reach Datadog's CDN to retrieve flag assignments, not
the Remote Configuration backend used by server-side SDKs. Updates the
testing-section callouts on the shared client _index and on each
per-platform client page so the language is accurate.

* feature_flags: soften testing guidance to present in-memory as one option

In-memory providers are one testing approach; running the real Datadog
provider against a dedicated test environment is equally valid. Updates
the shared _index sections and each per-language page so they present
both options up front, then proceed with the in-memory example as
'this section covers...' rather than 'do not use the Datadog provider'.

* feature_flags: add OpenFeature usage section to iOS page

Adds a new 'Use via OpenFeature' section covering install of
dd-openfeature-provider-swift, OpenFeatureAPI setup, evaluation context,
typed flag evaluation, and flag details. Existing FlagsClient-based
sections are unchanged. Includes a 'not yet production-ready' callout
since the bridge package is still in development.

* feature_flags: note getIntegerValue returns Int64 in iOS OpenFeature section

* Update content/en/feature_flags/client/_index.md

Co-authored-by: Tyler Potter <[email protected]>

* feature_flags/android: drop ProviderReady emit, note kotlin-sdk#226

Address review from @typotter:
- Remove events.emit(ProviderReady) from FakeProvider.initialize; the SDK
  emits ProviderReady after initialize returns.
- Soften the InMemoryProvider wording and link to open-feature/kotlin-sdk#226,
  where upstream in-memory provider support is tracked.

* feature_flags/server: mirror client step 3 rewording

Apply @typotter's step 3 rewrite from client/_index.md to server/_index.md
for consistency: convey that units under test get an injected OpenFeature
client backed by the InMemoryProvider.

* 1 picker only

Co-authored-by: Joe Peeples <[email protected]>

* remove dup from server

Co-authored-by: Joe Peeples <[email protected]>

* split sentence

Co-authored-by: Joe Peeples <[email protected]>

* action verb

Co-authored-by: Joe Peeples <[email protected]>

* with of

Co-authored-by: Joe Peeples <[email protected]>

* action

Co-authored-by: Joe Peeples <[email protected]>

* comma dotnet

Co-authored-by: Joe Peeples <[email protected]>

* comma nodejs

Co-authored-by: Joe Peeples <[email protected]>

* fast no promises

Co-authored-by: Joe Peeples <[email protected]>

* fast server no promises

Co-authored-by: Joe Peeples <[email protected]>

* ios bridge

Co-authored-by: Joe Peeples <[email protected]>

* these

Co-authored-by: Joe Peeples <[email protected]>

* Apply suggestions from code review

Co-authored-by: Joe Peeples <[email protected]>

---------

Co-authored-by: Tyler Potter <[email protected]>
Co-authored-by: Joe Peeples <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

editorial review Waiting on a more in-depth review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants