Add support for deterministic mock generation#7629
Conversation
🦋 Changeset detectedLatest commit: b7ac1ba The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📝 WalkthroughSummary by CodeRabbit
WalkthroughA new option Changes
Sequence DiagramsequenceDiagram
participant User
participant mockServer
participant addMocksToSchema
participant MockStore
participant takeOneOf
User->>mockServer: mockServer(schema, mocks, ..., mockGenerationBehavior?)
mockServer->>addMocksToSchema: addMocksToSchema({... mockGenerationBehavior})
addMocksToSchema->>MockStore: createMockStore({... mockGenerationBehavior})
MockStore->>MockStore: merge defaults (defaultCommon + (deterministic?deterministicMocks:randomMocks))
rect rgba(100,150,200,0.12)
Note over MockStore,takeOneOf: value selection during mock generation
MockStore->>takeOneOf: takeOneOf(values, mockGenerationBehavior)
alt deterministic
takeOneOf-->>MockStore: return first value
else random
takeOneOf-->>MockStore: return random value
end
end
MockStore-->>User: resolved mock values
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
🔇 Additional comments (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/mock/src/MockStore.ts (1)
748-768: Consider clarifying array length behavior in documentation.The JSDoc states "For Arrays an array of random length will be generated" in random mode, but
randomListLength()(line 14 of utils.ts) always returns 2. Consider either:
- Updating the documentation to reflect that arrays have a fixed length of 2
- Implementing actual random-length arrays for random mode (though this might introduce more flakiness)
The current behavior (always length 2) is actually less flaky, so updating the documentation might be the better approach.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
packages/mock/src/MockStore.ts(7 hunks)packages/mock/src/addMocksToSchema.ts(3 hunks)packages/mock/src/mockServer.ts(3 hunks)packages/mock/src/types.ts(1 hunks)packages/mock/src/utils.ts(2 hunks)packages/mock/tests/store.spec.ts(3 hunks)website/src/content/mocking.mdx(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (5)
packages/mock/src/mockServer.ts (2)
packages/utils/src/Interfaces.ts (1)
TypeSource(272-279)packages/mock/src/types.ts (2)
IMocks(10-22)MockGenerationBehavior(24-24)
packages/mock/src/utils.ts (1)
packages/mock/src/types.ts (1)
MockGenerationBehavior(24-24)
packages/mock/tests/store.spec.ts (1)
packages/mock/src/MockStore.ts (1)
MockStore(57-678)
packages/mock/src/MockStore.ts (2)
packages/mock/src/types.ts (3)
MockGenerationBehavior(24-24)IMocks(10-22)TypePolicy(28-37)packages/mock/src/utils.ts (1)
takeOneOf(20-26)
packages/mock/src/addMocksToSchema.ts (1)
packages/mock/src/types.ts (3)
IMockStore(84-216)IMocks(10-22)MockGenerationBehavior(24-24)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: Unit Test on Bun
- GitHub Check: Unit Test on Node 18 (windows-latest) and GraphQL v16
- GitHub Check: Unit Test on Node 24 (ubuntu-latest) and GraphQL v16
- GitHub Check: Unit Test on Node 18 (ubuntu-latest) and GraphQL v15
- GitHub Check: Unit Test on Node 22 (ubuntu-latest) and GraphQL v16
- GitHub Check: Unit Test on Node 18 (ubuntu-latest) and GraphQL v16
- GitHub Check: Full Check on GraphQL v16
🔇 Additional comments (17)
packages/mock/src/types.ts (1)
24-24: LGTM!The new
MockGenerationBehaviortype is cleanly defined and appropriately exported for use across the mocking system.packages/mock/src/utils.ts (2)
3-3: LGTM!Import correctly adds the new
MockGenerationBehaviortype to support the behavior-aware selection logic.
20-26: LGTM!The refactored
takeOneOffunction cleanly implements deterministic selection (first element) and random selection based on the behavior parameter.website/src/content/mocking.mdx (1)
415-428: LGTM!The documentation clearly explains the new
mockGenerationBehavioroption, its default value, and when to use the deterministic mode to reduce test flakiness.packages/mock/src/addMocksToSchema.ts (3)
14-14: LGTM!Import correctly adds the
MockGenerationBehaviortype for use in the options interface.
21-21: LGTM!The optional
mockGenerationBehaviorparameter properly extendsIMockOptionswhile maintaining backward compatibility.
97-97: LGTM!The parameter is correctly destructured from options and properly forwarded to
createMockStore, enabling the behavior to affect mock generation throughout the system.Also applies to: 117-117
packages/mock/src/mockServer.ts (3)
5-5: LGTM!Import correctly adds the
MockGenerationBehaviortype for the public API.
18-26: LGTM!The function signature is properly extended with the optional
mockGenerationBehaviorparameter, and the JSDoc clearly documents its purpose.
34-34: LGTM!The parameter is correctly forwarded to
addMocksToSchema, completing the integration through the public API surface.packages/mock/src/MockStore.ts (4)
82-86: Verify the variable name reference after fixing the typo.Once the typo in
defaultDeteministicMocksis fixed todefaultDeterministicMockson line 40, ensure this reference is also updated.
60-60: LGTM!The
mockGenerationBehaviorfield is properly added as a private member, correctly initialized in the constructor with a sensible default of 'random', and the constructor signature properly extends the options to accept this parameter.Also applies to: 70-70, 75-75, 81-81
570-570: LGTM!The enum value generation correctly uses
takeOneOfwith the instance'smockGenerationBehavior, enabling deterministic first-value selection when configured.
584-587: LGTM!Abstract type resolution correctly uses
takeOneOfwith the instance'smockGenerationBehaviorto select from possible types, enabling deterministic behavior when configured.packages/mock/tests/store.spec.ts (3)
20-20: LGTM!The schema additions (
verified: Booleanon User,rating: Floaton Book interface and implementations) provide the necessary fields to test the new mock generation behaviors comprehensively.Also applies to: 44-44, 50-50, 57-57
677-758: LGTM!The random behavior test suite comprehensively validates that the random mode produces varied values across Int, Float, Boolean, Enum, and Union types. The approach of generating 100 samples and checking for variation is robust.
760-832: LGTM!The deterministic behavior test suite thoroughly validates that the deterministic mode produces consistent, predictable values (1 for Int, 1.5 for Float, true for Boolean, first enum value, first union type, etc.). This provides strong confidence that the feature works as intended.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
packages/mock/tests/store.spec.ts (2)
677-758: Strong coverage for random behavior; fix misleading test title.The suite thoroughly validates randomness for Int, Float, Enum, Union, and Boolean. One nit: the test named “should return random values for Boolean scalars” actually checks Union variant randomness.
Apply this rename:
- it('should return random values for Boolean scalars', () => { + it('should return random union variants for UserImage', () => {
760-844: Deterministic suite is comprehensive; consider one extra assertion path.All key types are validated, including arrays length=2. Optionally add a companion test using
createMockStore({ mockGenerationBehavior: 'deterministic' })to ensure the factory path also enforces determinism (not justnew MockStore).packages/mock/src/MockStore.ts (1)
748-771: Clarify JSDoc for String behavior in random mode (nit).Doc specifies deterministic String = "Hello World"; it doesn’t state random mode explicitly. Since
Stringcomes fromdefaultCommonMocks, it’s "Hello World" in both modes. Consider an extra sentence to avoid confusion.Apply:
* - For String scalars we'll always return "Hello World" + * (Strings use the same default in both modes.)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/mock/src/MockStore.ts(7 hunks)packages/mock/tests/store.spec.ts(3 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
packages/mock/src/MockStore.ts (2)
packages/mock/src/types.ts (3)
MockGenerationBehavior(24-24)IMocks(10-22)TypePolicy(28-37)packages/mock/src/utils.ts (1)
takeOneOf(20-26)
packages/mock/tests/store.spec.ts (1)
packages/mock/src/MockStore.ts (1)
MockStore(57-678)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: Unit Test on Node 22 (ubuntu-latest) and GraphQL v16
- GitHub Check: Unit Test on Node 24 (ubuntu-latest) and GraphQL v16
- GitHub Check: Unit Test on Node 18 (ubuntu-latest) and GraphQL v16
- GitHub Check: Unit Test on Node 18 (ubuntu-latest) and GraphQL v15
- GitHub Check: Unit Test on Node 18 (windows-latest) and GraphQL v16
- GitHub Check: Unit Test on Bun
- GitHub Check: Full Check on GraphQL v16
🔇 Additional comments (6)
packages/mock/tests/store.spec.ts (2)
19-21: Schema field addition looks good.The new
verified: Booleanfield is consistent with later tests. No issues.
44-44: Rating field alignment across interface and implementors.Adding
rating: FloattoBook,TextBook, andColoringBookkeeps the schema coherent for interface field resolution. Looks good.Also applies to: 50-50, 57-57
packages/mock/src/MockStore.ts (4)
569-571: Enum selection correctly respects behavior.Switch to
takeOneOf(values, this.mockGenerationBehavior)is correct and minimal.
584-587: Abstract type resolution wired to behavior.Using
takeOneOfover possible types matches the new option; good change.
82-86: Mocks merge order is sensible.
defaultCommonMocks→ behavior-specific defaults → usermocksensures user overrides win. LGTM.
574-578: No change required—randomListLength()already returns 2 deterministically.The review comment assumes
randomListLength()generates varying lengths in random mode. In reality,randomListLength()(defined inpackages/mock/src/utils.ts:14-18) is hardcoded to always return2. The commented-out code at line 16 shows a prior random implementation, but the current version is deterministic. Both'random'and'deterministic'modes already produce arrays of length 2, matching the expected behavior.Likely an incorrect or invalid review comment.
|
Thanks for the PR @sanniassin ! Could you add a changeset with |
|
@ardatan Sure, added a changeset 🙌 |
3 participants
Description
This PR implements
mockGenerationBehavioroption inaddMocksToSchema, which gives control over randomness in autogenerated mocks. By default the current random behavior is preserved, but now it's also possible to passmockGenerationBehavior: 'deterministic'option.It aims to reduce risks of flakiness in apps with large sophisticated GraphQL schemas, where app's behavior may change because of some specific combination of boolean flag and status enums and the test's author may not be aware of all such combinations. With random generation this may cause flakiness in tests, because a specific non-working combination of flags and enums may only occur randomly once per 100 test runs.
Though this issue could already be worked around by passing explicit mocks for all types affected by randomness, this only works well for scalar types and simple schemas. In large codebases with hundreds of union and enum fields this can hardly be done well and maintained.
Related # (issue)
@graphql-tools/mock- controllable randomness when mocking out union types #5614Type of change
Please delete options that are not relevant.
How Has This Been Tested?
deterministicandrandombehaviors in packages/mock/tests/store.spec.ts.mockGenerationBehavioroption directly tocreateMockStoreand viaaddMocksToSchema.Test Environment:
Checklist:
CONTRIBUTING doc and the
style guidelines of this project