Repository navigation
Test | UX | Storybook snapshot testing (build-over-build) for regression prevention #444
Description
Overview
In our CI, we run yarn storybook:build to try and vet some issues where the storybook stories get out of sync with the codebase:
coder/.github/workflows/coder.yaml
Line 276 in 97399f8
However, it seems like this isn't sufficient - as several errors cropped up in storybook: #443
In v1, we used the Chromatic to validate our stories stayed consistent build-over-build - having a tool like that integrated into our CI again would catch these sorts of failures in the future.
Current behavior
We do not have visual snapshot testing of our Storybook
V1 behavior
In V1 we had a free version of Chromatic and used it successfully. However, when the free tier usage runs up, it was holding our CI hostage, so we temporarily suppressed it. We need to get an enterprise plan approval and update our pipeline for it.
Furthermore, we noticed that some components (example a component that purposefully threw an error) would have false positives. They were addressed by suppressing Chromatic like this:
export const ShowingError = Template.bind({})
ShowingError.parameters = {
// The FallbackDialog is noisy for chromatic, because it renders an actual error
// along with the stacktrace - and the stacktrace includes the full URL of
// scripts in the stack. This is problematic, because every deployment uses
// a different URL, causing the validation to fail.
chromatic: { disableSnapshot: true },
}Desired behavior
We have visual snapshot testing of our Storybook
Implementation details
We prefer using Chromatic because it's recommended by Storybook
Activity
bryphe-coder commentedon Mar 15, 2022
The challenge we had in V1 is that we were using the free plan, and once it expired... it basically held our CI hostage by stalling the run. So we'd need to consider upgrading to a paid plan, or using an alternative free tool.
greyscaled commentedon Mar 15, 2022
^ Was just about to add that context to the ticket, thanks for adding that!
bryphe-coder commentedon Mar 15, 2022
Haha, I saw you added the same comment in #443 in parallel - we're definitely on the same page!
greyscaled commentedon Mar 15, 2022
Updated ticket with details
[-]Test | UX | Bring in chromatic to validate storybook build-over-build[/-][+]Test | UX | Storybook snapshot testing (build-over-build) for regression prevention[/+]chore(site): configure chromatic snapshot tests
chore: configure chromatic snapshot tests (#896)
chore: speed up chromatic ci
chore: speed up chromatic ci (#904)