Automating CSS Regression Testing

The following is a guest post by Garris Shipon. We’ve touched on the four types of CSS testing here before. Regression testing is the hardest. It’s the type where you’re trying to test if a change you made to CSS resulted in any unexpected visual problems. This is made more difficult with responsive designs. Garris built a tool for doing this as he embarked upon a new responsive design for a large scale site. Here, he’ll walk you through the whole thing.

This article was originally published in December 2014. It has been re-written, updated, and republished now in April 2016 because BackstopJS, the main tool presented here, as been updated.

A use-case for visual regression testing

Do a search for “CSS regression testing” and a common theme becomes clear: breaking CSS is easy, testing it is hard.

This was the case at the onset of a responsive CSS refactoring project I scoped for a large online retailer. Like many other web companies at the time, we were in the process of adding responsive behavior to a massive e-commerce web app, which was originally designed for 1024px desktop screens.

I realized this would be a regression-prone job. Retrofitting multiple breakpoint behaviors meant we would likely have a lot of hard-to-find display bugs. I needed a way for our engineers to automate bug discovery before slamming our QA team with hundreds of ticky-tacky little layout issues.

Where BackstopJS fits in

The solution I wanted had to play nice with web developers. That is, easy to install locally, use familiar web dev paradigms, and give a reasonable amount of confidence that a selector change made for mobile isn’t going to result in a hard-to-find bug in a desktop layout.

At the time, there wasn’t anything out of-the-box that quite fit the bill. This was the reason for creating BackstopJS.

BackstopJS is a visual regression testing app which wraps CasperJS, PhantomJS and ResembleJS in an easy-to-configure test matrix across multiple app-states (URLs), DOM elements and screen sizes.

The following is a 15 minute walk-through of an installation and initial configuration of BackstopJS.

A visual regression-test tutorial

This instructional will be based on a simple demo project (download ZIP here). It is taken directly from the Bootstrap example page.

Expand the simple demo

Unzip the project download. We will install the testing framework right into this example project:

Here is what you’ll see if you open up myCoolProject/index.html in a web browser… Remember, it’s responsive. So, make sure to resize the browser window down to see the most narrow layout!

Install BackstopJS with NPM

Now go to your project root (`/myCoolProject/`) and run:

$ cd ~/path-to-myProjects/myCoolProject
$ npm install backstopjs

Your directory should now look like this:

Install complete! Now let’s get to some basic testing…

Generating a BackstopJS configuration template

The basic configuration process is straightforward from here. To help with things, BackstopJS can generate a config file that you can modify for your project. From the `myCoolProject/node_modules/backstopjs/` directory run:

$ cd ~/path-to-myProjects/myCoolProject/node_modules/backstopjs/
$ npm run genConfig

This will add files to your project root: folders for BackstopJS screenshots, `backstop_data`, and generating a boilerplate configuration file `backstop.json`.

The configuration file is where you’ll specify your testing rules. Let’s look at that file.

{
  "viewports": [
    {
      "name": "phone",
      "width": 320,
      "height": 480
    }, {
      "name": "tablet_v",
      "width": 568,
      "height": 1024
    }, {
      "name": "tablet_h",
      "width": 1024,
      "height": 768
    }
  ],
  "scenarios": [
    {
      "label": "My Homepage",
      "url": "http://getbootstrap.com",
      "hideSelectors": [],
      "removeSelectors": [
        "#carbonads-container"
      ],
      "selectors": [
        "header",
        "main",
        "body .bs-docs-featurette:nth-of-type(1)",
        "body .bs-docs-featurette:nth-of-type(2)",
        "footer",
        "body"
      ],
      "readyEvent": null,
      "delay": 500,
      "onReadyScript": null,
      "onBeforeScript": null
    }
  ],
  "paths": {
    "bitmaps_reference": "../../backstop_data/bitmaps_reference",
    "bitmaps_test": "../../backstop_data/bitmaps_test",
    "compare_data": "../../backstop_data/bitmaps_test/compare.json",
    "casper_scripts": "../../backstop_data/casper_scripts"
  },
  "engine": "phantomjs",
  "report": ["browser", "CLI"],
  "cliExitOnFail": false,
  "debug": false,
  "port": 3001
}

In this configuration you can see three viewports objects. One for phone, tablet vertical, and tablet horizontal, each with name and dimensions properties. You can add as many viewports objects as you need. BackstopJS requires at least one.

Then we have scenarios which include the URLs and element selectors that BackstopJS will test. It’s useful to think of every scenario object as a test for a specific static page or global app state. Add as many scenarios as you need. BackstopJS requires at least one.

Inside each scenario is a list of selectors. Selectors accept standard CSS notation. For each selector you specify, BackstopJS will take a screenshot and test that area of your layout. Your selector area could be as small as a button or as big as the body of your page — check the documentation for more on using this feature with dynamic web apps.

Modifying the configuration template

For our demo, make the following change and replace the scenarios node in `myCoolProject/backstop.json`.

"scenarios": [
  {
    "label": "My Local Test",
    "url": "../../index.html",
    "hideSelectors": [],
    "removeSelectors": [
    ],
    "selectors": [
      "nav",
      ".jumbotron",
      "body .col-md-4:nth-of-type(1)",
      "body .col-md-4:nth-of-type(2)",
      "body .col-md-4:nth-of-type(3)",
      "footer"
    ],
    "readyEvent": null,
    "delay": 0,
    "onReadyScript": null,
    "onBeforeScript": null
  }
],

Generating reference screenshots

From the `myCoolProject/node_modules/backstopjs/` directory run…

$ npm run reference

This task will create (or update an existing) screen captures representing all specified selectors at every breakpoint. When the process is complete, take a look inside `/myCoolProject/backstop_data/bitmaps_reference/`:

So far so good. We have our reference set. Now let’s run a test!

Running our first test

We are about to run our first test. But keep in mind, we haven’t changed anything in our CSS yet, so our tests should pass!

From the `myCoolProject/node_modules/backstopjs/` directory run:

$ npm run test

This task will create a new, timestamped-directory of test images inside `/myCoolProject/backstop_data/bitmaps_test/<timestamp>`.

Once the test images are generated, BackstopJS will open your web browser and display a report comparing the most recent test bitmaps against the current reference images. Significant differences (if any) are detected and shown.

In this instance, since we haven’t made any changes to our test page, BackstopJS should show all of our tests as passing. Now, let’s try changing our CSS and see what happens.

Updating our index file and running our second test

Here is what you’ll see if you open up our (unchanged) `myCoolProject/index.html` in a web browser. Notice the margin around the text:

Let’s mess that up! Open up `myCoolProject/index.html` and insert the following code just before the closing </head> tag:

<style>
  .jumbotron {
    padding: 0px;
  }
</style>

Here’s what the page looks like now:

This is exactly the kind of thing that happens all the time during web development. Some unscoped code gets in and hoses your layout just enough that you might not notice :(

Now, From the `myCoolProject/node_modules/backstopjs/` directory run:

$ npm run test

Our test should run again and errors should be found. Scroll the report down to see a visual diff of the issues we’ve just created…

Our visual diff contains the reference capture, the most recent test capture and the visual diff file.

There you have it: regression found!

This is a very simple example. In real life, designers and engineers may find themselves working on very large and or complex CSS projects. That is when a system like this really improves the quality and consistency of our work. By automating the repetitive visual tasks we can confidently pursue more creative ones.

About workflow

There are many ways to integrate this kind of test into your workflow. You could fire off tests every time you build or you could manually run tests (as you work or just before pushing to your next stage). You could even integrate BackstopJS into your CI pipeline, if that’s your thing. All of these topics are outside the scope of this article. Check the documentation for more info.

Next steps

Since first releasing in 2014, BackstopJS has grown substantially. There are loads of newly added features developed by the community:

• SPA testing support – Use Casper scripts and explicit web app triggers to ensure screenshots are captured at the correct time inside your web app (e.g. after API responses, after CSS animation completion, or wait for any other watchable async process).
• Simulating user interactions – Use Casper scripting inside your scenarios to simulate interactions with your on-screen components.
• CI pipeline integration – BackstopJS CLI features have enabled advanced users to make visual regression testing a part of their continuous integration process.
• Active configuration files – Enables active (node module) logic inside your config files which you can use to change testing behavior based on environment or other conditions, point to different config files to use your BackstopJS instance as a centralized test server for multiple environments, verticals, profiles, projects, or whatever.

More on BackstopJS

• BackstopJS.org
• Find documentation, file bugs, get troubleshooting help, learn about advanced features and contribute on GitHub!