Software Testing for

DevOps-Driven Teams
BY JUNE JUNG,
ENGINEERING MANAGER
github.com/junejung
If you’ve been hearing the term DevOps a lot much faster. The end goal is to increase both
recently but you’re not quite sure what it is productivity and the quality of the product
or whether it’s relevant to your organization, and the end user experience.
you’ve come to the right place. We’re here to
As we’ll address in this ebook, DevOps isn’t
break down exactly what DevOps is so you
right for every organization. If your current
can determine whether it would benefit your
workflow functions well, transitioning to
engineering team and, ultimately, your end
DevOps may be an unnecessary undertaking.
users.
Depending on your product, it may be the
We’ll cover this in depth, but for a short wrong move as well. After all, an iterative
introduction, DevOps is an approach to process wouldn’t serve a company that must
software engineering that brings the do major launches due to privacy and other
domains and skillsets of developers and concerns.
operators together to facilitate faster, better
However, DevOps is becoming increasingly
development infrastructures. It’s essentially
popular for a reason. In this ebook, we’ll walk
an iterative process that enables you to test
you through exactly what DevOps is, when it
early and often, and to consider the extended
makes sense to adopt it, and how and when
lifecycle and operability of code early on,
to test throughout your processes.
to identify problems and solve them that

Software Testing for DevOps-Driven Teams • 2

Table of 4 Moving to DevOps:
What tools do you really need?
Contents
16 How to Test Software, Part I:
Mocking, Stubbing, and Contract Testing

26 How to Test Software, Part II:

TDD and BDD

Software Testing for DevOps-Driven Teams • 3

Moving to DevOps:
What tools do you really need?

DevOps has been the latest in a long succession of Organizations often think that the top-down model
problem-solving processes that each come with a (“Use this! Do this!”) will get their teams innovating
digital garage full of tools: CI/CD systems, testing faster. Eager team leads will bring in a new CI/CD
frameworks, monitoring tools, and security audit tool and get everyone up and running on it. Providing
tools to name a few. proper tooling for individual contributors to follow
an adopted practice is important, but the problem
Thinking about DevOps raises a number of begins when team leads bring in a tool without fully
questions for organizations. Which of these tools understanding its value or why they are doing it.
do you need? Which will solve the problems, pains, Oftentimes, even the people who ordered the change
and slowdowns your organization faces? What will forget why they put it in place to begin with,
organizational structure do you need to have to or what they wanted to get out of it. The sad truth
support it? What tools should you implement? is how easy it is to start misusing tools once the
original reasoning is obscured, thus creating lost or
These are all fine questions to ask, but asked in
even negative value.
isolation, they miss the point. When you go straight
to these queries, you are thinking about solutions
before you’ve really assessed the problem that you
are trying to solve.

Software Testing for DevOps-Driven Teams • 4

We want DevOps! It also means getting all product contributors to care
more about the end result of what they’re working on
Many organizations are convinced that DevOps is — how does it function in the world? How are users
the solution to all their problems, if only they can get interacting with it? Getting people to truly care about
it up and running quickly. If you’re in this position, quality means caring about both business value and
ask yourself, “Why do we want DevOps in our usability. When everyone who is building a product
organization? What value do we think it will bring?” cares about both of these aspects, you know you’ve
achieved true DevOps adoption.
At this point, let’s briefly talk about what DevOps is,
and isn’t. In our experience, this kind of widespread buy-in
has been particularly difficult for software teams to
DevOps is a very specific collaboration between
achieve because it requires a lot of cooperation from
developers and operations teams. In essence, it
people with different skills and domain expertise.
indicates that you’ve culturally adopted development
Pulling this off depends on both cross-functional
practices into your infrastructure and operational
team structure and thoughtful communication
practices into your development cycle. What does
skills. For example, if an engineer needs to talk to
this look like in practice? It can mean maintaining
someone on the business side about a database
infrastructure as code, or creating immutable
problem, she needs to not just show the data she’s
infrastructure by building reusable components so
working from, but give necessary context to focus
you can tear down or up whenever you want, giving
that person’s attention on what they should care
you version control and a history of changes you’ve
about and why.
made.

Software Testing for DevOps-Driven Teams • 5

New tools can sometimes seem like a quick fix, them, given their cultures or their product needs.
but they are not a one-size-fits-all solution. In our We’ve seen waterfall work really well for some
experience, keeping the following considerations very successful organizations. For example, if
in mind before you bring in a new DevOps tool will confidentiality is a big part of your company’s
increase your chances of success. product strategy, then shipping incrementally to
get feedback would not work for you, as you’d
1. M
 ake sure everyone is on the same page with need to keep all product details under lock and
respect to what you’re trying to achieve with key until a big launch. In that environment, it
this transformation. Everyone should agree on would be very difficult, and counterproductive, to
the problem you’re trying to solve and should be build a DevOps culture.
aligned on the pain points.
4. A
 lways measure. Before you start any
2. A
 lways start small. Don’t try to make your improvement plan, get accurate metrics for
entire organization into a model DevOps team where you’re currently at (i.e. “our dev cycle takes
overnight. Instead, start with one team, and X time”). Measure before and after you make a
see if the process change works within that change, to see if you’ve improved. As an example:
group. If you see improvements, keep moving When Agile transformation was at its peak, a
incrementally. lot of companies adopted standups (brief daily
status meetings), without really understanding
3. D
 o what works for you. Know that DevOps might
why, and without measuring whether it had a
not be the right solution for your organization.
positive effect on their team. This likely wasted
Some companies have been successful for a long
more time than it saved.
time without DevOps, and it might not be right for

Software Testing for DevOps-Driven Teams • 6

5. D
 o not try to automate everything. At least not Still itching to automate? Dockerizing your
all at once. One misconception about DevOps application is a great way to automate because the
is that all the infrastructure provisioning and work you put in is likely to be re-used. Automating
the configuration management must be done pre-production environment creation is another
automatically. This is referred to as “infrastructure great way to implement automation. As another
as code.” But some things work better when they example: are you trying to automate firewall setup?
are manual; automation is not the solution for That might not be worth it given the lack of API
everything. Also, think about how many times support on a lot of current firewall software. While
you’re going to run that automation script and it’s prudent to prepare for disaster, you’d probably be
how much time it would take you to set up. Will putting so much more value into it than you’ll ever
you use it thousands of times or only three? get back out of it.
Additionally, sometimes you have to start the
manual way to even figure out what would be the If your org is thinking about DevOps transformation,
best solution for automation. start thinking about your speed of delivery and
quality of product. What’s getting in your way today?
Knowing the answers to these questions will help
everyone in your organization understand what your
pain points are, so you’ll be in the best position to
start improving on them.

Software Testing for DevOps-Driven Teams • 7

The path to production: how and where next, a build has to pass through these quality gates.
to segregate test environments The gates will be separated using various kinds of
tests. Once tests are passed, quality at that level has
Once you decide to transition to DevOps, bear in been assured, and the build can move on.
mind that bringing a new tool into an organization
is no small task. Adopting a CI/CD tool, or any other These pipeline stages, or build environments, are
tool should follow a period of research, analysis, and separated by the increasing scope of responsibility
alignment within your organization. that the developer assumes, ranging from their
local laptop to the team space to the application’s
The precursor to any successful tool adoption is entire codebase. As the scope of responsibility
about people: alignment on purpose, and setting grows, the cost of mistakes is higher. That’s why we
expectations appropriately, as well as getting some test incrementally before passing a build to each
“before” metrics to support your assessment. successive level.

The next step is about analysis: deducing exactly What’s more, each stage requires different types of
what pipeline problem you most need to solve. tests. As you move from staging toward production,
By examining your development process (which the tests go from lighter weight to heavy duty. The
we’ll refer to as your “path to production”) you can cost in resources increases with each stage as
pinpoint where your biggest problems are coming well. Heavy duty testing can only happen in more
from. Only when you know what problem you’re production-like environments, involving a full tech
trying to solve can you make a well-reasoned tooling stack or external dependencies. In order to do these
decision. tests properly, there’s more to spin up, and they
require more expensive machinery. Therefore, it’s
The goal in examining your path to production beneficial if you can do as much testing as possible
is creating clear stages which will serve as in earlier environments, which are much less costly.
checkpoints. In order to pass from one stage to the
Software Testing for DevOps-Driven Teams • 8
Now let’s look at some common types of tests. Integration test: These check how well each unit
from the previous stage works with the other
Unit/component test: These cover the smallest components, units, and functionalities. In a broader
possible component, unit, or functionality. They’re sense, it can test how services (such as APIs)
the cheapest and fastest tests to run since they integrate with one another.
don’t require a lot of dependencies or mocking.
These should be done early to get them out of the UI layer testing: This is automated browser-based
way. testing which tests basic user flow. It is expensive to
set up and slow to run, so it should happen later in
the pipeline.

Software Testing for DevOps-Driven Teams • 9

Note: This example uses a microservices
application, which allows us to test and
deploy each service separately.
Next up, we’ll talk about how these tests fit into a
software development pipeline. Each test has a
specific role and place.

Local environment
This environment is private, limited to a single Tests: The tests we recommend before moving out
developer and their laptop. It is the easiest in which of a local environment are unit tests, integration
to make changes and test your own implementation. testing with mocked components, and UI testing to
the degree that it’s possible. The more tests you can
The “local” we are talking about here is really just a do in this environment, the more room you’ll have to
“personal environment.” It could be on your laptop, be successful in the integration environment.
but you could just as easily use a cloud environment
if the application is too big to run on your local Scope of responsibility: The scope here covers
machine. The key here is it’s a smaller scale instance just the implementation or functionality of what
that’s yours alone, in which you can test and debug you’re building. Does your component work by
your implementation while you’re developing, itself? The surface area is relatively small. But
without interrupting other developers on your team. this is the last environment before you merge to
Because nothing in your local environment is visible a shared environment. Therefore, you’ll want to
to others, your team can agree to integrate a Git test responsibly so that you don’t break the shared
pre-push hook into your repo to ensure the local environment later and block active development by
environment is used and to run automated tests others.
before code gets pushed to a remote, or shared,
repository.

Software Testing for DevOps-Driven Teams • 11

CI environment
This is the shortest-lived environment; it lives with In this environment, you should be using mock
the build. It gets created when a build gets triggered external services and databases to keep things
and torn down once the build is done. It’s also the running fast.
most unstable environment. Our developers check
We suggest automating the lifetime of the CI
code into the CI environment. Since other developers
environment like this: as soon as you merge the
could be deploying at the same time, the CI
code, the CI environment automatically spins up,
environment has a lot of deployment activities that
runs that code, tells you whether it’s safe or not, and
are happening concurrently. As a result, the CI can,
then tears itself down. Using Docker to automatically
and often does, break. That’s ok–it’s meant to break.
spin up the environment will save time, and the
The key is fixing it when it does.
whole process of automated builds and environment
If you were unable to spin up your entire application creation will make it easier for team members to
in a local environment (which is highly likely), the CI commit more often.
environment will be the first in which you’ll be able
to do browser-driven testing. You’ll also be able to
do any UI testing that you weren’t able to do in your
local environment.

Software Testing for DevOps-Driven Teams • 12

Development environment
The development environment is a shared now required since it’s a fast moving environment
environment with other developers. In this with many different components. Since it sits in
environment, every service within the application is the middle of the path to production, when this
getting deployed every time. These environments are environment breaks, it has a detrimental effect on
very unstable because there are constant changes those that follow - a failure here blocks all changes
from different teams. It’s important to note here that and no code can move forward if the environment
your integration and browser-based tests may now breaks.
fail, even if they passed in the CI environment. That’s
In the development environment, some external
because they are now fully integrated with outside
services are mocked and some are not, depending
services and other services are also currently in
on how crucial each service is to what you’re testing
development.
and also the cost of connecting to that external
Whereas the CI environment is just for your team service. If you are using mocked data here, make
(or for one service of your product) and can be torn sure that a decent amount of test data available.
down between builds, this development environment
Everyone in the organization has visibility into this
is for your entire product codebase. If you choose
environment; any developer can log in and run it as
to merge into a development environment for just
an application. This environment can be used by all
your team, that will have a smaller scope of impact
the developers for testing and debugging.
than merging into the full development environment
shared by all teams. In the development
environment, system health check monitoring is

Software Testing for DevOps-Driven Teams • 13

QA environment
This is the first manually deployed environment In this environment, we now have the same
in this scenario. It is manually deployed because infrastructure and application as in production. We
the QA team needs to decide which features are are using a representative subset of production
worth testing on their own, based on the structure data–close enough to production data to test.
of the changes. They might take a stacked change The QA engineers or testers understand what they
(Change A, B and C) and test them each separately. should focus their tests on. We recommend manual
In this scenario, they test Change A, and once deployment at this stage so that small changes can
they have assurance that it’s working as expected, be tested in an isolated environment to help identify
they can move on and test Change B and when it bugs. The closer the QA environment can get to
throws an error, they know that the issue is isolated production, the higher confidence you will have in
to Change B. Otherwise, in the case of just testing the results of the tests.
Change C, if it were to throw an error, it would block
progress on C, B, and A.

The QA environment is a controlled and integrated

environment. Here, the QA team is controlling what
change is coming in; in contrast, in the development
environment, any change can happen at any time.
The build is now integrated with the services it will
interact with in the application.

Software Testing for DevOps-Driven Teams • 14

Staging environment
This is the last environment before production. The This is the last gate before production to test the
purpose of staging is to have an environment almost implementation, migration, configuration and
exactly the same as production. When you deploy business requirements, so we strongly encourage
something into staging, and it works, you can be you to get business signoff before you get to
reasonably assured that that version won’t fail in staging. Otherwise, it will be very expensive to make
production and cause an outage. All environments a change. While you’re doing development, ensure
help you catch potential issues; staging is the final that whoever owns the product or requested feature
check of confidence. Here it is important to have thoroughly understands what it is you’re building. Be
almost the same amount of data as you would in on the same page the whole way through.
production. This enables you to do load testing, and
Every developer who is implementing code in their
test the scalability of the application in production.
local environment should be able to have visibility
Production-ready code should be deployed to into how it will be deployed into production as well.
this environment; again, almost the same as in Only by being aware of how their code is being
production. Infrastructure, databases, and external deployed can developers make the greatest impact
service integration should be exactly the same as in reducing errors before they move onto the later
in production. It will be expensive to maintain and stages.
build – the only thing more expensive is not doing it,
and breaking production. The scale can be smaller,
but your setup and your configuration has to be the
same.

Software Testing for DevOps-Driven Teams • 15

How to Test Software, Part I:
Mocking, Stubbing, and Contract Testing
In this section, we’ll cover the techniques of mocking As we mentioned, unit or component tests (shown
and stubbing, and contract testing to help each at the bottom of our pyramid) are inexpensive and
testing layer. Refer back to the test pyramid above, fast to perform. Rely heavily on these. Only once
which helps illustrate the difference between types you’ve exhausted what these tests can do, move on
of tests and when it’s advantageous to do each. to more time- and resource-intensive tests, such as
integration and UI layer tests.

Software Testing for DevOps-Driven Teams • 16

Mocking and stubbing Mocking and Stubbing in unit +
A lot of people think that mocking and stubbing are component tests
used just for unit and component tests. However, we
want to show you how mock objects or stubs can be
used in other layers of testing as well. Mocking of external functionality
Let’s start with some definitions. We recommend mocking or stubbing when your
code uses external dependencies like system calls,
Mocking means creating a fake version of an or accessing a database. For example, whenever
external or internal service that can stand in for the you run a test, you’re exercising the implementation.
real one, helping your tests run more quickly and So when a delete or create function happens, you’re
reliably. When your implementation interacts with letting it create a file, or delete a file. This work is
an object’s properties, rather than its function or not efficient, and the data it creates and deletes
behavior, a mock can be used. is not actually useful. Furthermore, it’s expensive
Stubbing, like mocking, means creating a stand-in, to clean up, because now you have to manually
but a stub only mocks the behavior, not the entire delete something every time. This is a case where
object. This is used when your implementation only mocking/stubbing can help a lot.
interacts with a certain behavior of the object. For Using mocks and stubs to fake the external
a more in depth look at the differences between functionality help you create tests that are
mocking and stubbing, check out Martin Fowler’s independent. For instance, say that the test writes a
post “Mocks Aren’t Stubs.” Let’s discuss how we file to /tmp/test_file.txt and then the system under
can apply these methods to improve our testing in the test deletes it. The problem then is not that the
all levels of the pyramid above.

Software Testing for DevOps-Driven Teams • 17

test is not independent; it is that the system calls The code above interacts with Python’s built-in
take a lot of time. In this instance, you can stub the open function which interacts with a system call to
file system call’s response, which will take a lot less actually look for the file from the given file path. This
time because it immediately returns. means that wherever and whenever you run the test
for that function:
Another benefit is that you can reproduce complex
scenarios more easily. For instance, it is much 1. Y
 ou will need to ensure that the file that the test
easier to test the many error responses you might will be looking for exists; when it does not exist,
get from the filesystem then to actually create the test fails.
the condition. Say that you only wanted to delete
corrupt files. Writing a corrupt file can be difficult 2. T
 he test will need to wait for the system call’s
programmatically, but returning the error code response; if the system call times out, the test
associated with a corrupt file is a matter of just fails.
changing what a stub returns.
Neither case of failure means your implementation
See this example code: failed to do its job. These tests are now neither
isolated (since they’re dependent on the system
def read_and_trim(file_path) call’s response) nor efficient (since the system call
return os.open(file_path).rstrip(“\n”) connection will take time to deliver the request and
response).

This method will call system call to look for the file from the
given file path and read the content from them and removing
new line terminator.

Software Testing for DevOps-Driven Teams • 18

The test code for the implementation above looks like this:

from unittest.mock import patch

content = ”fake file content\n”

trimed_content = content.rstrip(“\n”)
@patch(“builtins.open”, new_callable=mock_open, read_data=content)
def test_read_trim_content(self, mock_object):
file_path = “/fake/file/path”
self.assertEqual(read_and_trim(file_path), trimed_content)
mock_object.assert_called_with(file_path)

We are using a Python mock patch to mock the have to manually create the entity. Now the test
built-in open call. In this way, we are only testing fails. There was no file to delete since they didn’t
what we actually built. know they had to create the entity, so this is not an
independent test.
Another good example of using mocks and stubs in
unit testing is faking database calls. For example, In cases like these, you’ll want to prevent modifying
let’s say you are testing whether your function the data or making operating system calls to remove
deletes the entity from a database. For the first test, the file. This will prevent tests from being flaky
you manually create a file so that there’s one to be whenever someone accidentally fails to create test
deleted. The test passes. But then, the second time, data.
someone else (who isn’t you) doesn’t know that they

Software Testing for DevOps-Driven Teams • 19

Mocking and stubbing of internal create a lot of potential failure points from services
functions you do not control, adding time and complexity to
your testing. Try narrowing it down by writing a few
Mocks and stubs are very handy for unit tests. They service integration tests using mocks and stubs,
help you to test a functionality or implementation which will make your test suite more reliable.
independently, while also allowing unit tests to
remain efficient and cheap. In integration testing, the rules are different from unit
tests. Here, you should only test the implementation
A great application of mocks and stubs in a unit/ and functionality that you have the control to edit.
component test is when your implementation Mocks and stubs can be used for this purpose. First,
interacts with another method or class. You can identify which integrations are important. Then, you
mock the class object or stub the method behavior can decide which external or internal services can be
that your implementation is interacting with. mocked.
Mocking or stubbing the other functionality or class,
and therefore only testing your implementation logic, Let’s say your code interacts with the GitHub API,
is the key benefit of unit tests, and the way to reap like in the example below. Since you personally can’t
the biggest benefit from performing them. change how the GitHub API is responding from your
request call, you don’t have to test it. Mocking the
expected GitHub API’s response lets you focus more
Mocking in integration testing on testing the interactions within your internal code
base.
With integration tests, you are testing relationships
between services. One approach might be to get
all the dependent services up and running for the
testing environment. But this is unnecessary. It can

Software Testing for DevOps-Driven Teams • 20

 @unittest.mock.patch(‘Github’)
def test_parsed_content_from_git(self,
mocked_git):
expected_decoded_content = “b’# Sample Hello World\n\n> How to run this app\n\n- installation\n\n dependencies\n”
mocked_git.get_repo.return_value =
expected_decoded_content
parsed_content = read_parse_from content(repo=’my/repo’,
file_to_read=’README.md’)
self.assertEqual(parsed_content[‘titles’], [‘Sample Hello World’])

In the test code above, the read_parse_from_content For example, if the expected_decoded_content in
method is integrated with the class that parses the the code example above is not how GitHub returns
JSON object from the GitHub API call. In this test, we the repo file content, incorrect assumptions from the
are testing the integration between two classes. mocked test can lead to unexpected breakage. Before
writing the test that will have the mocked response,
Since we are using a mock in the test above, your it’s best to make the actual snapshot of the external
test will be faster and less dependent by avoiding dependency call and use it as a mocked response.
making the call to the GitHub API. This will also save Once you have created the mocked response with
time and effort by not needing internet access for the the snapshot, that should not change often since
environment that will run the test. However, in order the Application Programming Interface should
for you to have reliable testing while mocking the almost always be backward compatible. However,
dependent external services, it’s extremely important it is important to validate the API regularly for the
for you to understand how external dependencies will occasional unexpected change.
behave in the real world.

Software Testing for DevOps-Driven Teams • 21

Software Testing for DevOps-Driven Teams • 22
Mocks and stubs in contract-based
testing (in a microservices architecture)
When two different services integrate with each
You can use the idea of contracts to test internal
other, they each have “expectations,” i.e. standards
services as well. When testing a large scale
about what they’re giving and what they expect to
application using microservices architecture it
get in return. We can think of these as contracts
could be costly to install the entire system and
between integrated endpoints. Because of this
infrastructure. Such applications can benefit greatly
standardization, contract tests can be used to test
from using contract testing. In the testing pyramid,
integrations.
contract testing sits in between the unit/component
Let’s walk through an example. The version-tagged testing and integration testing layers, depending on
API should not change often, possibly not ever. For the coverage of the contract testing in your system.
any API you choose, you will generally be able to Some organizations utilize contract testing to
find documentation about that API, including what completely replace end-to-end or functional testing.
to expect from it. When you decide to use a certain
version of an API, you can rely on the return of that
API call. This is the presumed contract between the
engineers who provide the API and the engineers
who will use its data.

Software Testing for DevOps-Driven Teams • 23

Contract-based testing can cover two important response. Since there is a contract between two
things: services, the endpoint and response should not
change. This will free both services from depending
1. C
 hecking the connectivity of endpoint that has on each other during tests, allowing tests to be
been agreed upon faster and more reliable.
2. C
 hecking the response from the endpoint with a It can be useful to run the same test in a different
given argument environment with a different configuration. Contract
tests are one of the great examples of the latter
As an example, let’s imagine a weather-reporting
case. We can achieve different goals when running
application involving a weather service interacting
contract tests in different environments with
with a user service. When the user service connects
different configurations. When it’s a lower layer
to the endpoint of the weather service with the date
environment such as Dev or CI, running the test
(the request), the user service processes the date
with a mocked contract would serve the purpose
data to get the weather for that date. These two
of testing our internal implementation within the
services have a contract: the weather service will
constraints of the environment. However, when it
maintain the endpoint to be always accessible by the
goes to an upper layer environment such as QA
user service and provide the valid data that the user
or Staging, the same test can be used without
service is requesting, and in the same format.
a mocked contract but with the actual external
Now, let’s take a look at how we can utilize mocks dependency connection. Mbtest is one tool that can
and stubs in the contract test. Instead of the help with the kind of contract testing and mocking
user service making the actual request call to the response explained above.
weather service in the test, you can create a mocked

Software Testing for DevOps-Driven Teams • 24

 2. You have the flexibility to scope the test to cover
just the parts you can control and change. With
external services, you are powerless in the case
that they’re wrong or the test fails. Mocking
ensures you are scoping tests for work that you
can actually do and not giving yourself problems
you can’t fix.

3. Mocking external API calls helps your test to be

more reliable

4. Contract testing empowers service teams to be

more autonomous in development

Next up, we’ll explore the principles of test-

driven development (TDD) and behavior-driven
We’ve taken a look at examples of different layers of development (BDD), and see how they can improve
testing using mocks and stubs. Now let’s recap why outcomes for everything from functional testing to
they are useful: unit testing.

1. T
 ests with mocks and stubs go faster because
you don’t have to connect with external services.
There’s no delay waiting for them to respond.

Software Testing for DevOps-Driven Teams • 25

How to Test Software, Part II:
TDD and BDD
So far, we’ve discussed what mocks and stubs TDD: test-driven development
are, as well as how to use them in various testing
scenarios to give yourself more flexibility, speed up TDD (test-driven development) is known as a
your tests, and get more determinism out of your method for writing unit tests. We are going to talk
test suite. about using TDD principles for everything from
functional testing to unit testing.
Now we’re going to cover two methods for software
development that take testing into consideration
at the outset: test-driven development (TDD) and
behavior-driven development (BDD). Using these Red-Green Refactor: test-driven
methodologies will improve the way you think about development principles
software development, and greatly enhance the
With TDD, you design your code before you
efficacy of your tests. Let’s dive in:
implement it. Therefore, TDD forces you to think
about your components’ behavior before you write
it. It’s also a great way to keep you focused on what
you are trying to deliver. In TDD, you write tests for
your method or implementation to test what that
implementation should do.

Software Testing for DevOps-Driven Teams • 26

Remember, with TDD, your test will always fail first.
You haven’t written the code yet so there is no
functionality. This is a good thing! It proves that your
test won’t just pass any old implementation!

Next, it’s time for you to prove that your test will
pass when the implementation is valid and it serves
its purpose. Once you check that your test fails
when the implementation doesn’t work correctly
and passes when implementation does function can feel confident it’s deterministic later on when
correctly, you can refactor your code to be better things get more complex. Later, in the green stage,
and clearer. Since you already have the test right your focus isn’t “How can I write the best code?”
there, refactoring will be much easier and you but rather “How can I write code that meets the
can do it with the assurance that your tests will requirements?” Think of this as the stage to prove
tell you whether you changed the code’s behavior that your test is passing for the valid use cases.
successfully. This is called red-green refactoring.
In the next stage, refactoring, you will have a change
Red: First, you make your test fail. Green: Then, to revisit your code. The refactoring stage is when
make it pass. you can write cleaner, more intelligent code, and
make improvements. Often, engineers start writing
Test first, refactor after. This ensures that your in the beginning and lose focus of what they were
code is clean and production-ready. It’s important to supposed to deliver. Other times, engineers will
actually go through red and green before you make write tests at the same time as they are creating the
your code perfect. The red stage will verify that your implementation and create unexpected bugs. TDD
test is not just going to pass all the time. You helps avoid those mistakes.

Software Testing for DevOps-Driven Teams • 27

The test you write might look like this:

describe(‘sum()’, function () {

it(‘should return the sum of given numbers’, function () {

expect(simpleCalculator.sum(1,2)).to.equal(3);
expect(simpleCalculator.sum(5,5)).to.equal(10);
});
})

1. Red: Your implementation is currently empty. You haven’t implemented yet, so the tests will fail. You want to
verify that your test is deterministic: it will tell you when it should fail or pass.

var Calculator = function () {

return true // implementation goes here
}

Software Testing for DevOps-Driven Teams • 28

2. Green: Implement it. Make the test pass. Here we’ll write the code that will make the test pass.

var Calculator = function () {

return{
sum: function(number1, number2){
return number1 + number2;
}
};
}

Now your test will be satisfied, because we’ve added the likelihood of a functionality failure in your source
the function. code. If you need to change the tests, make sure to
do so in the red/green stage.
Refactoring code: Now, refactor your code to be
clearer and more readable. The first two steps Once you have a valid test, you can refactor the code
made it so that your test is reliable and you to be cleaner and more aligned with the style or
don’t have to worry about modifying your code’s overall class.
behavior accidentally. Remember, you verified the
functionality of your code with the test that went
through the red and green stages. Once the code
is in the refactoring stage, the tests should not be
changed. If you make a change now, you increase

Software Testing for DevOps-Driven Teams • 29

Putting it all together From there, we start working on unit/component
testing or integration testing, depending on the work
The example above is for a unit test. But how can we itself. If the architectural design is clear before we
use TDD on other layers of the test pyramid? dig into the code base, we start writing integration
tests. These will also fail for a while. While they may
not be complete at the moment you write them, they
still serve the function of helping you think about
what you are trying to build and what the initial
design is.

When we move onto the unit/component test, we

finally start red-green refactoring in the unit test
layer, leaving UI and integration layers in ‘red’ stage
and completing the unit tests to the refactoring
stage. Then we move back to the integration tests
and make the text green, then refactor. Afterward,
When we make an implementation, we’ll start with the same step applies to the UI testing
UI layer testing first (using BDD, which we’ll explain in
As you can see, we are applying the TDD principles
the next section). Even though these UI tests will not
throughout all the layers of testing. The principle is
pass for a long time, starting from the tests helps
the same, the only difference is the scale.
us focus on what we are actually trying to build and
how the backend code will interact with the frontend
layer. This approach allows developers to design
their implementation before they write it.

Software Testing for DevOps-Driven Teams • 30

BDD: behavior-driven The user journey story represents a user’s behavior.
Using the business requirements provided, the
development developer can think about scenarios of how a user
will use this new functionality. And those scenarios
User Journey Story and Given, When can be used to write the tests. This is called
and Then behavior-driven development (BDD).

Any time there is a new feature request, people from BDD is a widely-used method in UI-driven testing. It
the product side of the business write story-level is written in a structure known as “Given, When and
tasks for engineers, including user story and user Then.”
acceptance criteria. This way, you as an engineer Given: the state of the system that will receive the
can understand the value to the business and think behavior/action
from the user’s perspective about the functionality
that they will implement. By seeing user stories, When: the behavior/action that happens and causes
engineers can also better understand the scope of the result in the end
the work.
Then: the result caused by the behavior in the state
User-acceptance testing (UI-driven testing) builds
on this user acceptance criteria and user story. UI- It’s a good idea to think about the user journey and
driven testing usually uses tools like Selenium or user’s behavior first, so that when you implement
Cucumber which help test against the user’s journey your feature, it is with consideration of how the user
on a site that’s up and running. will interact with it.

Software Testing for DevOps-Driven Teams • 31

Here is an example: Here are some simple test code examples with
Cypress (for an easy integration with this tool,
Scenario: the user signs up to the site
explore the Cypress orb):
Given: the user visited the site
When: the user clicked the signup button

Then: ensure the user can access the signup page

describe(‘User can signup to the test-example site’, function() {

it(‘clicking “signup” navigate to a signup url’, function() {

// Given
cy.visit(‘https://test-example.com/’)
// When
cy.contains(‘signup’).click()
//Then
cy.url().should(‘include’, ‘/signup’)
})
})

Using BDD in UI layer testing make sense since it involves the part of the application that the user will interact
with. Other layers of testing won’t be as well-suited to using BDD. While UI layer testing with BDD is invaluable to
the process of building quality software, it’s very expensive and inefficient.

Software Testing for DevOps-Driven Teams • 32

As we mentioned earlier, utilizing different layers what is failing and be able to fix it more quickly. This
and kinds of testing means that when something reduces debugging time and enables you to detect
goes wrong, it will be way faster to pinpoint exactly low-level problems much more cheaply and quickly.

Software Testing for DevOps-Driven Teams • 33

Conclusion: happy path and edge cases
When you write tests, it’s easy to think about what When engineers understand the system and
will happen when everything goes well. It can be a circumstances better, it becomes easier to think
challenge for engineers to think about edge cases. about edge cases. This results in the tests in the
That is expected. edge cases being covered better by the lower layer
of the test pyramid - which is always preferable, as
When we think about UI layer testing, we are they’re more efficient. That said, maintaining tests
assuming that (almost) the entire site is already are also part of an engineer’s job. When your code
up and running and your tests are running against evolves, your tests need to be changed as well.
them. Now, it’s hard to imagine every single Having meaningful and behavior-driven tests is more
pathway that a user may take, and it would be very important than the number of tests that you have.
expensive to test every single pathway that could The purpose of testing, after all, is to deliver quality
possibly occur. Therefore, it’s a good practice to software to production.
focus on the happy path and major failure path:
these will cover both the main behaviors and the Making your code base more testable is a
worst-case scenario. Often the edge case bugs are worthwhile investment, and it will help you scale
discovered from QA exploratory testing in a QA- your business and software in the long term. This
like environment. In this process, QA will analyze fundamental work will allow you to optimize your
the business risk and communicate the edge case software’s path to production, giving you more
scenario to the engineers to ensure that they fix the confidence every time you deploy.
bugs before the code goes to production, and write
new tests to cover the edge case scenarios.

Software Testing for DevOps-Driven Teams • 34

Conclusion Learning when and how to use DevOps, and
mastering the ins and outs of testing, goes a long
way toward ensuring that you’re delivering the
highest quality software possible. It also reduces
wasted time and resources, maximizing both
efficiency and output. To learn more about how to
make your software delivery faster and more robust,
visit circleci.com.

Software Testing for DevOps-Driven Teams • 35