Infrastructure as Code

e d
v
July 2017

h i
This paper has been archived.

rc
For the latest technical content about the AWS Cloud, see the

A
AWS Whitepapers & Guides page:
[Link]
© 2017, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Notices
This document is provided for informational purposes only. It represents AWS’s
current product offerings and practices as of the date of issue of this document,
which are subject to change without notice. Customers are responsible for
making their own independent assessment of the information in this document
and any use of AWS’s products or services, each of which is provided “as is”

d
without warranty of any kind, whether express or implied. This document does
not create any warranties, representations, contractual commitments,

e
conditions or assurances from AWS, its affiliates, suppliers or licensors. The
responsibilities and liabilities of AWS to its customers are controlled by AWS

v
agreements, and this document is not part of, nor does it modify, any agreement

i
between AWS and its customers.

rc h
A
Contents
Introduction to Infrastructure as Code 1
The Infrastructure Resource Lifecycle 1
Resource Provisioning 3
AWS CloudFormation 4
Summary 9

d
Configuration Management 10
Amazon EC2 Systems Manager 10

e
AWS OpsWorks for Chef Automate 14

v
Summary 17

i
Monitoring and Performance 18

h
Amazon CloudWatch 18
Summary 21

c
Governance and Compliance 21

r
AWS Config 22
AWS Config Rules 23

A
Summary 25
Resource Optimization 25
AWS Trusted Advisor 26
Summary 27
Next Steps 28
Conclusion 28
Contributors 30
Resources 30
Abstract
Infrastructure as Code has emerged as a best practice for automating the
provisioning of infrastructure services. This paper describes the benefits of
Infrastructure as Code, and how to leverage the capabilities of Amazon Web
Services in this realm to support DevOps initiatives.

DevOps is the combination of cultural philosophies, practices, and tools that

increases your organization’s ability to deliver applications and services at high

d
velocity. This enables your organization to be more responsive to the needs of
your customers. The practice of Infrastructure as Code can be a catalyst that

e
makes attaining such a velocity possible.

h i v
rc
A
Amazon Web Services – Infrastructure as Code

Introduction to Infrastructure as Code

Infrastructure management is a process associated with software engineering.
Organizations have traditionally “racked and stacked” hardware, and then
installed and configured operating systems and applications to support their
technology needs. Cloud computing takes advantage of virtualization to enable
the on-demand provisioning of compute, network, and storage resources that
constitute technology infrastructures.

d
Infrastructure managers have often performed such provisioning manually. The
manual processes have certain disadvantages, including:

e
• Higher cost because they require human capital that could otherwise go

v
toward more important business needs.

i
• Inconsistency due to human error, leading to deviations from
configuration standards.

h
• Lack of agility by limiting the speed at which your organization can
release new versions of services in response to customer needs and

c
market drivers.

r
• Difficulty in attaining and maintaining compliance to corporate or
industry standards due to the absence of repeatable processes.

A
Infrastructure as Code addresses these deficiencies by bringing automation to
the provisioning process. Rather than relying on manually performed steps,
both administrators and developers can instantiate infrastructure using
configuration files. Infrastructure as Code treats these configuration files as
software code. These files can be used to produce a set of artifacts, namely the
compute, storage, network, and application services that comprise an operating
environment. Infrastructure as Code eliminates configuration drift through
automation, thereby increasing the speed and agility of infrastructure
deployments.

The Infrastructure Resource Lifecycle

In the previous section, we presented Infrastructure as Code as a way of
provisioning resources in a repeatable and consistent manner. The underlying
concepts are also relevant to the broader roles of infrastructure technology
operations. Consider the following diagram.

Page 1
Amazon Web Services – Infrastructure as Code

e d
h i v
c
Figure 1: Infrastructure resource lifecycle

r
Figure 1 illustrates a common view of the lifecycle of infrastructure resources in
an organization. The stages of the lifecycle are as follows:

A
1. Resource provisioning. Administrators provision the resources
according to the specifications they want.

2. Configuration management. The resources become components

of a configuration management system that supports activities such
as tuning and patching.

3. Monitoring and performance. Monitoring and performance

tools validate the operational status of the resources by examining
items such as metrics, synthetic transactions, and log files.

4. Compliance and governance. Compliance and governance

frameworks drive additional validation to ensure alignment with
corporate and industry standards, as well as regulatory
requirements.

Page 2
Amazon Web Services – Infrastructure as Code

5. Resource optimization. Administrators review performance data

and identify changes needed to optimize the environment around
criteria such as performance and cost management.

Each stage involves procedures that can leverage code. This extends the benefits
of Infrastructure as Code from its traditional role in provisioning to the entire
resource lifecycle. Every lifecycle then benefits from the consistency and
repeatability that Infrastructure as Code offers. This expanded view of
Infrastructure as Code results in a higher degree of maturity in the Information
Technology (IT) organization as a whole.

d
In the following sections, we explore each stage of the lifecycle – provisioning,

e
configuration management, monitoring and performance, governance and
compliance, and optimization. We will consider the various tasks associated

v
with each stage and discuss how to accomplish those tasks using the capabilities

i
of Amazon Web Services (AWS).

h
Resource Provisioning

c
The information resource lifecycle begins with resource provisioning.
Administrators can use the principle of Infrastructure as Code to streamline the

r
provisioning process. Consider the following situations:

A
• A release manager needs to build a replica of a cloud-based production
environment for disaster recovery purposes. The administrator designs a
template that models the production environment and provisions
identical infrastructure in the disaster recovery location.

• A university professor wants to provision resources for classes each

semester. The students in the class need an environment that contains
the appropriate tools for their studies. The professor creates a template
with the appropriate infrastructure components, and then instantiates
the template resources for each student as needed.

• A service that has to meet certain industry protection standards requires

infrastructure with a set of security controls each time the service is
installed. The security administrator integrates the security controls into
the configuration template so that the security controls are instantiated
with the infrastructure.

Page 3
Amazon Web Services – Infrastructure as Code

• The manager of a software project team needs to provide development

environments for programmers that include the necessary tools and the
ability to interface with a continuous integration platform. The manager
creates a template of the resources and publishes the template in a
resource catalog. This enables the team members to provision their own
environments as needed.

These situations have one thing in common: the need for a repeatable process
for instantiating resources consistently. Infrastructure as Code provides the
framework for such a process. To address this need, AWS offers AWS

d
CloudFormation. 1

e
AWS CloudFormation

v
AWS CloudFormation gives developers and systems administrators an easy way

i
to create, manage, provision, and update a collection of related AWS resources
in an orderly and predictable way. AWS CloudFormation uses templates written
in JSON or YAML format to describe the collection of AWS resources (known as

h
a stack), their associated dependencies, and any required runtime parameters.
You can use a template repeatedly to create identical copies of the same stack

c
consistently across AWS Regions. After deploying the resources, you can modify

r
and update them in a controlled and predictable way. In effect, you are applying
version control to your AWS infrastructure the same way you do with your
application code.

A
Template Anatomy
Figure 2 shows a basic AWS CloudFormation YAML-formatted template
fragment. Templates contain parameters, resource declaration, and outputs.
Templates can reference the outputs of other templates, which enables
modularization.

Page 4
Amazon Web Services – Infrastructure as Code

---
AWSTemplateFormatVersion: "version date"

Description:
String

Parameters:
set of parameters

Mappings:

d
set of mappings

e
Conditions:
set of conditions

v
Transform:

i
set of transforms

Resources:

h
set of resources

c
Outputs:

r
set of outputs

Figure 2: Structure of an AWS CloudFormation YAML template

A
Figure 3 is an example of an AWS CloudFormation template. The template
requests the name of an Amazon Elastic Compute Cloud (EC2) key pair from the
user in the parameters section. 2 The resources section of the template then
creates an EC2 instance using that key pair, with an EC2 security group that
enables HTTP (TCP port 80) access.

Page 5
Amazon Web Services – Infrastructure as Code

Parameters:
KeyName:
Description: The EC2 key pair to allow SSH access to the
instance
Type: AWS::EC2::KeyPair::KeyName
Resources:
Ec2Instance:
Type: AWS::EC2::Instance
Properties:
SecurityGroups: !Ref InstanceSecurityGroup

d
KeyName: !Ref KeyName
ImageId: ami-70065467

e
InstanceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:

v
GroupDescription: Enable HTTP access via port 80

i
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '80'

h
ToPort: '80'
CidrIp: [Link]/0

rc
Figure 3: Example of an AWS CloudFormation YAML template

A
Change Sets
You can update AWS CloudFormation templates with application source code to
add, modify, or delete stack resources. The change sets feature enables you to
preview proposed changes to a stack without performing the associated
updates. 3 You can control the ability to create and view change sets using AWS
Identity and Access Management (IAM). 4 You can allow some developers to
create and preview change sets, while reserving the ability to update stacks or
execute change sets to a select few. For example, you could allow a developer to
see the impact of a template change before promoting that change to the testing
stage.

There are three primary phases associated with the use of change sets.

1. Create the change set.

Page 6
Amazon Web Services – Infrastructure as Code

To create a change set for a stack, submit the changes to the template or
parameters to AWS CloudFormation. AWS CloudFormation generates a
change set by comparing the current stack with your changes.

2. View the change set.

You can use the AWS CloudFormation console, AWS CLI, or AWS
CloudFormation API to view change sets. The AWS CloudFormation
console provides a summary of the changes and a detailed list of changes
in JSON format. The AWS CLI and AWS CloudFormation API return a

d
detailed list of changes in JSON format.

3. Execute the change set.

e
You can select and execute the change set in the AWS CloudFormation
console, use the aws cloudformation execute-change-set

v
command in the AWS CLI, or the ExecuteChangeSet API.

i
The change sets capability allows you to go beyond version control in AWS
CloudFormation by enabling you to keep track of what will actually change from

h
one version to the next. Developers and administrators can gain more insight
into the impact of changes before promoting them and minimize the risk of

c
introducing errors.

r
Reusable Templates

A
Many programming languages offer ways to modularize code with constructs
such as functions and subroutines. Similarly, AWS CloudFormation offers
multiple ways to manage and organize your stacks. Although you can maintain
all your resources within a single stack, large single-stack templates can become
difficult to manage. There is also a greater possibility of encountering a number
of AWS CloudFormation limits. 5

When designing the architecture of your AWS CloudFormation stacks, you can
group the stacks logically by function. Instead of creating a single template that
includes all the resources you need, such as virtual private clouds (VPCs),
subnets, and security groups, you can use nested stacks or cross-stack
references. 6, 7

The nested stack feature allows you to create a new AWS CloudFormation stack
resource within an AWS CloudFormation template and establish a parent-child
relationship between the two stacks. Each time you create an AWS

Page 7
Amazon Web Services – Infrastructure as Code

CloudFormation stack from the parent template, AWS CloudFormation also

creates a new child stack. This approach allows you to share infrastructure code
across projects while maintaining completely separate stacks for each project.

Cross-stack references enable an AWS CloudFormation stack to export values

that other AWS CloudFormation stacks can then import. Cross-stack references
promote a service-oriented model with loose coupling that allows you to share a
single set of resources across multiple projects.

d
Template Linting
As with application code, AWS CloudFormation templates should go through

e
some form of static analysis, also known as linting. The goal of linting is to
determine whether the code is syntactically correct, identify potential errors,

v
and evaluate adherence to specific style guidelines. In AWS CloudFormation,

i
linting validates that a template is correctly written in either JSON or YAML.

AWS CloudFormation provides the ValidateTemplate API that checks for

h
proper JSON or YAML syntax. 8 If the check fails, AWS CloudFormation returns
a template validation error. For example, you can run the following command to

c
validate a template stored in Amazon Simple Storage Service (Amazon S3): 9

r
aws cloudformation validate-template --template-url \
s3://examplebucket/example_template.template

A
You can also use third-party validation tools. For example, cfn-nag performs
additional evaluations on templates to look for potential security concerns.
Another tool, cfn-check, performs deeper checks on resource specifications to
identify potential errors before they emerge during stack creation. 10, 11

Best Practices
The AWS CloudFormation User Guide provides a list of best practices for
designing and implementing AWS CloudFormation templates. 12 We provide
links to these practices below.

Planning and organizing

• Organize Your Stacks By Lifecycle and Ownership 13

• Use IAM to Control Access 14

Page 8
Amazon Web Services – Infrastructure as Code

• Reuse Templates to Replicate Stacks in Multiple Environments 15

• Use Nested Stacks to Reuse Common Template Patterns 16

• Use Cross-Stack References to Export Shared Resources 17

Creating templates

• Do Not Embed Credentials in Your Templates 18

• Use AWS-Specific Parameter Types 19

d
• Use Parameter Constraints 20

• Use AWS::CloudFormation::Init to Deploy Software Applications on

e
Amazon EC2 Instances 21

v
• Use the Latest Helper Scripts 22

i
• Validate Templates Before Using Them 23

• Use Parameter Store to Centrally Manage Parameters in Your

h
Templates 24

c
Managing stacks

r
• Manage All Stack Resources Through AWS CloudFormation 25
• Create Change Sets Before Updating Your Stacks 26

A
• Use Stack Policies 27

• Use AWS CloudTrail to Log AWS CloudFormation Calls 28

• Use Code Reviews and Revision Controls to Manage Your Templates 29

• Update Your Amazon EC2 Linux Instances Regularly 30

Summary
The information resource lifecycle starts with the provisioning of resources.
AWS CloudFormation provides a template-based way of creating infrastructure
and managing the dependencies between resources during the creation process.
With AWS CloudFormation, you can maintain your infrastructure just like
application source code.

Page 9
Amazon Web Services – Infrastructure as Code

Configuration Management
Once you provision your infrastructure resources and that infrastructure is up
and running, you must address the ongoing configuration management needs of
the environment. Consider the following situations:

• A release manager wants to deploy a version of an application across a

group of servers and perform a rollback if there are problems.

• A system administrator receives a request to install a new operating

d
system package in developer environments, but leave the other
environments untouched.

e
• An application administrator needs to periodically update a

v
configuration file across all servers housing an application.

i
One way to address these situations is to return to the provisioning stage,
provision fresh resources with the required changes, and dispose of the old

h
resources. This approach, also known as infrastructure immutability, ensures
that the provisioned resources are built anew according to the code baseline

c
each time a change is made. This eliminates configuration drift.

r
There are times, however, when you might want to take a different approach. In
environments that have high levels of durability, it might be preferable to have

A
ways to make incremental changes to the current resources instead of
reprovisioning them. To address this need, AWS offers Amazon EC2 Systems
Manager and AWS OpsWorks for Chef Automate. 31, 32

Amazon EC2 Systems Manager

Amazon EC2 Systems Manager is a collection of capabilities that simplifies
common maintenance, management, deployment, and execution of operational
tasks on EC2 instances and servers or virtual machines (VMs) in on-premises
environments. Systems Manager helps you easily understand and control the
current state of your EC2 instance and OS configurations. You can track and
remotely manage system configuration, OS patch levels, application
configurations, and other details about deployments as they occur over time.
These capabilities help with automating complex and repetitive tasks, defining
system configurations, preventing drift, and maintaining software compliance
of both Amazon EC2 and on-premises configurations.

Page 10
Amazon Web Services – Infrastructure as Code

Table 1 lists the tasks that Systems Manager simplifies.

Tasks Details

Run Command 33 Manage the configuration of managed instances at scale by distributing

commands across a fleet.

Inventory 34 Automate the collection of the software inventory from managed instances.

State Manager 35 Keep managed instances in a defined and consistent state.

Maintenance Define a maintenance window for running administrative tasks.

Window 36

d
Patch Deploy software patches automatically across groups of instances.
Manager 37

e
Automation 38 Perform common maintenance and deployment tasks, such as updating Amazon
Machine Images (AMIs).

v
Parameter Store, control, access, and retrieve configuration data, whether plain-text data

i
Store 39 such as database strings or secrets such as passwords, encrypted through AWS
Key Management System (KMS).

h
Table 1: Amazon EC2 Systems Manager tasks

c
Document Structure
A Systems Manager document defines the actions that Systems Manager

r
performs on your managed instances. Systems Manager includes more than a
dozen preconfigured documents to support the capabilities listed in Table 1. You

A
can also create custom version-controlled documents to augment the
capabilities of Systems Manager. You can set a default version and share it
across AWS accounts. Steps in the document specify the execution order. All
documents are written in JSON and include both parameters and actions. As
with AWS OpsWorks for Chef Automate, documents for Systems Manager
become part of the infrastructure code base, bringing Infrastructure as Code to
configuration management.

The following is an example of a custom document for a Windows-based host.

The document uses the ipconfig command to gather the network configuration
of the node and then installs MySQL.

{
"schemaVersion": "2.0",
"description": "Sample version 2.0 document v2",
"parameters": {},

Page 11
Amazon Web Services – Infrastructure as Code

"mainSteps": [
{
"action": "aws:runPowerShellScript",
"name": "runShellScript",
"inputs": {
"runCommand": ["ipconfig"]
}
},
{
"action": "aws:applications",

d
"name": "installapp",
"inputs": {

e
"action": "Install",
"source":
"[Link]

v
installer-community-[Link].msi"

i
}
}
]

h
}

c
Figure 4: Example of a Systems Manager document

r
Best Practices

A
The best practices for each of the Systems Manager capabilities appear below.

Run Command

• Improve your security posture by leveraging Run Command to access

your EC2 instances, instead of SSH/RDP. 40

• Audit all API calls made by or on behalf of Run Command using AWS
CloudTrail.

• Use the rate control feature in Run Command to perform a staged

command execution. 41

• Use fine-grained access permissions for Run Command (and all Systems
Manager capabilities) by using AWS Identity and Access Management
(IAM) policies. 42

Page 12
Amazon Web Services – Infrastructure as Code

Inventory

• Use Inventory in combination with AWS Config to audit your

application configuration overtime.

State Manager

• Update the SSM agent periodically (at least once a month) using the
preconfigured AWS-UpdateSSMAgent document. 43

d
• Bootstrap EC2 instances on launch using EC2Config for Windows. 44

• (Specific to Windows) Upload the PowerShell or Desired State

e
Configuration (DSC) module to Amazon S3, and use AWS-
InstallPowerShellModule.

v
• Use tags to create application groups. Then target instances using the

i
Targets parameter, instead of specifying individual instance IDs.
• Automatically remediate findings generated by Amazon Inspector by

h
using Systems Manager. 45

c
• Use a centralized configuration repository for all of your Systems
Manager documents, and share documents across your organization. 46

r
Maintenance Windows

A
• Define a schedule for performing disruptive actions on your instances
such as OS patching, driver updates, or software installs.

Patch Manager

• Use Patch Manager to roll out patches at scale and to increase fleet
compliance visibility across your EC2 instances.

Automation

• Create self-serviceable runbooks for infrastructure as Automation

documents.

• Use Automation to simplify creating AMIs from the AWS Marketplace or

custom AMIs, using public documents, or authoring your own
workflows.

Page 13
Amazon Web Services – Infrastructure as Code

• Use the documents AWS-UpdateLinuxAmi or AWS-

UpdateWindowsAmi or create a custom Automation document to build
and maintain images.

Parameter Store

• Use Parameter Store to manage global configuration settings in a

centralized manner. 47

• Use Parameter Store for secrets managements, encrypted through AWS

d
KMS. 48

• Use Parameter Store with Amazon EC2 Container Service (ECS) task

e
definitions to store secrets. 49

v
AWS OpsWorks for Chef Automate

i
AWS OpsWorks for Chef Automate brings the capabilities of Chef, a
configuration management platform, to AWS. OpsWorks for Chef Automate

h
further builds on Chef’s capabilities by providing additional features that
support DevOps capabilities at scale. Chef is based on the concept of recipes,

c
configuration scripts written in the Ruby language that perform tasks such as
installing services. Chef recipes, like AWS CloudFormation templates, are a

r
form of source code that can be version controlled, thereby extending the
principle of Infrastructure as Code to the configuration management stage of

A
the resource lifecycle.

OpsWorks for Chef Automate expands the capabilities of Chef to enable your
organization to implement DevOps at scale. OpsWorks for Chef Automate
provides three key capabilities that you can configure to support DevOps
practices: workflow, compliance, and visibility.

Workflow
You can use a workflow in OpsWorks for Chef Automate to coordinate
development, test, and deployment. The workflow includes quality gates that
enable users with the appropriate privileges to promote code between phases of
the release management process. This capability can be very useful in
supporting collaboration between teams. Each team can implement its own
gates to ensure compatibility between the projects of each team.

Page 14
Amazon Web Services – Infrastructure as Code

Compliance
OpsWorks for Chef Automate provides features that can assist you with
organizational compliance as part of configuration management. Chef Automate
can provide reports that highlight matters associated with compliance and risk.
You can also leverage profiles from well-known groups such as the Center for
Internet Security (CIS).

Visibility
OpsWorks for Chef Automate provides visibility into the state of workflows and

d
compliance within projects. A Chef user can create and view dashboards that
provide information about related events and query the events through a user

e
interface.

v
Recipe Anatomy

i
A Chef recipe consists of a set of resource definitions. The definitions describe
the desired state of the resources and how Chef can bring them to that state.

h
Chef supports over 60 resource types. A list of common resource types appears
below.

rc
Resource Name Purpose

Bash Execute a script using the bash interpreter

A
Directory Manage directories

Execute Execute a single command

File Manage files

Git Manage source resources in Git repositories

Group Manage groups

Package Manage packages

Route Manage a Linux route table entry

Service Manage a service

User Manage users

Table 2: Common Chef resources

The following is an example of a Chef recipe. This example defines a resource

based on the installation of the Apache web server. The resource definition
includes a check for the underlying operating system. It uses the case operator
to examine the value of node[:platform] to check for the underlying

Page 15
Amazon Web Services – Infrastructure as Code

operating system. The action: install directive brings the resource to the
desired state (that is, it installs the package).

package 'apache2' do
case node[:platform]
when 'centos','redhat','fedora','amazon'
package_name 'httpd'
when 'debian','ubuntu'
package_name 'apache2'

d
end
action :install
end

v e
Figure 5: Example of a Chef recipe

i
Recipe Linting and Testing

h
A variety of tools is available from both Chef and the Chef user community that
support linting (syntax checking) and unit and integration testing. We highlight

c
some of the most common platforms in the following sections.

r
Linting with Rubocop and Foodcritic
Linting can be done on infrastructure code such as Chef recipes using tools such
as Rubocop and Foodcritic. 50, 51, 52 Rubocop performs static analysis on Chef

A
recipes based on the Ruby style guide. (Ruby is the language used to create Chef
recipes.) This tool is part of the Chef Development Kit and can be integrated
into the software development workflow. Foodcritic checks Chef recipes for
common syntax errors based on a set of built-in rules, which can be extended by
community contributions.

Unit Testing with ChefSpec

ChefSpec can provide unit testing on Chef cookbooks. 53 These tests can
determine whether Chef is being asked to do the appropriate tasks to
accomplish the desired goals. ChefSpec requires a configuration test
specification that is then evaluated against a recipe.

For example, ChefSpec would not actually check whether Chef installed the
Apache package, but instead checks whether a Chef recipe asked to install
Apache. The goal of the test is to validate whether the recipe reflects the
intentions of the programmer.

Page 16
Amazon Web Services – Infrastructure as Code

Integration Testing with Test Kitchen

Test Kitchen is a testing platform that creates test environments and then uses
bussers, which are test frameworks, to validate the creation of the resources
specified in the Chef recipes. 54

By leveraging the previous testing tools in conjunction with OpsWorks for Chef
Automate workflow capabilities, developers can automate the testing of their
infrastructures during the development lifecycle. These tests are a form of code
themselves and are another key part of the Infrastructure as Code approach to

d
deployments.

Best Practices

e
The strategies, techniques, and suggestions presented here will help you get the

v
maximum benefit and optimal outcomes from AWS OpsWorks for Chef

i
Automate:

• Consider storing your Chef recipes in an Amazon S3 archive. Amazon S3

h
is highly reliable and durable. Explicitly version each archive file by
using a naming convention. Or use Amazon S3 versioning, which

c
provides an audit trail and an easy way to revert to an earlier version.

r
• Establish a backup schedule that meets your organizational governance
requirements.

A
• Use IAM to limit access to the OpsWorks for Chef Automate API calls.

Summary
Amazon EC2 Systems Manager lets you deploy, customize, enforce, and audit an
expected state configuration to your EC2 instances and servers or VMs in your
on-premises environment. AWS OpsWorks for Chef Automate enables you to
use Chef recipes to support the configuration of an environment. You can use
OpsWorks for Chef Automate independently or on top of an environment
provisioned by AWS CloudFormation. The run documents and policies
associated with Systems Manager and the recipes associated with OpsWorks for
Chef Automate can become part of the infrastructure code base and be
controlled just like application source code.

Page 17
Amazon Web Services – Infrastructure as Code

Monitoring and Performance

Having reviewed the role of Infrastructure as Code in the provisioning of
infrastructure resources and configuration management, we now look at
infrastructure health. Consider how the following events could affect the
operation of a website during periods of peak demand:

• Users of a web application are experiencing timeouts because of latency

of the load balancer, making it difficult to browse the product catalogs.

d
• An application server experiences performance degradation due to
insufficient CPU capacity and can no longer process new orders.

e
• A database that tracks session state doesn’t have enough throughput.

v
This causes delays as users transition through the various stages of an

i
application.

These situations describe operational problems arising from infrastructure

h
resources that don’t meet their performance expectations. It’s important to
capture key metrics to assess the health of the environment and take corrective

c
action when problems arise. Metrics provide visibility. With metrics, your
organization can respond automatically to events. Without metrics, your

r
organization is blind to what is happening in its infrastructure, thereby
requiring human intervention to address all issues. With scalable and loosely

A
coupled systems written in multiple languages and frameworks, it can be
difficult to capture the relevant metrics and logs and respond accordingly. To
address this need, AWS offers the Amazon CloudWatch services. 55

Amazon CloudWatch
Amazon CloudWatch is a set of services that ingests, interprets, and responds to
runtime metrics, logs, and events. CloudWatch automatically collects metrics
from many AWS services, such as Amazon EC2, Elastic Load Balancing (ELB),
and Amazon DynamoDB. 56, 57, 58 Responses can include built-in actions such as
sending notifications or custom actions handled by AWS Lambda, a serverless
event-driven compute platform. 59 The code for Lambda functions becomes part
of the infrastructure code base, thereby extending Infrastructure as Code to the
operational level. CloudWatch consists of three services: the main CloudWatch
service, Amazon CloudWatch Logs, and Amazon CloudWatch Events. We now
consider each of these in more detail.

Page 18
Amazon Web Services – Infrastructure as Code

Amazon CloudWatch
The main Amazon CloudWatch service collects and tracks metrics for many
AWS services such as Amazon EC2, ELB, DynamoDB, and Amazon Relational
Database Service (RDS). You can also create custom metrics for services you
develop, such as applications. CloudWatch issues alarms when metrics reach a
given threshold over a period of time.

Here are some examples of metrics and potential responses that could apply to
the situations mentioned at the start of this section:

d
• If the latency of ELB exceeds five seconds over two minutes, send an

e
email notification to the systems administrators.

• When the average EC2 instance CPU usage exceeds 60 percent for three

v
minutes, launch another EC2 instance.

i
• Increase the capacity units of a DynamoDB table when excessive
throttling occurs.

h
You can implement responses to metrics-based alarms using built-in

c
notifications, or by writing custom Lambda functions in Python, [Link], Java,
or C#. Figure 6 shows an example of how a CloudWatch alarm uses Amazon

r
Simple Notification Service (Amazon SNS) to trigger a DynamoDB capacity
update.

A
CloudWatch SNS Lambda
Alarm Notification Function
ThrottledEvents > 2 Publish to Call API
over 5 minutes DynamoDB Topic [Link]

Figure 6: Example of a CloudWatch alarm flow

Amazon CloudWatch Logs

Amazon CloudWatch Logs monitors and stores logs from Amazon EC2, AWS
CloudTrail, and other sources. EC2 instances can ship logging information using
the CloudWatch Logs Agent and logging tools such as Logstash, Graylog, and
Fluentd. 60 Logs stored in Amazon S3 can be sent to CloudWatch Logs by
configuring an Amazon S3 event to trigger a Lambda function.

Page 19
Amazon Web Services – Infrastructure as Code

Ingested log data can be the basis for new CloudWatch metrics that can, in turn,
trigger CloudWatch alarms. You can use this capability to monitor any resource
that generates logs without writing any code whatsoever. If you need a more
advanced response procedure, you can create a Lambda function to take the
appropriate actions. For example, a Lambda function can use the
[Link] or [Link] APIs to publish information to a Slack channel
when NullPointerException errors appear in production logs. 61, 62

Log processing and correlation allow for deeper analysis of application

d
behaviors and can expose internal details that are hard to figure out from
metrics. CloudWatch Logs provides both the storage and analysis of logs, and
processing to enable data-driven responses to operational issues.

e
Amazon CloudWatch Events

i v
Amazon CloudWatch Events produces a stream of events from changes to AWS
environments, applies a rules engine, and delivers matching events to specified
targets. Examples of events that can be streamed include EC2 instance state

h
changes, Auto Scaling actions, API calls published by CloudTrail, AWS console
sign-ins, AWS Trusted Advisor optimization notifications, custom application-

c
level events, and time-scheduled actions. Targets can include built-in actions

r
such as SNS notifications or custom responses using Lambda functions.

The ability of an infrastructure to respond to selected events offers benefits in

A
both operations and security. From the operations perspective, events can
automate maintenance activities without having to manage a separate
scheduling system. With regard to information security, events can provide
notifications of console logins, authentication failures, and risky API calls
recorded by CloudTrail. In both realms, incorporating event responses into the
infrastructure code promotes a greater degree of self-healing and a higher level
of operational maturity.

Best Practices
Here are some recommendations for best practices related to monitoring:

• Ensure that all AWS resources are emitting metrics.

• Create CloudWatch alarms for metrics that provide the appropriate

responses as metric-related events arise.

Page 20
Amazon Web Services – Infrastructure as Code

• Send logs from AWS resources, including Amazon S3 and Amazon EC2,
to CloudWatch Logs for analysis using log stream triggers and Lambda
functions.

• Schedule ongoing maintenance tasks with CloudWatch and Lambda.

• Use CloudWatch custom events to respond to application-level issues.

Summary
Monitoring is essential to understand systems behavior and to automate data-

d
driven reactions. CloudWatch collects observations from runtime environments,
in the form of metrics and logs, and makes those actionable through alarms,

e
streams, and events. Lambda functions written in Python, [Link], Java, or C#
can respond to events, thereby extending the role of Infrastructure as Code to

v
the operational realm and improving the resiliency of operating environments.

i
Governance and Compliance

h
Having considered how you can use Infrastructure as Code to monitor the

c
health of your organization’s environments, we now turn our focus to
technology governance and compliance. Many organizations require visibility

r
into their infrastructures to address industry or regulatory requirements. The
dynamic provisioning capabilities of the cloud pose special challenges because

A
visibility and governance must be maintained as resources are added, removed,
or updated. Consider the following situations:

• A user is added to a privileged administration group, and the IT

organization is unable to explain when this occurred.

• The network access rules restricting remote management to a limited set

of IP addresses are modified to allow access from additional locations.
• The RAM and CPU configurations for several servers has unexpectedly
doubled, resulting in a much larger bill than in previous months.

Although you have visibility into the current state of your AWS resource
configurations using the AWS CLI and API calls, addressing these situations
requires the ability to look at how those resources have changed over time. To
address this need, AWS offers the AWS Config service. 63

Page 21
Amazon Web Services – Infrastructure as Code

AWS Config
AWS Config enables you to assess, audit, and evaluate the configurations of
your AWS resources. AWS Config automatically builds an inventory of your
resources and tracks changes made to them. Figure 7 shows an example of an
AWS Config inventory of EC2 instances.

e d
h i v
rc Figure 7: Example of an AWS Config resource inventory

A
AWS Config also provides a clear view of the resource change timeline,
including changes in both the resource configurations and the associations of
those resources to other AWS resources. Figure 8 shows the information
maintained by AWS Config for a VPC resource.

Page 22
Amazon Web Services – Infrastructure as Code

e d
v
Figure 8: Example of AWS Config resource timeline

i
When many different resources are changing frequently and automatically,
automating compliance can become as important as automating the delivery

h
pipeline. To respond to changes in the environment, you can use AWS Config
rules.

rc
AWS Config Rules
With AWS Config rules, every change triggers an evaluation by the rules

A
associated with the resources. AWS provides a collection of managed rules for
common requirements such as IAM users having good passwords, groups and
policies, or for determining if EC2 instances are on the correct VPCs and
Security Groups. AWS Config rules can quickly identify noncompliant resources
and help with reporting and remediation. For validations beyond those
provided by the managed rules, AWS Config rules also support the creation of
custom rules using Lambda functions. 64 These rules become part of the
infrastructure code base, thus bringing the concept of Infrastructure as Code to
the governance and compliance stages of the information resource lifecycle.

Rule Structure
When a custom rule is invoked through AWS Config rules, the associated
Lambda function receives the configuration events, processes them, and returns
results. The following function determines if Amazon Virtual Private Cloud
(Amazon VPC) flow logs are enabled on a given Amazon VPC.

Page 23
Amazon Web Services – Infrastructure as Code

import boto3
import json

def evaluate_compliance(config_item, vpc_id):

if (config_item['resourceType'] != 'AWS::EC2::VPC'):
return 'NOT_APPLICABLE'
elif is_flow_logs_enabled(vpc_id):
return 'COMPLIANT'
else:
return 'NON_COMPLIANT'

d
def is_flow_logs_enabled(vpc_id):

e
ec2 = [Link]('ec2')
response = ec2.describe_flow_logs(
Filter=[{'Name': 'resource-id','Values': [vpc_id]},],

v
)

i
if len(response[u'FlowLogs']) != 0: return True

def lambda_handler(event, context):

h
invoking_event = [Link](event['invokingEvent'])
compliance_value = 'NOT_APPLICABLE'

c
vpc_id = invoking_event['configurationItem']['resourceId']

r
compliance_value = evaluate_compliance(
invoking_event['configurationItem'], vpc_id)

A
config = [Link]('config')
response = config.put_evaluations(
Evaluations=[
{
'ComplianceResourceType':
invoking_event['configurationItem']['resourceType'],
'ComplianceResourceId': vpc_id,
'ComplianceType': compliance_value,
'OrderingTimestamp':
invoking_event['configurationItem']['configurationItemCaptureTim
e']
},
],
ResultToken=event['resultToken'])

Figure 9: Example of a Lambda function to support AWS Config rules

Page 24
Amazon Web Services – Infrastructure as Code

In this example, when a configuration event on an Amazon VPC occurs, the

event passes to the function lambda_handler. This code extracts the ID of the
Amazon VPC and uses the describe_flow_logs API call to determine whether
the flow logs are enabled. The Lambda function returns a value of COMPLIANT
if the flow logs are enabled and NON_COMPLIANT otherwise.

Best Practices
Here are some recommendations for implementing AWS Config in your
environments:

d
• Enable AWS Config for all regions to record the configuration item

e
history, to facilitate auditing and compliance tracking.

• Implement a process to respond to changes detected by AWS Config.

v
This could include email notifications and the use of AWS Config rules

i
to respond to changes programmatically.

h
Summary
AWS Config extends the concept of infrastructure code into the realm of

c
governance and compliance. AWS Config can continuously record the

r
configuration of resources while AWS Config rules allow for event-driven
responses to changes in the configuration of tracked resources. You can use this
capability to assist your organization with the monitoring of compliance

A
controls.

Resource Optimization
We now focus on the final stage in the information resource lifecycle, resource
optimization. In this stage, administrators review performance data and identify
changes needed to optimize the environment around criteria such as security,
performance, and cost management. It’s important for all application
stakeholders to regularly evaluate the infrastructure to determine if it is being
used optimally.

Consider the following questions:

• Are there provisioned resources that are underutilized?

Page 25
Amazon Web Services – Infrastructure as Code

• Are there ways to reduce the charges associated with the operating
environment?

• Are there any suggestions for improving the performance of the

provisioned resources?

• Are there any service limits that apply to the resources used in the
environment and, if so, is the current usage of resources close to
exceeding these limits?

To answer these questions, we need a way to interrogate the operating

d
environment, retrieve data related to optimization, and use that data to make
meaningful decisions. To address this need, AWS offers AWS Trusted Advisor. 65

e
AWS Trusted Advisor

i v
AWS Trusted Advisor helps you observe best practices by scanning your AWS
resources and comparing their usage against AWS best practices in four
categories: cost optimization, performance, security, and fault tolerance. As part

h
of ongoing improvement to your infrastructure and applications, taking
advantage of Trusted Advisor can help keep your resources provisioned

c
optimally. Figure 10 shows an example of the Trusted Advisor dashboard.

A r
Figure 10: Example of the AWS Trusted Advisor dashboard

Checks
Trusted Advisor provides a variety of checks to determine if the infrastructure is
following best practices. The checks include detailed descriptions of

Page 26
Amazon Web Services – Infrastructure as Code

recommended best practices, alert criteria, guidelines for action, and a list of
useful resources on the topic. Trusted Advisor provides the results of the checks
and can also provide ongoing weekly notifications for status updates and cost
savings.

All customers have access to a core set of Trusted Advisor checks. Customers
with AWS Business or Enterprise support can access all Trusted Advisor checks
and the Trusted Advisor APIs. Using the APIs, you can obtain information from
Trusted Advisor and take corrective actions. For example, a program could

d
leverage Trusted Advisor to examine current account service limits. If current
resource usages approach the limits, you can automatically create a support case
to increase the limits.

e
Additionally, Trusted Advisor now integrates with Amazon CloudWatch Events.

v
You can design a Lambda function to notify a Slack channel when the status of

i
Trusted Advisor checks changes. These examples illustrate how the concept of
Infrastructure as Code can be extended to the resource optimization level of the

h
information resource lifecycle.

c
Best Practices

r
The best practices for AWS Trusted Advisor appear below.

• Subscribe to Trusted Advisor notifications through email or an

A
alternative delivery system.

• Use distribution lists and ensure that the appropriate recipients are
included on all such notifications.

• If you have AWS Business or Enterprise support, use the AWS Support
API in conjunction with Trusted Advisor notifications to create cases
with AWS Support to perform remediation.

Summary
You must continuously monitor your infrastructure to optimize the
infrastructure resources with regard to performance, security, and cost. AWS
Trusted Advisor provides the ability to use APIs to interrogate your AWS
infrastructure for recommendations, thus extending Infrastructure as Code to
the optimization phase of the information resource lifecycle.

Page 27
Amazon Web Services – Infrastructure as Code

Next Steps
You can begin the adoption of Infrastructure as Code in your organization by
viewing your infrastructure specifications in the same way you view your
product code. AWS offers a wide range of tools that give you more control and
flexibility over how you provision, manage, and operationalize your cloud
infrastructure.

Here are some key actions you can take as you implement Infrastructure as

d
Code in your organization:

e
• Start by using a managed source control service, such as AWS
CodeCommit, for your infrastructure code.

v
• Incorporate a quality control process via unit tests and static code

i
analysis before deployments.

• Remove the human element and automate infrastructure provisioning,

h
including infrastructure permission policies.

c
• Create idempotent infrastructure code that you can easily redeploy.

r
• Roll out every new update to your infrastructure via code by updating
your idempotent stacks. Avoid making one-off changes manually.

A
• Embrace end-to-end automation.

• Include infrastructure automation work as part of regular product

sprints.

• Make your changes auditable, and make logging mandatory.

• Define common standards across your organization and continuously

optimize.

By embracing these principles, your infrastructure can dynamically evolve and

accelerate with your rapidly changing business needs.

Conclusion
Infrastructure as Code enables you to encode the definition of infrastructure
resources into configuration files and control versions, just like application

Page 28
Amazon Web Services – Infrastructure as Code

software. We can now update our lifecycle diagram and show how AWS
supports each stage through code.

AWS CloudFormation

e d
AWS OpsWorks for
AWS Trusted Advisor
Chef Automate

v
Amazon EC2

i
Systems Manager

h
AWS Config

c
Amazon CloudWatch

r
Figure 11: Information resource lifecycle with AWS

A
AWS CloudFormation, AWS OpsWorks for Chef Automate, Amazon EC2
Systems Manager, Amazon CloudWatch, AWS Config, and AWS Trusted
Advisor enable you to integrate the concept of Infrastructure as Code into all
phases of the project lifecycle. By using Infrastructure as Code, your
organization can automatically deploy consistently built environments that, in
turn, can help your organization to improve its overall maturity.

Page 29
Amazon Web Services – Infrastructure as Code

Contributors
The following individuals and organizations contributed to this document:

• Hubert Cheung, solutions architect, Amazon Web Services

• Julio Faerman, technical evangelist, Amazon Web Services

• Balaji Iyer, professional services consultant, Amazon Web Services

• Jeffrey S. Levine, solutions architect, Amazon Web Services

d
Resources

e
Refer to the following resources to learn more about our best practices related

v
to Infrastructure as Code.

i
Videos

h
• AWS re:Invent 2015 – DevOps at Amazon 66

c
• AWS Summit 2016 - DevOps, Continuous Integration and Deployment
on AWS 67

r
Documentation & Blogs

A
• DevOps and AWS 68

• What is Continuous Integration 69

• What is Continuous Delivery 70

• AWS DevOps Blog 71

Whitepapers

• Introduction to DevOps on AWS 72

• AWS Operational Checklist 73

• AWS Security Best Practices 74

• AWS Risk and Compliance 75

Page 30
Amazon Web Services – Infrastructure as Code

AWS Support

• AWS Premium Support 76

• AWS Trusted Advisor 77

Notes
1 [Link]
2 [Link]

d
3 [Link]
[Link]

e
4 [Link]

v
5

i
[Link]
[Link]

h
6 [Link]
[Link]

c
7

r
[Link]
[Link]

A
8

[Link]
_ValidateTemplate.html
9 [Link]
10 [Link]
development-process-of-a-cloudformation-template-with-cfn-nag/
11 [Link]
12 [Link]
[Link]
13 [Link]
[Link]#organizingstacks
14 [Link]
[Link]#use-iam-to-control-access

Page 31
Amazon Web Services – Infrastructure as Code

15 [Link]
[Link]#reuse
16 [Link]
[Link]#nested
17 [Link]
[Link]#cross-stack
18 [Link]

d
[Link]#creds
19 [Link]

e
[Link]#parmtypes
20 [Link]

v
[Link]#parmconstraints

i
21 [Link]
[Link]#cfninit

h
22 [Link]

c
[Link]#helper-scripts
[Link]

r
23

[Link]#validate
24 [Link]

A
25 [Link]
[Link]#donttouch
26 [Link]
[Link]#cfn-best-practices-changesets
27 [Link]
[Link]#stackpolicy
28 [Link]
[Link]#cloudtrail
29 [Link]
[Link]#code
30 [Link]
[Link]#update-ec2-linux
31 [Link]
32 [Link]

Page 32
Amazon Web Services – Infrastructure as Code

33 [Link]
[Link]
34 [Link]
[Link]
35 [Link]
[Link]
36 [Link]

d
[Link]
37 [Link]

e
38 [Link]

v
39 [Link]

i
40 [Link]
ec2-systems-manager/

h
41 [Link]
[Link]

c
42 [Link]

r
[Link]
43 [Link]

A
ec2-systems-manager/
44 [Link]
[Link]
45 [Link]
inspector-security-findings-automatically/
46 [Link]
[Link]
47 [Link]
[Link]
48 [Link]
[Link]
49 [Link]
applications-using-parameter-store-and-iam-roles-for-tasks/
50 [Link]
51 [Link]

Page 33
Amazon Web Services – Infrastructure as Code

52 [Link]
53 [Link]
54 [Link]
55 [Link]
56 [Link]
57 [Link]

d
58 [Link]
59 [Link]

e
60

[Link]

v
[Link]

i
61 [Link]

h
62 [Link]
63 [Link]

c
64 [Link]

r
config_develop-[Link]
65 [Link]

A
66 [Link]
67 [Link]
68 [Link]
69 [Link]
70 [Link]
71 [Link]
72 [Link]
73 [Link]
74

[Link]
[Link]
75

[Link]
nce_Whitepaper.pdf

Page 34
Amazon Web Services – Infrastructure as Code

76 [Link]
77 [Link]

e d
h i v
rc
A
Page 35