Leveraging DevOps and DevSecOps to

Accelerate AI Development and Deployment

Hasan Yasar
Technical Director, Adjunct Faculty Member
Software Engineering Institute | Carnegie Mellon University

Software Engineering Institute

Carnegie Mellon University
Pittsburgh, PA 15213

[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
1
2020 Carnegie Mellon University
Copyright 2020 Carnegie Mellon University.
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8702-15-D-0002
with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and
development center.
The view, opinions, and/or findings contained in this material are those of the author(s) and should not be construed as an
official Government position, policy, or decision, unless designated by other documentation.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON
AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS
TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY,
OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF
ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see
Copyright notice for non-US Government use and distribution.
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form
without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to
the Software Engineering Institute at permission@[Link].
DM20-0810

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
2
2020 Carnegie Mellon University
Outline

Modern SW Development: DevOps

Fundamentals,
DoD Specific -ATO
AI for DevOps
DevOps for AI

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
3
2020 Carnegie Mellon University
Modern SW Development: DevOps

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
4
2020 Carnegie Mellon University
 DevOps is a set of principles and practices emphasizing collaboration and communication
between software development teams and IT operations staff along with acquirers, suppliers,
and other stakeholders in the lifecycle of a software system

DevSecOps is a model on integrating the software development and operational process

considering security activities: requirements, design, coding, testing , delivery , deployment
and incident response.

Mature DevOps practices are constantly testing, deploying and validating that software
meets every requirement and allows for fast recovery in the event of a problem. As a result
we can easily say,

“DevSecOps is DevOps done right”

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
5
2020 Carnegie Mellon University
Who are Dev?

• Follow Agile methodologies

• Using Scrum, Kanban and modern
development approaches
• Self directing, self managed, self organized
• Using any new technology
• Each Dev has own development strategy
• OpenSource,
• Allowed to have
• Close relationships with the business
• Software driven economy

Want to deliver software faster with new requirements…

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
6
2020 Carnegie Mellon University
Who are Ops?

• Operations
• Runs the application
• Manages the infrastructure
• Support the applications
• Operations provides
• Service Strategy
• Service Design
• Service Transition
• Service Operations
• Secure systems

Want to maintain stability, reliability and security…

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
7
2020 Carnegie Mellon University
DevOps aims to Increase…

…the pace of innovation

…responsiveness to business needs

…collaboration

…software stability and quality

... continuous feedback

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
8
2020 Carnegie Mellon University
DevOps has four Fundamental Principles

•Collaboration: between project team roles

•Infrastructure as Code: all assets are versioned, scripted, and shared where
possible

•Automation: deployment, testing, provisioning, any manual or human-error-prone

process

•Monitoring: any metric in the development or operational spaces that can inform
priorities, direction, and policy

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
9
2020 Carnegie Mellon University
SW Development Phases

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
10
2020 Carnegie Mellon University
Collaboration: Many stakeholders
Collaboration: Many stakeholders
IT Operations Business Analyst
Scalability
Business Constraints

Infrastructure Deployment Developers Legal Issues

User
Requirements
Networks Performance
Maintenance Market Needs
Functional
Updates Requirements Budgets / Timelines
Programming

Technical Testing
Documentation Monitoring
Code Review
Data Privacy
Release
User Interface Incident response
Review
User Security Intrusion
Documentation
Detection

Quality Assurance Information Security

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
11
2020 Carnegie Mellon University
Collaboration: Silos Inhibit Collaboration and poor
communication

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
12
2020 Carnegie Mellon University
Infrastructure as Code (IaC)
A program that creates infrastructure,

A concretely defined description of the environment is good material for

conversation between team members.
Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
13
2020 Carnegie Mellon University
Automation : Continuous Integration (CI)

Continuous integration is a process that continually merges a system’s artifacts, including source code updates and configuration
items from all stakeholders on a team, into a shared mainline to build and test the developed system.

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
14
2020 Carnegie Mellon University
Automation : Continuous Delivery / Deployment (CD)

Shift Left Operational Concerns Enforced by Continuous Delivery with parity across
various environment

Continuous delivery is a software engineering practice that allows for frequent releases of new software to
staging or various test environments through the use of automated testing.
Continuous deployment is the automated process of deploying changes to production by verifying intended
features and validations to minimize risk.

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
15
2020 Carnegie Mellon University
Integrated Development Pipeline - General

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
16
2020 Carnegie Mellon University
DevOps Stack: Exemplary DoD tool stack

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
17
2020 Carnegie Mellon University
Automation with IaC, CI, CD

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
18
2020 Carnegie Mellon University
Key Benefits of DevOps

• Reduced errors during deployment

• Reduced time to deploy and resolve
discovered errors
• Repeatable steps
• Continuous availability of pipeline
and application
• Increased innovation time
• Responsiveness to business needs
• Traceability throughout the
application lifecycle
• Increased stability and quality
• Continuous feedback

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
19
2020 Carnegie Mellon University
DoD Specific

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
20
2020 Carnegie Mellon University
DoD Does Not Control SW Development

• DoD does not produce most of the software it

uses, but it must maintain that software
• Latent cyber vulnerabilities, those exposed during
operations, and those due to underlying
dependencies are putting the DoD at risk
• Finding and fixing problems late causes rework
and drives up costs
• Software cost overruns are overwhelming program
delivery and sustainment

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
21
2020 Carnegie Mellon University
Barriers to DevSecOps

• Complex systems (Safety critical, Realtime, Embedded Systems..

• Sustainment of DevOps pipeline
• Lack of iterative and incremental mindset cultural issues
• Organizational Structure
• Legacy Systems
• Lack of modular architecture, old tools/language
• Aged bureaucracy and waterfall process
• Lack of Metrics and Measurements
• Inconsistent Environments
• RMF- ATO Compliances

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
22
2020 Carnegie Mellon University
ATO: Authority to Operate
What is an ATO? • An AO makes a risk-based decision to
• An ATO is Authority to Operate grant an ATO for use of the system
- Authorizes the system to be placed on a • The decision has to be formalized in an
production network ATO letter
- Interface with other components within the
- An ATO letter must explicitly state the AO’s
DoD
acceptance of:
- Authorizes access by end users to
• Use of the system at the Agency at the
leverage these resources to execute
determined FIPS 199 impact level
mission
• All leveraged external services supporting
• Key staff in the ATO process the system
- AO (Authorizing Official) • Any exceptions or exclusions of the Chief
- ISSO (Information System Security Security Officer (CSO) for use at the
Officer) Agency
- Security assessor

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
23
2020 Carnegie Mellon University
 Continuous ATO(cATO) is the Goal

cATO authorizes the

platform, process, and the
team that produces the
product under a
continuous monitoring
process that maintains the
residual risk within the risk
tolerance of the
Authorizing Official (AO)

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
24
2020 Carnegie Mellon University
cATO and DevSecOps

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
25
2020 Carnegie Mellon University
Proposed DoD Enterprise DevSecOps

• Develop and implement common DevSecOps implementation (DoD CIO DevSecOps

workgroup) [Link]
• Create and Maintain DevSecOps pipelines (and not just DevOps) to avoid each DoD
service building their own stack and reinventing the wheel.
• Create hardened Container images in a dedicated artifacts repository with security built-in
and compliance with FedRAMP/NIST (similar to gold images concept).
• Create a Microservice Service Architecture with Service Mesh (ISTIO).
• Standardize metrics and define acceptable thresholds for test coverage, security,
documentation etc. to enable complete continuous deployment with pre-ATO embedded.
• Leverage Kubernetes for Orchestration to ensure automation, rolling-update, scale,
security and visibility thanks to the sidecar security container concept.

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
26
2020 Carnegie Mellon University
 DevOps with Security for cATO

Security from
inception to
deployment and Continuous
improvement Authorization on
with every every phases
delivery

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
27
2020 Carnegie Mellon University
AI for DevOps

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
28
2020 Carnegie Mellon University
AI For DevOps
Using ML and AI to ‘inform’ a
DevOps factory or pipeline of
notable events, usually to help
improve the process over time, or
help make decisions based on
real-time event.
Requirements:
• Monitoring each step
• Must develop models that
allow for ‘actionable’ events.

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
29
2020 Carnegie Mellon University
DevOps for AI

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
30
2020 Carnegie Mellon University
Traditional vs ML Based Approaches?

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
31
2020 Carnegie Mellon University
Code in AI System

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
32
2020 Carnegie Mellon University
More formally…

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
33
2020 Carnegie Mellon University
ML Development Cycle

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
34
2020 Carnegie Mellon University
DevOps for AI
Using DevOps concepts and
methodologies in every aspect of ML
and AI enabled software systems.

• Data curation
• Training data
• Model creation, storage
• Deployment
• Monitoring
• Re-training

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
35
2020 Carnegie Mellon University
DataOps and MLOps exist

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
36
2020 Carnegie Mellon University
Important considerations

• Data must be prepared before model training

• Model release requires operationalization
• Post-deployment monitoring should record all real-world data serving as input to the
deployed model
• Team members include
• Data engineers, Data scientists
• ML engineers, DevOps
• Developers
• Model performance
• Deployment strategies
• Model storage and sharing

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
37
2020 Carnegie Mellon University
AI Engineering Framework
AI Technologies Infrastructure
and Components • Data, data management, • Platform choices and transferability
• Knowledge representation and data pipelines • Tooling
• Modeling and abstraction • Scalability, performance, • Sensors and actuators
and evaluation
• Algorithms • Future architectures
• Computational resources and
• Scalability and performance
considerations (CSWaP)
• Robust component design
• Processing placement and
• Component V&V deployment, data locality
• Novelty and Uncertainty

AI System Process and Policy

• Architecture, analysis and design • AI system acquisition • Economic considerations
• Model-based engineering • AI-ready modern software practices • AI-centric Threat Model
• Virtual integration • ML lifecycle management • Risk and resiliency
• Robustness and resiliency • Data lifecycle management • Security Coordination (AI C/PSIRT)
• Build security-in Secure AI • Standards • Ethics
• System V&V • Benchmarks • Bias
• Socio-technical issues • Privacy
Human-Machine Teaming
• Interpretability and explainability • Co-learning
• Human-machine trust • Communicating uncertainty
• Machine-human trust • Data Presentation

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
38
2020 Carnegie Mellon University
Necessary DevOps Factory additions

1. Embrace an MLOps culture to facilitate an

ML- driven factory
2. Establish a cultural focus on data-driven
development to facilitate ML model creation
3. Include data scientists and data engineers in
software development teams
4. Automate model deployment via continuous
delivery/deployment
5. Establish continuous feedback including
model monitoring like model inputs, model
outputs and decisions, user action and
rewards and model fairness.

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
39
2020 Carnegie Mellon University
DevOps for data curation

Data is a common critical element of an AI system

• The general process for data processing:
- Develop business cases
- Ingest
- Classify/transform/analyze
- Insights
- Availability of Data for DS
- Validating Data

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
40
2020 Carnegie Mellon University
Data curation

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
41
2020 Carnegie Mellon University
Data curation - workflow

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
42
2020 Carnegie Mellon University
Monitoring deployed AI systems
• Include a return loop of data to the starting point of the pipeline
• Archive all ingress data to the model for future training
• Record and analyze the model’s output for functionality and integrity
• Determine if a model requires modification or re-training
• Model inputs: what data, predictions or recommendations
• Model outputs and decisions
• Model interpretability outputs
• Example: using EFK stack for monitoring and observability
• Elasticsearch: an open source search engine.
• FluentD: an open source data collector for unified logging layer.
• Kibana: an open source web UI that makes it easy to explore and visualize the data
indexed by Elasticsearch.
Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
43
2020 Carnegie Mellon University
Additional guidance for an AI/ML Pipeline

• Capable of ingesting multiple data types

• Data maintained and versioned
• Data Version Control ([Link])
• Real-time monitoring
• Responsive to changing conditions discovered
during monitoring
• Traceability
• Language standardization

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
44
2020 Carnegie Mellon University
Exemplary AI+DevSecOps implementation scenarios

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
45
2020 Carnegie Mellon University
Exemplary AI+DevSecOps implementation scenarios

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
46
2020 Carnegie Mellon University
Summary

• Use DevOps to build, deploy, and monitor systems so that a pathway exists to take
action on a ML/AI enabled system.
• These ‘actions’ could improve model performance, system security, and many other
possibilities

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
47
2020 Carnegie Mellon University
For more information…

DevOps: [Link]
DevOps Blog: [Link]
Webinar : [Link]
Podcast : [Link]

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
48
2020 Carnegie Mellon University
Thank You
Hasan Yasar
Technical Director, Adjunct Faculty Member
Continuous Deployment of Capability
hyasar@[Link]
@securelifecycle

Leveraging DevOps and DevSecOps to Accelerate AI Development and Deployment [DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
49
2020 Carnegie Mellon University