Containers vs VMs

[ the eternal misunderstanding ]

Before diving into the details of WHAT

containers are and HOW they work, I'd like to
clear out the most confusing one of all. We
love comparing Containers to VMs because
it's simple to explain. But they are
COMPLETELY different. And ket's see WHY.

Imagine taking the whole

Windows OS, along with your app

VM is a complete
inside, adding it to ZIP archive and
distributing it further. Sure, you
Operating System are guaranteed that all
dependencies are there (they're
with your App being ZIPPED after all!) but whoever
part of it wants to run it has to bootstrap
the whole OS (yes, that includes
that famous Windows startup
sound as well!). It's bulky!

In contrast to VMs, container

would be like taking your
Container is just
executable and zipping it along your app with it's
with it's direct dependencies.
That's it! No OS, no startup sound,
dependencies. That's
nothing else - just the essentials! all!
It's as light as travleing with a
backpack!

Both could be used to carry data

It's like comparing with you, true, but would you
really always carry laptop if you
USB flash to Laptop. could just put your data to an USB
They serve similar stick? I hope not! It's the same
with Docker vs VM. They do
purpose, but are they achieve the same thing, but are

the same thing? completely different technology.

One is super-light but can't run on
it's own, while other is bulky and
standalone, but not practical when
You
You should
should know:
know you want to carry lots of them!
Yes, Windows can run containers natively as well! This
is a common question and source of confusion. It
wasn't possible in the beginning, and we'll dig into
WHY, but these days - yes, it's possible! And I'll be
talking more about it in the following graphics!
BitesizedEngineering.com
 Docker. Containers
[ there's more than Docker! ]

Interestingly enough, many people equate

"Docker" with "Containers". In principle, that's
OK, because Docker IS about Containers, but
Docker IS NOT the only solution out there.
Docker is just ONE OF the possible solutions that
happens to be most popular at the moment.

Containers
It is absolutely possible to start a
container fully manually, without
are hard Docker and other stuff. But the

to
problem is that there's a lot you
need to take care of and it just
maintain takes away too much time.
Enthusiasts can google for "runc"
or "hcsshim" for Windows
containers.

Docker abstracts a lot of things

away from you, and as such

Docker
provides a Platform for running
Containers. But if you go deep-
down, what it really does is takes that
pain away
orchestrates "dockerd", which in
turn talks to "runc" and ensures
that your containers are
bootstrapped and run properly.

Docker One thing that DID come out of

introduced
Docker is concept of "Images" and
"Layers". We will talk more about
"Images" them in the future graphics, but
what you need to know is that,
before Docker, you always had to
pack EVERYTHING in your
Container Bundle. And distribution
was simply pain in the ass!

Another great thing that came out

OCI
of Docker effort is Open Container
Initiative. OCI is a de-facto

specification
standard crafted by Docker & other
leaders in the field, that specifies a
format which you can use to
define, build & run containers on is born
any platform. What this means?
You're free to use whichever
solution you want, as long as you
conform to standards!
BitesizedEngineering.com
 Open Container
Initiative
[ OCI explained ]

If you dealt with containers recently, chances are

that you have heard of OCI. OCI was established
in order to standardize the way we PACK,
DISTRIBUTE and RUN containers. And yes, it
wasn't standardized up until 2015th. Let's learn
more about it!

In order to understand WHY we have

OCI, you need to remember the
Container as (modern) purpose of having

a Unit of
containers. They are supposed to
represent a BASIC UNIT OF A
Software SOFTWARE DELIVERY. Think of an
Delivery .EXE file runnable anywhere
(Windows, Linux, OSC, etc.) without
any additional requirements. That's
what Container is!

Open Container Initiative aims at

defining a SINGLE format and
standard which EVERY platform can
adopt. And by conforming to those
OCI defines
HOW to
standards, you enable users to
execute containers on ANY
environment. In order to achieve this, achieve this!
OCI defines THREE specifications:
Runtime Spec, Image Spec and
Distribution Spec.

Runtime Spec is all about HOW you

pack your Container Bundle so that
Runtime any environment can interpret and run
it. It boils down to defining a manifest
Specification file (config.json) and filesystem bundle
needed in order to execute your
application.

Image Spec allows you to segment

your Runtime Spec into Layers so that
Image you can reuse them. Think of a

Specification
Windows app - one layer could be
Windows filesystem itself, and second
layer could be your app. When
rebuilding, you'd just change your App
Layer, while Windows one remains
cached.

Distribution Spec tells you how to

Distribution distribute your Images with ease.
Think of it as a GitHub for Containers -
Specification it caches a lot and allows you to
pull/push only the relevant
differences.

BitesizedEngineering.com
 "Container is not a VM"
[ and what it really means ]

I used to think that "Container is not a VM"

means that "VM" carries it's own OS, whereas
Container only carries it's dependencies; and I
was deadly wrong! It's a true statement that
'Container packs only it's dependencies', but due
to intertwined dependencies, turns out that most
of the time this really means 'the whole OS'. Let
me give you the main simplest difference.

vm container

Host OS

hardware hardware

Main Difference is that VM talks directly

to Hardware, whereas container talks to
Host OS' Kernel!
VM packs a FULL OS, including OS Kernel. This

VM carries has lots of implications, but main ones are that

security & isolation are super-strong (you don't
it's own have NOTHING with Host OS, and you can't
Kernel 'escape' the isolation) and that you can really
run it on any OS (in contrast to containers
which are OS dependent).

Even though most containers pack almost the

whole OS, they still rely on Host's OS to execute
Container
relies on
stuff for them. This makes them lighter & allows
faster boot times (i.e. no need to start Kernel), but
on the other hand imposes security risks (i.e. since Host OS'
you are executing code within Host OS, there's a
chance that malicious code could try escaping from
kernel
the boundaries of container and end up in Host
OS!).

Given that Containers rely on Host OS' Kernel, this

means that if you containerize a Windows App
Containers with Windows dependencies, you can only

can execute
execute it on Windows, because you need
Windows' kernel to run it. Same is true for Linux
on same OS apps - they can run only on boxes that have Linux

only
Kernel. In order to run Linux Containers on
Windows (aka LCOW), you need Linux's Kernel
and that's where VMs could jump in (i.e. start
Linux VM and execute containers on top of it).

BitesizedEngineering.com
 Containers are like
The Truman Show
[ they think they are independent; and yet ]

From the POV of Container - they believe they

are independent. They see only their process list,
they see their fully functional file system, etc. As
far as they know - they are on their own and they
talk directly to hardware. And yet - they are
actually locked inside Host OS' playroom and
Host OS knows all about them. So, are they free?

I'm on my
container 1 own OS!
process a
I'm on my
container 2 own OS!
process b

Host OS Hahaha!

Containers
If you snoop around folders & files inside Container,
you'd be made to believe you have a full OS on your
own. It's actually an interesting task to figure out

think they are whether you are in a Container or on a full OS. Truth
is - they aren't. They are locked inside Host OS and

on their own it's the container runtime that proxies all the Kernel
stuff to Host OS to execute.

Thing is - from the outside, Container is just a

process that is locked to operate inside a
Host OS sees
specific folder; and this folder is pretty much
visible to Host OS. all your stuff
One of the reasons people opt to use
containers is that the stuff inside them is
Isolation is ISOLATED. And it is, but it's a WEAK isolation
(in comparison to VM, for example). It's weak

rather weak because you are still executing INSIDE the

boundaries of Host OS. And since you are
INSIDE host OS, this also means that you could
find ways to exploit this and ESCAPE out of
container and on to host OS!

Hyper-V isolation mode was built as an answer

Use Hyper-V
to "how do I build stronger isolation". Instead of
having container communicate directly to Host
OS, it runs containers inside a specialized and
light-weight VM, along with its own Kernel. This
provides a way stronger isolation boundary,
mode for Strong
and at the same time this VM was made to be
able to use Host OS' resources, making it boot
Isolation
much faster.

You should know:

There's an interesting talk called "A COW (Container on Windows)

that escaped the Silo. It demonstrates how malicious app could
try and escape the container boundaries, gain access over host
Kernel and pretty much get a full access to the system. It also
points that if you are running unknown apps, you MUST use
Hyper-V isolation mode if you want to stay safe.
BitesizedEngineering.com
 How do COWs (Containers
on Windows) Work
[ few words on communication channels ]

As you could imagine, Linux and Windows are quite

different in terms of architecture and how they
handle stuff. That means that the typical Container
enablers on Linux (namespaces, cgroups, union fs,
etc.) are not really available on Windows. So, how
do containers talk to Windows Kernel then?

On Linux: On Windows:

container executes app

containerd helps start, stop containerd

containerd and manage containers

runc (on Linux) and HCSSHIM hcsshim

runc
(on Windows) run the actual
show. They are the "runtime"
engines that bridges the Host Compute
Linux communication between Service
Kernel Container and Kernel
Windows
Kernel

Host Compute Service (HCS) is a brand-new API

Host developed by Windows Team in order to expose
Compute underlying Kernel, while providing functionality

Service!
needed by Docker (and OCI in general)! It's the
MAIN PIECE OF PUZZLE that actually enables
Containers on Windows (COWs)!

Host Compute Service (HCS) Shim is an API

bridge (shim) that on one end exposes Docker-
friendly API (like runc does), while on the other HCS
side converts those calls into HCS friendly
commands. HCS Shim is available on GitHub and
Shim
currently has APIs for Go (needed for Docker)
and C#.

containerd is a well known container lifecycle

containerd
manager available both for Windows and Linux
systems. It provides familiar APIs for Containers
to interact with. Fun fact: It was created for
Docker and then donated to OCI initiative!

Containers that are being executed HAVE TO be Containers

OCI compliant. Given that most of today's solutions must be
OCI
are OCI compliant, this is a no brainer. OCI, and
especially Runtime and Image Specs guarantee the
smooth execution regardless of the platform. Compliant

BitesizedEngineering.com
 Container
Networking
[ and how they communicate at all ]

Isolating your app from the rest of the system is

amazing! It allows for tons of scenarios where you
don't have to think about overlaps of any kind. But
one thing certain - without ability to communicate
to outside world, they are pretty much useless.
Let's talk about Container Networking.

1. Container Runtime
doesn't say anything
CNI about Networking and
how it should work!
Containe Plugin
r Runtime
2. That's where
Networking Specs
Jump In

hcsshim 3. Container Network

Interface (CNI) is one of
the Frameworks that
Host Host dictate the "HOW"
Compute Networking
Service Service
4 . CNI introduces concept of
"Plugins" that enhance
Host's
Containers by introducing
Kernel
Networking Capabilities

Container Network Interface (CNI)

specification introduces concept of
CNI Spec "Plugins", which are nothing more than
dictates the programs that get executed with
"HOW" defined inputs, and are expected to set
up the networking interfaces inside the
container.

Step 1 Step 2
Add
config
starts file
Network
Interface

Runtime Container CNI CNI Container

Engine Plugin

1.Network Address Translation (NAT)

CNI Plugins
available For 2. Software Defined Networking
Windows Containers: (SDN) Bridge
3. SDN Overlay

BitesizedEngineering.com
 Container Network
Interface (CNI)
[ make containers feel connected ]
Hello?

Your Container has

internet
NO Network
Connectivity Out of
your the box!
container other
container
Set up No prob! Use this OK!
Network! What you
need?
config file! On it!

config
json

Container CNI
Runtime Framework

ADD network "vnet",

with single NIC...
name: "vnet"
plugins: {
type: "nat"
ipam: {
subnet: 10.0.0.0/8 CHECK if all is
OK?
gateway: 10.0.0.1
routes: ...
etc. your happily
connected
container
config
json

Commonly referred to as
"Infrastructure-as-a-Code", CNI outlines
Describe & a FRAMEWORK that allows you to

Apply
SPECIFY how you want your network to
look like (e.g. this many adapters, and
Approach have them bridged, and what not) and it
also gets called by Runtime in order to
do the actual work of APPLYING things
specified in config.

CNI doesn't say ANYTHING on HOW

the stuff is to be applied to container. It

CNI Uses
just specifies the FORMAT in which
config (or manifest) has to be. After
that, it really relies on plugins, which are "Plugins" to
really just executable files that take do the dirty
config and container id on input and work
then it's up to them to actually DO the
network setup. It's quite flexible!

Those are THE ONLY four commands

ADD, DEL, that CNI will use to communicate to
CHECK & Plugins. ADD to add a new interface,

VERSION
DEL to remove it, CHECK to validate it
and VERSION returns CNI protocol
version being used. That's all there is!

BitesizedEngineering.com
 What's inside the
Container Image?
[ penetrating the perimeter ]

You've likely heard of Container Images. If not -

simplest way to think of them is like "templates" for
creating containers. You "pull an image" and then
you instantiate one or more containers out of it.
But what's INSIDE the image? Let's talk about it!

Blueprint Dependencies, Container

Executables, Image
etc.

Images are often compared to "classes"

All you need in OOP languages. And that's true.
Images carry all the necessary files that
to start a you need to instantiate an actual living

Container
instance - a container. In case of
Windows Containers - that means
Windows OS + your executables!

Great thing about specifications is

that they specify how stuff should
work. OCI Image spec CLEARLY
OCI Image Spec
dictates WHAT you need to have clearly defines
inside the Image, which blueprints
must be there and how it has to be the content
laid out.

You don't have to trust me but go and

look for yourself. Check what's inside the

You can image! Use "docker export" or

containerd's "export" functionality
extract image (google it!). Just execute "docker export
IMAGE-url" and you will end up with a
content! .tar ball, which is nothing more than an
archive that you can extract and look
into. It's just files and folders inside!

"github/wagoodman/dive" is an amazing
graphical tool that you can use to literally
explore the contents of images.
Check out
Usecases are many - from sneak-peaking
to trying to compress the images. Try it
the "dive"
out! tool
You should know:
At this moment (January 2022), there are 15 Container Runtimes available
and not all of them are OCI compliant (e.g. Firecracker is not). Info that I
shared here is valid for those that ARE OCI compliant (Docker, containerd,
etc.) and for the rest feel free to look up their own docs. BitesizedEngineering.com
 Inspecting Container
Image Content
[ and getting our hands dirty ]

Previous article gave you a bird's eye view of what's

inside Container Images and how you can export
and inspect the content by yourself. This one will
provide a bit more details on WHAT each file and
folder is and what's the purpose of it.

I usually like to say that what Image

The WHAT contains is "The WHAT" (i.e. the actual
binaries and dependencies of app you

and the HOW are containerizing) and "The HOW" - a

description (a blueprint if you will) to the
runtime on HOW your container needs to
behave (e.g which process to start).

The "HOW" Extract any image and you will find two files -
"oci-layout" and "index.json". Former is not
much interesting. It communicates to runtime
that this is an OCI compliant image, and
specifies which version is it. Latter one is way
more interesting! Index.json contains list of
Manifests (for mortals - simple config files in
JSON format) for each platform (Linux,
oci-layout & Windows, etc.) that describe how the container
index.json should behave. Google "OCI Image Index
Specification" for details!

The "WHAT"
Per OCI Image spec, there's another mandatory
folder - "blobs". And this one keeps the actual
interesting bits! Inside this folder you will find
bunch of FILES whose names are hashes of the
content itself (called "digests" but more on that
in future). These files are either TEXT files (i.e.
various Manifests) or actual .tar balls (e.g. a

blobs\ Windows OS itself, your app's source code, etc.).

You can really untar (tar -xvf) them and check
folder for yourself!

The Meat! You probably heard of the "Image Layers"

before. It's actually more about "Filesystem
Layers" and there will be a separate article
dedicated to it! For now, what you need to know
is that Containers use a concept of "Layering" in
order to build an actual file system inside the
container. It's really simple - think of it as
stacking stuff on top of each other - first you
add core OS' files, then you add some optional
Filesystem packages (e.g. install some dependencies), then
Layers you add your app's binaries, etc. It's all about
STACKING the layers of files and them unifying
them in a single layout! More about it in future
articles!
BitesizedEngineering.com
 Container's Image
Layers Explained
[ and how they are combined ]

If you ever worked with Containers, you likely heard

about "Layers". You also likely know that whatever
you write in Container gets wiped out once it dies.
So, what's happening behind the scenes? Let's see!

ayer
W L
R/

Union
Your App
Mount!
A d ditio n al A pps
(google it!)
W in d o w s O S

Containers use a concept of Layers.

File system is built Each Layer contains SOME files (e.g.
Layer 1 is Windows OS, Layer 2 could
by combining Files be some additional apps, and Layer 3 is
from Multiple your app) and by stacking them on top
of each other, and taking a UNION, you
Image Layers end up with a Filesystem you see inside
your container.
Ephemeral means that whatever you
write inside of container will be
removed once the container exits. This
is achieved by container runtime adding Thin Read/Write
ADDITIONAL read/write layer on top of Layer on Top
your app. And once the container exits -
this layer vanishes and no changes are makes containers
persisted. This means you can write all Ephemeral
you want inside container and your
original files won't be affected!

If you open up the blobs\ folder, you'll see

bunch of filenames that resemble some
Layers are in hashes. They in fact ARE hashes of the
Content actual bytes of each Layer (e.g. sha256()
of Windows OS binaries). This means that
Addressable each Layer is "Addressable" by its content"
Storage (CAS) (i.e. your bits are your unique signature).
These Digests turn out to be a great way
both for Reusing & Referencing layers!

If you try modifying a file from any layer

(e.g. a Windows' System DLL from Base
Layer), this file will first be COPIED to
the top layer (the one that makes stuff Copy-on-Write
ephemeral) and your modifications will Prevents
be made against that COPY. This
ensures that whatever change you
modifications of
made will NOT be carried to original original files
Layers. This also makes WRITES a bit
slower than on regular File System!
BitesizedEngineering.com
 Why use
Containers?
[ so that you can convince your boss to use 'em ]

Container images are

crafted for distribution. Build
the Image once, Upload to
Image Repository, and
deploy to as many places as
Distribution you want!

Containers isolate your

App from the rest of the
system. Whatever happens
in Container stays in
Container.
Isolation

Images are built by

stacking multiple Layers. If
anything goes wrong in
one layer, you can simply
revert to previous state.
Versioning
Long-gone are the days
where you of monitor &
scaling each service!
Automated solutions like
Service Fabric &
Kubernetes do all the dirty Orchestration
work for you!

Cloud changed everything

and Containers are just
built for the cloud. Build
once, scale many times
Cloud and don't worry about the
downtimes!
Friendly
BitesizedEngineering.com
 What is Container
Runtime Engine?
[ driving engine explained ]

I keep mentioning "Container Runtime" but I never

actually explained it. Container Runtime is to
Containers the same thing that Car Engine is to rest
of the car. It's the thing that makes things happen.

Runtime Engine
provides bits and
pieces needed to
connect all the moving
Container parts for running
Runtime Engine containers!
Container

Runtime
Engine

You can have Strictly speaking, you don't need Runtime

Engine in order to achieve what you get with
"containers" Containers. You could do all or parts of it by

without hand (e.g. just 'isolate' the process or create

Union Mount). But it's a waste of time and
Runtime Engine Runtime Engine bundles it all for you.

Given the sheer number of

OCI Runtime implementations, Open Container Initiative
(OCI) published a proposal on HOW TO
Specification standardize the Runtime Engines, which

provides the
interfaces they have to provide, what are
the standard errors, etc. This is a big thing
standard given that you can interchangably use
different runtimes and find the one that
suits you the best!

There are fifteen Containerd (used by Docker), CRI-O,

Firecracker, gVisor, etc. Google for "Cloud
runtime engines Native Landscape" and navigate to

available at the
Container Runtimes to see the list of all
engines. Do keep in mind that each one
moment comes with some pros & cons and not all of
them are OCI compliant!

BitesizedEngineering.com
 Why is Kubernetes?
[ also known as K8s ]

Did you know that "Kubernetes" means "Helmsman"

in ancient Greek? Or that K8s is abbreviation for K
(then 8 letters) than s (i.e. Kubernetes)? Let's talk a
bit about WHAT and WHY!

If you are running one or two App Servers,

Running
one or two DBs, and eventually some
caching solution - life's good. You can
things on monitor, scale, upgrade and deploy by
Scale is HARD hand. But what if you had tens of
thousands or hundreds of thousands of
servers to take care of? Good luck doing
that by hand!

ubernetes
K

Con
iners tai
ta
Con urope in U ners
SA
in E

iners
ta
Con Asia
in

Among other things they solve, Containers

allow you to pack your app into an Image
and deploy it anywhere you need. Just spin
up that Container Runtime Engine, pull Containers
those Images and off you go! Life's good as
solve lots of
problems!
long as you have number of services that
are manageable manually. But again - once
you go to tens of thousands of instances,
chances are that there's something
exploding at every single moment!

Kubernetes is really "just an orchestrator".

Piece of software built to monitor, scale,
K8s helps you upgrade and deploy servers & containers
manage automatically! And given that it was
containers developed by Google, it was built and
at scale! battle-tested on MASSIVE scales! Long
gone are the days where you had to do
massive things manually!

Kubernetes is most certainly the most

popular solution, but definitely not the only
There's
Service
one! Service Fabric (SF) is a rival solution
offered by Microsoft, which works just as
well! Pretty much most of Azure is actually Fabric as
running on Service Fabric. What's more, SF well!
is actually older than Kubernetes!
You should know
Kubernetes is pronounced as "koo·br·neh·teez". Some people prefer
calling it "Koo-br-neights", and although understandable, you should
likely stick to proper version ;) BitesizedEngineering.com
 How are Images
Distributed?
[ story of ACR, ECR & GCR ]

So far we covered a lot about how Containers are

started, how they are packed into images and how
they can be orchestrated on scale. But one thing we
didn't talk about is how are all of them distributed
(i.e. shared among interested parties). Let's talk
about Container Registries now!
Co
ns
the wil umer
h
ifies Pus s
cod l sim of t
mod d & e con e an ply p he c
l h d u od
v er Bui to t t
ge ainer resta ll th e
e
ho will age ry tt
he s in o rt th e
W e t new rd e
d e im egis est er t
co
th Pu
R
sh ll bit o
Pu s.

Container
Registry
(ACR, ECR,
GCR)

Image Image
Creator Consumer

Just like most of our code is distributed

via GitHub, GitLab and other providers,
Containers so are the Container Images
have their distributed via centralized registries.
own GitHub! Azure Container Registry (ACR), Elastic
Container Registry (ECR) & Google
Container Registry (GCR) are some of
the most popular hosting providers.

Remember when we spoke about Image

Layers and Content Addressable
Storage? This is one of the places where Power of
it shines! Registries allow you to push & Layers is
pull only the layers that were modified, fully utilized!
saving you time & bandwith for the ones
that haven't changed.

Given that Runtime & Image

There's an specification are standardized, it
OCI shouldn't come as a surprise that there's
standard for a standard on how Images are to be
it! distribured. And it called Open
Container Initiative (OCI) Distribution
Specification (google it!)

BitesizedEngineering.com