AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

4th Slide Set Cloud Computing

Prof. Dr. Christian Baun

Frankfurt University of Applied Sciences

(1971–2014: Fachhochschule Frankfurt am Main)
Faculty of Computer Science and Engineering
[email protected]

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 1/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Agenda for Today

Amazon Web Services (AWS)

Reasons for using the AWS
Examples of applications that use the AWS
Elastic Compute Cloud (EC2)
Elastic Block Store (EBS)
Elastic Load Balancing (ELB)
Simple Storage Service (S3)
Google Cloud Storage and further alternative service offerings
Solutions for running private cloud infrastructure services
Eucalyptus
OpenStack

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 2/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Amazon Web Services (AWS)

The AWS is a collection of different public cloud services
Launched in 2002
Billed according to consumption
Services of the AWS are among others. . .
Elastic Compute Cloud (EC2) =⇒ Infrastructure service for virtual servers
Simple Storage Service (S3) =⇒ Storage service for web objects
Elastic Block Store (EBS) =⇒ Storage service for virtual storage volumes
Elastic Load Balancing (ELB) =⇒ Service for virtual load balancers
CloudWatch =⇒ Service for monitoring AWS resources
Auto Scaling =⇒ Service for scaling EC2 capacities
SimpleDB =⇒ Service for distributed database
Amazon Simple Queue Service (SQS) =⇒ Service for message queues
Amazon Mechanical Turk =⇒ HuaaS/Crowdsourcing marketplace
Attention!
Many screenshots in this slide set are from the years 2012/2013/2014
The web interfaces of cloud service providers often change
=⇒ Many screenshots are outdated! Sorry for that!
The functionality and technical terms are seldom modified

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 3/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Overview – http://aws.amazon.com

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 4/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Why AWS?

Why should a company use the AWS, instead of buying own resources?
How many resources does the company need in the future?
Scenario: A web offering of a startup company
How many resources will be consumed?
What costs will arise?
How much time is required to acquire additional resources and include
them into the infrastructure?

Without a credit card, the AWS cannot be used

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 5/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Own physical Infrastructure compared with the Cloud

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 6/61
 AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Customer Success Story: Animoto (1/2)

Users can create videos from their own pictures and music
http://animoto.com
The software analyzes the pictures and the music and generates videos
in the style of a trailer or a MTV music video
Videos can be uploaded to YouTube and exported to various formats

Image source: Google image search

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 7/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Customer Success Story: Animoto (2/2)

2006-2008: Only few users used the

service
April 2008: Facebook application
launched
750,000 new users in 3 days
At the peak, up to 25,000 people
tried to render a video in a single
hour
Slashdot effect!
Automatic adjustment of the
instances to render the videos from
2 up to 450

http://www.youtube.com/watch?v=VwDS6MexKEo

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 8/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Slashdot Effect

Linear increase of traffic is unrealistic

Huge problem for startup companies with own resources
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 9/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Customer Success Story: New York Times

2007: The New York Times wants to create PDF versions from the
articles from the years 1851-1980
The newspaper planned to make the articles from the years 1851-1922
available online for free
The raw version of the articles were 11 million
scanned images
Each article had to be composed of several TIFF
files and had to be scaled
First, 4 TB TIFF files had to be uploaded to S3
100 EC2 instances required approximately 24
hours for the calculation
Result: 1.5 TB of PDF files inside S3
https://timesmachine.nytimes.com
https://aws.amazon.com/de/blogs/aws/new-york-times/
http://open.blogs.nytimes.com/2007/11/01/self-service-prorated-super-computing-fun/
http://open.blogs.nytimes.com/2008/05/21/the-new-york-times-archives-amazon-web-services-timesmachine/

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 10/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Customer Success Story: reddit

2012: reddit has 4 billion page views per month

Scalable infrastructure based of AWS
Server capacity was doubled in minutes for President Obama’s live
Q&A session in 2012

http://www.youtube.com/watch?v=BPMNB29zDvk
Update (May 2018): The video is not online any more. . .

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 11/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS – Check your Account Activity !!! State: October 2016

Please regularly check their user account!

Login at the AWS page and check the Billing & Cost Management
page
Running lots of instances all the time quickly melts together your credit
If the credit is spend and resources are still consumed, the credit card will
be charged
The account holder is responsible for resulting costs
You can specify limits and alerts =⇒ do it!

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 12/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Amazon Elastic Compute Cloud (EC2)

Users can create, use and control virtual server instances in Amazons
data centers
Supported operating systems:
Linux: Amazon Linux, Debian, Ubuntu, SuSE, OpenSuSE, Kali, RedHat
Enterprise, CentOS, Mint, Gentoo, Fedora, RancherOS, Alpine,. . .
Windows Server: 2003/2008/2012/2016/2019
BSD: FreeBSD, NetBSD (d), MidnightBSD
Solaris: OpenSolaris (d)
MacOS X https://aws.amazon.com/de/about-aws/whats-new/2020/
11/announcing-amazon-ec2-mac-instances-for-macos/
Virtual servers are created from Amazon Machine Images (AMI)
These are like a blueprint to be used when creating new virtual servers
Amazon provides prebuilt images
Besides Amazon, many third-party vendors, such as IBM, Oracle and
SAP, provide AMIs including proprietary software packages
End users as well can create their own images for later reuse
End users can publish their AMIs and put them on the market using a
product ID (paid instances)
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 13/61
 AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

EC2 Terminology State: October 2016

EC2 provides 11 sites (regions) with resources:

Virginia, California, Oregon, Ireland, Frankfurt, Singapore, Sydney,
Tokyo, Seoul, Mumbai, Sao Paulo
Each region contains availability zones
Each availability zone is a cluster
Region Availability Zones
Virgina us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1e
California us-west-1a, us-west-1b, us-west-1c
Oregon us-west-2a, us-west-2b, us-west-2c
Ireland eu-west-1a, eu-west-1b, eu-west-1c
Frankfurt eu-central-1a, eu-central-1b
Singapore ap-southeast-1a, ap-southeast-1b
Sydney ap-southeast-2a, ap-southeast-2b, ap-southeast-2c
Tokyo ap-northeast-1a, ap-northeast-1b, ap-northeast-1c
Seoul ap-northeast-2a, ap-northeast-2c
Mumbai ap-south-1a, ap-south-1b
Sao Paulo sa-east-1a, sa-east-1b, sa-east-1c

Update June 2019: The list of availability zones did grow: Missing in this slide are Oregon, Canada, Paris, Osaka, Stockholm and
Hong Kong

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 14/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

EC2 Instance Types State: May 2018

···

···

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 15/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

EC2 - Required Steps to work with the Service (1/2)

The user needs a key pair to authenticate at its instances

Login without password (public key method)
Public keys are stored inside the instances
Private keys are stored on the users client
A new key pair can be created or an existing key pair can be used
User decides, which ports must be open
The fewer ports are opened, the better is the security
The user creates for the instance a security group, in which the required
ports are opened
The user can also use an existing security group
User decides which operating system (AMI) and which instance
type meets his requirements best
User decides which region and availability zone he prefers
The instance is created according to the decisions made before

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 16/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

EC2 - Required Steps to work with the Service (2/2)

After the virtual server has been created, a public and a private IP
address is dynamically assigned to the instance
With the public address the instance can be accessed from the Internet
With the private address it can be accessed by other instances inside the
Amazon cloud
Private and public addresses are assigned dynamically each time a new
instance is created
Dynamically assigned addresses are not suited for the long-term operation
of a server
Servers need to be restarted from time to time
Solution: elastic IP addresses
Users can assign Elastic IPs – once reserved – their own server instances
again and again

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 17/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Persistence of Data in EC2

At the termination of an instance all changes are lost

Valuable data must be stored outside the instance
Large amounts of structured data can be stored in S3
EBS provides block-based storage

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 18/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

EC2 Pricing

=⇒ https://aws.amazon.com/ec2/pricing/

On-Demand instances
Spot instances
Instances have flexible start and end times
Reserved instances
Customers can rent EC2 instances over a 1 or 3 year term to reduce their
total costs
Dedicated Hosts

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 19/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

EC2 Pricing: Internet Data Transfer State: November 2017

The import of data to AWS resources is for free

If data is copied between AWS resources it is for free if these resources
are inside the same availability zone

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 20/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

EC2 Pricing: Elastic IP und Load Balancing State: November 2017

Elastic IP adresses

Elastic Load Balancers

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 21/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

EC2 Pricing: CloudWatch State: November 2017

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 22/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Simple Monthly Calculator

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 23/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Amazon Elastic Block Store (EBS)

EBS is a part of EC2

Inside each availability zone, the users can create EBS volumes
Size: Up to 16 TB
An EBS volume implements persistent storage
A new EBS volume behaves just like an unformatted block device
an EBS volume can only be mounted to one single instance
Volume and instance must be located in the same availability zone
A volume can contain any filesystem
The way of using a volume is equal to using an USB flash drive
Note: EBS is storage for people and S3 is storage for applications
Volume snapshots can be created (and stored in S3) any time

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 24/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Pricing of EBS State: December 2020

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 25/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Management Console (EBS Volumes)

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 26/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Amazon Elastic Load Balancing (ELB) State: November 2017

ELB is a part of EC2

Users can create elastic load balancers inside each availability zone
The user assigns each of its load balancers a pool of instances
An elastic load balancer automatically distributes incoming requests to
the EC2 instances of its pool
A ELB identified failed instances inside its pool and distributes the
requests automatically to the working instances of the pool

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 27/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Amazon Simple Storage Service – S3 (1/2)

Data is stored as (web-)objects

No files or folders exist, but only objects
The size of each object can be 1 Byte to 5 TB
For each object, 2 KB metadata is stored
Each user can store an unlimited number of objects
Each object is assigned to a bucket
Buckets have unique names and contain no other buckets
=⇒ Directories are impossible
The name of an object is also called key

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 28/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Amazon Simple Storage Service – S3 (2/2)

Objects are accessible online

http://s3.amazonaws.com/bucket/object
http://bucket.s3.amazonaws.com/object
Access to buckets and objects is done via REST or SOAP (deprecated)
Objects can also be downloaded via BitTorrent

Users can specify for all their objects and buckets the access privileges
Access Control List (ACL)

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 29/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Flat Name Space of S3

S3 does not support folders

Only buckets and objects can be created
But folders can be simulated
S3Fox, Google and KOALA simulate folder by attaching _$folder$ at
the end of an objects key
Objects, which are assigned to such a folder, have a key with the naming
scheme folder/subfolder/object

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 30/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

S3 Pricing (Storage) November 2017

Standard Storage is designed for 99.999999999% durability and

99.99% availability of objects over a given year
Reduced Redundancy Storage (RRS) is designed to provide 99.99%
durability and 99.99% availability of objects over a given year
This durability level corresponds to an average annual expected loss of
0.01% of the objects
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 31/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

S3 Pricing (Storage) November 2017

Glacier is designed for 99.999999999% durability and 99.99%

availability of objects over a given year
Extremely low-cost storage service option for data archival
Stores data for as little as $0,0045 (in EU-Frankfurt) per GB per month
Optimized for data that is infrequently accessed and for which retrieval
times of several hours are suitable

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 32/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

S3 Pricing (Requests) November 2017

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 33/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

S3 Pricing (Data Transfer) November 2017

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 34/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Import/Export Disk

Helps to transfer large amounts of data into or out from the cloud

The customers sends a storage device (HDD) to Amazon

The device concent is copied by Amazon employees into a S3 bucket
File systems: NTFS, ext2, ext3 and FAT32 with a mximum size of 16 TB
Pricing per storage device: $80
Pricing for the transfer at Amazon site per hour: $2.49
https://aws.amazon.com/snowball/disk/
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 35/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Import/Export Snowball Image Source: Amazon

Amazon offers the Snowball Appliances for importing data into S3

https://aws.amazon.com/snowball/

Capacity: 50 TB
or 80 TB
10 Gbit Ethernet
interface
AES 256-bit
encryption
Price: $200 or
$250 per device
for 10 days
Each additional
day costs $15
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 36/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Similar offering – Cloud Mass Data Migration Image Source: IBM

IBM offers a similar import

service for its own IaaS
offerings
Cloud Mass Data
Migration
Capacity: 120 TB
AES 256-bit encryption
RAID-6
10 Gbit Ethernet interface
Price: $395 per device for
10 days
Each additional day: +$30
Customers can migrate 120 TB of data in seven days, with round-trip
use of UPS Next Day Air included in the overall service
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 37/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

AWS Snowmobile Image Source: AWS

Helps to transfer very large amounts of data into the cloud

Customers can transfer up to 100 PB per Snowmobile
Data is copied by Amazon employees into a S3 bucket or into Glacier
Snowmobile is a 45-foot long shipping container, pulled by a truck
Includes security personnel, GPS tracking, alarm monitoring, 24/7 video
surveillance, and an optional escort security vehicle while in transit
All data is encrypted with 256-bit encryption keys
https://aws.amazon.com/snowmobile/

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 38/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Using S3 with s3cmd

s3cmd is a simple to use command line tool for uploading, retrieving

and managing data in Amazon S3
http://s3tools.org/s3cmd
Configure login information s3cmd –configure
List own buckets s3cmd ls
Create bucket s3cmd mb s3://Bucket
Upload object s3cmd put LocalFile s3://Bucket/DistantObjekt
List content of a bucket s3cmd ls s3://Bucket
Download object s3cmd get s3://Bucket/DistantObjekt LocalFile
Erase objekt s3cmd del s3://Bucket/DistantObjekt
Erase (empty) bucket s3cmd rb s3://Bucket

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 39/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

S3 Alternatives: Google Cloud Storage https://cloud.google.com/storage/

Storage service for web objects

Interface is compatible with S3
Functionality is (almost) identical to S3
Objects are accessible online
https://storage.googleapis.com/bucket/object
https://bucket.storage.googleapis.com/object
Access to buckets and objects is done via REST or SOAP
Users can specify for all their objects and buckets the access privileges
Access Control List (ACL)
Provides the command line tool GSutil and the software service (SaaS)
Google Storage Manager
GSutil can interact with Google Cloud Storage and S3
GSutil is based on the Python library boto

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 40/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Some further S3-compatible Service Offerings

This list from November 2017 is not complete! Many more S3-compatible service offerings exist

Service Offering URL/Status

Aruba Object Storage service https://www.arubacloud.com
BetterServers Object Storage https://www.betterservers.com
e24cloud https://www.e24cloud.com/en/cloud-features/
Rackspace Cloud Files https://www.rackspace.com/de/cloud/files
Caringo Cloud Storage https://www.caringo.com
Cloudian http://www.cloudian.com
DreamHost DreamObjects https://www.dreamhost.com/cloud/storage/
Dunkel S3 https://www.dunkel.de/s3
S3FOR.ME http://www.s3for.me
Connectria Cloud Storage It is unclear if this service is still available
HP Helion Public cloud Defunct since January 2016
Host Europe Cloud Storage Defunct since end 2014
Nirvanix Defunct since September 2013

Never forget. . .
a Cloud Service Providers may modify of service offering (functionality) or even go out of business
at any time

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 41/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Solutions for running Private Cloud Infrastructure Services

Several free solutions exist run infrastructure services

CloudStack http://cloudstack.apache.org
Eucalyptus https://www.eucalyptus.cloud
Nimbus http://www.nimbusproject.org (d)
OpenNebula http://www.opennebula.org
OpenStack http://www.openstack.org

This is not a complete list!

These solutions are used mainly for the construction of private clouds
Some solutions can also be used for the construction of public cloud
services

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 42/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Eucalyptus

EUCALYPTUS – Elastic Utility Computing Architecture for Linking

Your Programs To Useful Systems
Allows execution and control of virtual instances (Xen or KVM) on
different physical resources
Developed at UC Santa Barbara
Further development by Eucalyptus Systems, Inc.
Interface compatible to AWS
EC2 + EBS + ELB + AutoScaling and S3
Use of poplar AWS-compatible tools is possible:
e.g.: S3 Curl, Elasticfox, s3cmd,. . .
Free software: GPLv3 (until 11/2017). Now BSD

In 2008 + 2009, Eucalytus was a major step forward in establishing an API standard for cloud
infrastructure services

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 43/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Eucalyptus Services

Eucalyptus consists of several UNIX

services
Cloud Controller (CLC)
Cluster Controller (CC)
Node Controller (NC)
Walrus
Storage Controller (SC)
The services communicate via web
services (SOAP+REST)
Eucalyptus infrastructures consist
of one or more sites
Redundant operation of the services CLC, CC,
Storage Controller and Walrus became a
feature with Eucalyptus v3.0 in 2011
This feature was removed with v4.2 in 2015

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 44/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Node Controller (NC)

Runs on every physical node, where

instances are planned to run
Controls the KVM hypervisor

Xen is not supported any longer since v4.0

VMware ESX(i) is not supported any longer
since v4.1

Each NC transmits information

about the utilization of their own
resources to the CC of the site
Number of virtual processors
Free memory
Free storage

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 45/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Cluster Controller (CC)

Exactly a single CC per site is

required
Controls the distribution of the
virtual machines to the NCs
Collects free resource information
from the NCs
In small infrastructures CLC and
CC usually run on the same
physical server
In each site, the NCs communicate
with the CC via a virtual network
(VLAN)
The VLAN ensures that all
instances within a site share the
same subnet
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 46/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Cloud Controller (CLC)

Exactly a single CLC per

Eucalyptus infrastructure is required
Acts as a meta-scheduler in the
cloud infrastructure
Collects resource information from
the CCs
Runs per default on the same
physical server as the storage
services Walrus and Storage
Controller

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 47/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Walrus

Storage service, which implements

the S3 REST API
Eucalyptus stores here the images
Usually runs on the physical server
which hosts the CLC/CC
Can be outsourced from the CLC
since v1.6
Walrus is not a distributing service
Operates only in single-node
mode
In order to improve the read/write
performance of the object-based
storage, Walrus can be replaced by
a Riak Cloud Storage (Cluster)
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 48/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Storage Controller

Storage service, which implements

the EBS API
In infrastructures with only a single
site, the Storage Controller usually
runs on the physical server which
hosts the CLC/CC
Can be outsourced from the CLC
since v1.6
If the infrastructure contains
multiple sites, each site has its own
storage controller

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 49/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Launch of an Instance in Eucalyptus (1/2)

1 A user or applications, which tries

to start instances, provides the CLC
these parameters:
Image
Instance type
Number of instances
2 CLC selects a CC with enough free
resources in its cluster
3 CC selects in the local cluster one
(or more) NC(s), with enough free
resources and commands the start
of the instance(s)

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 50/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Launch of an Instance in Eucalyptus (2/2)

4 If the required image is not

available on the NC, the NC
requests the image from the CLC
5 CLC transmits the image from
Walrus via an encrypted
transmission via Secure Copy
(SCP) to the NC
6 The transmission duration for
images from Walrus to the NCs
depends of:
Network technology used
Number of required transmissions
Size of the images

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 51/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Some Facts about Eucalyptus

The Installation of Eucalyptus is simple when CentOS is used

and it is horror when other Linux distributions are used
Stable operation of an Eucalyptus infrastructure is possible
If administrators are willing to invest some time. . .
Single services need to reboot from time to time
Commercial support is available
Extensions and modifications in the source code are difficult
Source code of services appears obscure
Some history. . .
HP acquired Eucalyptus systems in 9/2014
2015: Eucalpytus became a part of HPE, which separated from HP
2017: Eucalpytus became a part of DXC Technology =⇒ d (?!)

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 52/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

A new Hope for Eucalpytus?

DXC stopped developing the

product in late 2017
AppScale Systems forked the code
in 2018 and started supporting the
product and continued the
development
=⇒ AppScale ATS
https://www.eucalyptus.cloud
Latest versions:
5.0 from December 2020
4.4.6 from November 2020
4.4.5 from December 2018
https://github.com/corymbia/
eucalyptus/
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 53/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Eucalyptus Installation (the simple way on a single node)

Check the installation tutorial
https://docs.eucalyptus.cloud/eucalyptus/4.4.6/index.
html#shared/install_section.html
Create a virtual machine with CentOS 7.3 minimal
Execute:
bash <( curl - Ls https :// eucalyptus . cloud / install )

The script will ask a few questions (e.g. about spare IP addresses)
Hope the best
Eucalyptus 5
https://docs.eucalyptus.cloud/eucalyptus/5/install_guide/automated_install/

A faststart iso image „Cloud in a Box“ existed until 2015/2016

The last revision came with Eucalyptus 3.4.2 and CentOS 6
It was the most simple way to install an Eucalyptus IaaS on a single node or a cluster mode
It is not available any more
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 54/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Review of NASA regarding Eucalyptus

Source: http://www.theregister.co.uk/2010/07/20/why_nasa_is_dropping_eucalyptus_from_its_nebula_cloud/

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 55/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

2011: Ubuntu switches from Eucalpytus to OpenStack

Source: http://talkincloud.com/ubuntu-cloud-openstack-wins-eucalyptus-loses

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 56/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

OpenStack Image Source: OpenStack

Initiated by NASA and Rackspace Cloud

Supported by AMD, Dell, IBM, Intel, Red Hat, SuSE, Yahoo and more
Free software (Apache License v2.0)
Contains several services which communicate via REST

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 57/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

OpenStack – Services (1/3)

Compute (Nova)
Infrastructure service
Implements the EC2
API
Highly scalable (up to
tens of thousands of
nodes)

Object Storage (Swift)

Redundant, highly scalable (petabyte range), object-based storage service
Objects are stored on multiple hardware
Automatic replication when nodes fail or are added
Implements the S3 API

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 58/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

OpenStack – Services (2/3)

Image Service (Glance)

Service for the search,
register and request of
images
Supported image
formats: Raw, AMI,
VHD (Hyper-V), VDI
(VirtualBox), qcow2
(Qemu/KVM), VMDK
and OVF (VMWare)
Block Storage (Cinder )
Storage service for persistent block-based storage devices
Virtual storage devices can be created, erased, attached to and detached
from instances
Implements the EBS API

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 59/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

OpenStack – Services (3/3)

Identity Service
(Keystone)
Central directory of
users for the other
OpenStack services
Provides user
authentication
Can interact (query)
existing user directory
services (e.g. LDAP)
Dashboard (Horizon)
Provides a graphical web-interface for administrators and users
Networking (Neutron)
Service for managing IP addresses and distributing them to instances
Administrators specify, if all instances are connected to the same network,
or if they are separated from each other via VLAN
Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 60/61
AWS EC2 (+ EBS and ELB) S3 Private Cloud IaaS (Eucalyptus + OpenStack)

Infrastructure Services and their Compatibility to the AWS

AWS APIs implemented

Project/Solution EC2 S3 EBS (EC2)
CloudStack partly — —
Eucalyptus partly partly (Walrus) partly (SC)
Nimbus partly partly (Cumulus) —
OpenNebula partly — partly
OpenStack partly (Nova) partly (Swift) partly (Cinder)

Many free private cloud solutions exist, which implement an

object-based storage service and provide the S3 API
MinIO, Ceph-RGW, Fake S3, Riak CS, S3ninja, S3rver, Scality S3
Server,. . .

Freie Objektspeichersoftware mit S3-API, Christian Baun. iX 9/2017, P.76-79

https://www.heise.de/ix/heft/Eimerweise-3807215.html

Prof. Dr. Christian Baun – 4th Slide Set Cloud Computing – Frankfurt University of Applied Sciences – WS2021 61/61