CLOUD COMPUTING AT AMAZON

INTRODUCTION • In mid-2000 Amazon introduced Amazon Web Services (AWS), based on the IaaS delivery model.

MODULE 1 • Infrastructure-as-a-Service (IaaS) In this model the cloud service provider offers an infrastructure consisting of compute and storage
servers interconnected by high-speed networks that support a set of services to access these
• Platform-as-a-Service (PaaS) resources.

CLOUD INFRASTRUCTURE • Software-as-a-Service (SaaS)

Amazon is a pioneer in IaaS, Google’s efforts are focused on SaaS and PaaS delivery An application developer is responsible for installing applications on a platform of his or her choice
MANAGEMENT
• •
models, and Microsoft is involved in PaaS and managing the resources provided by Amazon

• Private clouds are an alternative to public clouds. Open-source cloud computing

Cloud computing at Amazon, Cloud computing the Google perspective, Microsoft platforms such as Eucalyptus , OpenNebula, Nimbus, and OpenStack can be used as • Amazon was the first provider of cloud computing; it announced a limited public beta release of its
Windows Azure and online services, Open source software platforms for private
clouds, Cloud storage diversity and vendor lock-in, Energy use and ecological a control infrastructure for a private cloud Elastic Computing platform called EC2 in August 2006.
impact, Service level agreements, User experience and software licensing
Challenges of cloud computing, Architectural styles of cloud computing, Workflows:
• Our discussion of the cloud infrastructure with an overview of service level agreements
Coordination of multiple activities, Coordination based on a state machine model: (SLAs) and the responsibility sharing between users and cloud service providers,
The Zookeeper, The Map Reduce programming model.
followed by a brief discussion of software licensing, energy consumption, and
ecological impact of cloud computing. We conclude with a section covering user
experiences with current systems

CLOUD COMPUTING AT AMAZON CLOUD COMPUTING AT AMAZON CLOUD COMPUTING AT AMAZON

Figure 3.1 shows the palette of AWS services accessible via the Management Console • Elastic Compute Cloud (EC2) is a Web service with a simple interface for launching instances of an • A user can interact with EC2 using a set of SOAP messages and can list available AMI images, boot
application under several operating systems, such as several Linux distributions, Microsoft Windows an instance from an image, terminate an image, display the running instances of a user, display
Server 2003 and 2008, OpenSolaris, FreeBSD, and NetBSD console output, and so on.
• An instance is created either from a predefined Amazon Machine Image (AMI) digitally signed and
stored in S3 or from a user-defined image.
• The user has root access to each instance in the elastic and secure computing environment of EC2.
• The image includes the operating system, the run-time environment, the libraries, and the application The instances can be placed in multiple locations in different regions and availability zones.
desired by the user. AMI images create an exact copy of the original image but without configuration-
dependent information such as the hostname or the MAC address.
• EC2 allows the import of virtual machine images from the user environment to an instance through
• A user can:
a facility called VM import. It also automatically distributes the incoming application traffic among
(i) Launch an instance from an existing AMI and terminate an instance; multiple instances using the elastic load-balancing facility.
(ii) start and stop an instance;
(iii) create a new image; • EC2 associates an elastic IP address with an account; this mechanism allows a user to mask the
failure of an instance and remap a public IP address to any instance of the account without the
(iv) add tags to identify an image; and
need to interact with the software support team.
(v) reboot an instance.

CLOUD COMPUTING AT AMAZON CLOUD COMPUTING AT AMAZON CLOUD COMPUTING AT AMAZON

• Simple Storage System (S3) is a storage service designed to store large objects. It supports a • Elastic Block Store (EBS) provides persistent block-level storage volumes for use with Amazon • Simple DB is a nonrelational data store that allows developers to store and query data items via Web
minimal set of functions: write, read, and delete. EC2 instances. services requests. Simple DB creates multiple geographically distributed copies of each data item
and supports high-performance Web applications; at the same time, it automatically manages
• S3 allows an application to handle an unlimited number of objects ranging in size from one • A volume appears to an application as a raw, unformatted, and reliable physical disk; the
infrastructure provisioning, hardware and software maintenance, replication and indexing of data
byte to five terabytes. size of the storage volumes ranges from one gigabyte to one terabyte.
items, and performance tuning
• An object is stored in a bucket and retrieved via a unique developer-assigned key. A bucket • The volumes are grouped together in availability zones and are automatically replicated in
can be stored in a region selected by the user. each zone.
• Simple Queue Service (SQS) is a hosted message queue. SQS is a system for supporting automated
• S3 maintains the name, modification time, an access control list, and up to four kilobytes of • An EC2 instance may mount multiple volumes, but a volume cannot be shared among
workflows; it allows multiple Amazon EC2 instances to coordinate their activities by sending and
user-defined metadata for each object. multiple instances.
receiving SQS messages. Developers can access SQS through standards-based SOAP and Query
• The object name are global. Authentication mechanisms ensure that data is kept secure; • The EBS supports the creation of snapshots of the volumes attached to an instance and then interfaces. Queues can be shared with other AWS accounts and anonymously; queue sharing can
objects can be made public, and rights can be granted to other users. uses them to restart an instance. also be restricted by IP address and time-of-day
• S3 supports PUT, GET, and DELETE primitives to manipulate objects but does not support • The storage strategy provided by EBS is suitable for database applications, file systems, and
primitives to copy, rename, or move an object from one bucket to another. Appending to applications using raw data devices
• CloudWatch is a monitoring infrastructure used by application developers, users, and system
an object requires a read followed by a write of the entire object
administrators to collect and track metrics important for optimizing the performance of applications
• The Amazon S3 SLA guarantees reliability. S3 uses standards-based REST and SOAP interfaces; and for increasing the efficiency of resource utilization
the default download protocol is HTTP
 CLOUD COMPUTING AT AMAZON CLOUD COMPUTING AT AMAZON CLOUD COMPUTING:GOOGLE PERSPECTIVE
• Virtual Private Cloud (VPC) provides a bridge between the existing IT infrastructure of an • Elastic Beanstalk, a service that interacts with other AWS services, including EC2, S3, SNS,
• Google’s effort is concentrated in the area of Software-as-a-Service (SaaS). Services such as Gmail,
organization and the AWS cloud. The existing infrastructure is connected via a virtual private Elastic Load Balance, and Auto Scaling, automatically handles the deployment, capacity
Google Drive, Google Calendar, Picasa, and Google Groups are free of charge for individual users
network (VPN) to a set of isolated AWS compute resources. VPC allows existing management provisioning, load balancing, Auto Scaling, and application monitoring functions.
and available for a fee for organizations. These services are running on a cloud and the data for
capabilities such as security services, firewalls, and intrusion detection systems to operate
• The service automatically scales the resources as required by the application, either up, or these services is stored in data centers on the cloud.
seamlessly within the cloud.
down based on default Auto Scaling settings.
• Some of the management functions provided by the service are:
• The Gmail service hosts emails on Google servers and, provides a Web interface to access them.
• Auto Scaling exploits cloud elasticity and provides automatic scaling of EC2 instances. The
(i) deployment of a new application version (or rollback to a previous version); Google Docs is Web-based software for building text documents, spreadsheets, and presentations.
service supports grouping of instances, monitoring of the instances in a group, and defining
triggers and pairs of CloudWatch alarms and policies, which allow the size of the group to be (ii) access to the results reported by CloudWatch monitoring service;
scaled up or down. Typically, a maximum, a minimum, and a regular size for the group are
(iii) email notifications when application status changes or application servers are added or • Google Calendar is a browser-based scheduler; it supports multiple calendars for a user, the ability to
specified.
removed; and share a calendar with other users, the display of daily/weekly/monthly views, and the ability to search
events. Event reminders can be received via SMS, desktop popups, or emails
(iv) access to server login files without needing to login to the application servers.

• Picasa is a tool to upload, share, and edit images; it provides 1 GB of disk space per user free of
charge. Users can add tags to images and attach locations to photos using Google Maps. Google
Groups allows users to host discussion forums to create messages online or via email

CLOUD COMPUTING:GOOGLE PERSPECTIVE CLOUD COMPUTING: AZURE AND ONLINE SERVICES CLOUD COMPUTING: AZURE AND ONLINE SERVICES
• Google is also a leader in the Platform-as-a-Service (PaaS) space. AppEngine is a developer • Azure and Online Services are, respectively, PaaS and SaaS cloud platforms from Microsoft. Windows • Compute, which provides a computation environment; Storage for scalable storage; and Fabric
platform hosted on the cloud. Initially it supported only Python, but support for Java was added later. Azure is an operating system, SQL Azure is a cloud-based version of the SQL Server, and Azure Controller, which deploys, manages, and monitors applications; it interconnects nodes consisting of
The database for code development can be accessed with Google Query Language (GQL) with a AppFabric (formerly .NET Services) is a collection of services for cloud applications. servers, high-speed connections, and switches
SQL-like syntax
• Windows Azure has three core components
• The Content Delivery Network (CDN) maintains cache copies of data to speed up computations. The
• Google Base is a service allowing users to load structured data from different sources to a central Connect subsystem supports IP connections between the users and their applications running on
repository that is a very large, self-describing, semi-structured, heterogeneous database. It is self Windows Azure. The API interface to Windows Azure is built on REST, HTTP, and XML
describing because each item follows a simple schema: (item type, attribute names).

• The computations carried out by an application are implemented as one or more roles; an
• Google Drive is an online service for data storage that has been available since April 2012. It gives application typically runs multiple instances of a role. We can distinguish
users 5 GB of free storage and charges $4 per month for 20 GB
(i) Web role instances used to create Web applications;
(ii) Worker role instances used to run Windows-based code; and
• Google has also redefined the laptop with the introduction of the Chromebook, a purely Web-
(iii) VM role instances that run a user-provided Windows Server 2008 R2 image
centric device running Chrome OS. Cloud-based applications, extreme portability, built-in 3G
connectivity, almost instant-on, and all-day battery life are the main attractions of this device with a
keyboard

OPEN-SOURCE SOFTWARE PLATFORMS FOR PRIVATE OPEN-SOURCE SOFTWARE PLATFORMS FOR PRIVATE
CLOUD COMPUTING: AZURE AND ONLINE SERVICES CLOUDS CLOUDS
• Scaling, load balancing, memory management, and reliability are ensured by a fabric controller. The
• Open source cloud computing platforms such as Eucalyptus, OpenNebula, and Nimbus can be used • Eucalyptus supports several operating systems including CentOS 5 and 6, RHEL 5 and 6, Ubuntu 10.04
fabric controller decides where new applications should run; it chooses the physical servers to
as a control infrastructure for a private cloud. LTS, and 12.04 LTS.
optimize utilization using configuration information uploaded with each Windows Azure application.

• Schematically, a cloud infrastructure carries out the following steps to run an application: • The components of the system are:
• The configuration information is an XML-based description of how many Web role instances, how
many Worker role instances, and what other resources the application needs. The fabric controller • Retrieves the user input from the front end. • Virtual machine. Runs under several VMMs, including Xen, KVM, and Vmware.
uses this configuration file to determine how many VMs to create.
• Retrieves the disk image of a VM from a repository. • Node controller. Runs on every server or node designated to host a VM and controls the activities
of the node. Reports to a cluster controller.
• Locates a system and requests the VMM running on that system to set up a VM
• Blobs, tables, queues, and drives are used as scalable storage. A blob contains binary data; a • Cluster controller. Controls a number of servers. Interacts with the node controller on each server
• Invokes the DHCP7 and the IP bridging software to set up a MAC and IP address for the VM.
container consists of one or more blobs. Blobs can be up to a terabyte and they may have to schedule requests on that node. Cluster controllers are managed by the cloud controller.
associated metadata (e.g., the information about where a JPEG photograph was taken).
• Cloud controller. Provides the cloud access to end users, developers, and administrators. It is
accessible through command-line tools compatible withEC2 and through a Web-based
Dashboard. Manages cloud resources, makes high-level scheduling decisions, and interacts with
• The Microsoft Azure platform currently does not provide or support any distributed parallel computing
cluster controllers.
frameworks, such as MapReduce, Dryad, or MPI, other than the support for implementing basic
queue-based job scheduling
 OPEN-SOURCE SOFTWARE PLATFORMS FOR PRIVATE OPEN-SOURCE SOFTWARE PLATFORMS FOR PRIVATE
CLOUDS CLOUDS
• The procedure to construct a virtual machine:
• Storage controller. Provides persistent virtual hard drives to applications. It is the correspondent of • The eucatools front end is used to request a VM.
EBS. Users can create snapshots from EBS volumes. Snapshots are stored in Walrus and made
available across availability zones. • The VM disk image is transferred to a compute node.
• Storage service (Walrus). Provides persistent storage and, similarly to S3, allows users to store • This disk image is modified for use by the VMM on the compute node.
objects in buckets. • The compute node sets up network bridging to provide a virtual network interface controller
(NIC) with a virtual Media Access Control (MAC) address.
• The system supports a strong separation between the user space and the administrator space; users • In the head node the DHCP is set up with the MAC/IP pair.
access the system via a Web interface, whereas administrators need root access. • VMM activates the VM.
• The user can now ssh directly into the VM.
• The system supports a decentralized resource management of multiple clusters with multiple cluster
controllers, but a single head node for handling user interfaces. It implements a distributed storage
system.

OPEN-SOURCE SOFTWARE PLATFORMS FOR PRIVATE OPEN-SOURCE SOFTWARE PLATFORMS FOR PRIVATE OPEN-SOURCE SOFTWARE PLATFORMS FOR PRIVATE
CLOUDS CLOUDS CLOUDS
• Open-Nebula is a private cloud with users actually logging into the head node to access cloud • Nimbus is a cloud solution for scientific applications based on the Globus software. • Table summarizes the features of the three systems.
functions.
• The system is centralized and its default configuration uses NFS (Network File System).
• The system inherits from Globus the image storage, the credentials for user authentication, and
• The procedure to construct a virtual machine consists of several steps: the requirement that a running Nimbus process can ssh into all compute nodes.
(i) the user signs into the head node using ssh;
(ii) the system uses the onevm command to request a VM; • Customization in this system can only be done by the system administrators
(iii) the VM template disk image is transformed to fit the correct size and configuration within
the NFS directory on the head node;
(iv) the oned daemon on the head node uses ssh to log into a compute node;
(v) the compute node sets up network bridging to provide a virtual NIC with a virtual MAC; • The conclusions of the comparative analysis are as follows: Eucalyptus is best suited for a large
corporation with its own private cloud because it ensures a degree of protection from user malice
(vi) the files needed by the VMM are transferred to the compute node via the NFS; and mistakes.
(vii) the VMM on the compute node starts the VM; and • OpenNebula is best suited for a testing environment with a few servers.
(viii) the user is able to ssh directly to the VM on the compute node • Nimbus is more adequate for a scientific community less interested in the technical internals of
the system than with broad customization requirements

CLOUD STORAGE DIVERSITY AND VENDOR LOCK-IN CLOUD STORAGE DIVERSITY AND VENDOR LOCK-IN CLOUD STORAGE DIVERSITY AND VENDOR LOCK-IN
• A solution to guarding against the problems posed by the vendor lock-in is to replicate the data to • The disk controller distributes the sequential blocks of data to the physical disks and computes a
• There are several risks involved when a large organization relies solely on a single cloud provider. multiple cloud service providers. Straightforward replication is very costly and, at the same time, parity block by bit-wise XOR-ing of the data blocks.
Cloud services may be unavailable for a short or even an extended period of time. Such an poses technical challenges.
interruption of service is likely to negatively impact the organization. The potential for permanent
• Another solution could be based on an extension of the design principle of a RAID-5 system used
data loss in case of a catastrophic system failure poses an equally great danger. • The parity block is written on a different disk for each file to avoid the bottleneck possible when all
for reliable data storage.
parity blocks are written to a dedicated disk, as is done in the case of RAID-4 systems.
• A RAID-5 system uses block-level stripping with distributed parity over a disk array, as shown in
• Last but not least, a Cloud Service Provider (CSP) may decide to increase the prices for service Figure
and charge more for computing cycles, memory, storage space, and network bandwidth than • This technique allows us to recover the data after a single disk loss.
other CSPs. The alternative in this case is switching to another provider. Unfortunately, this solution • For example, if Disk 2 in Figure is lost, we still have all the blocks of the third file, c1, c2, and c3, and
could be very costly due to the large volume of data to be transferred from the old to the new we can recover the missing blocks for the others as follows:
provider.
• a2 = (a1) XOR (aP) XOR (a3)
• b2 = (b1) XOR (bP) XOR (b3)
• Reliability is a major concern, and here we discuss a solution that addresses both avoidance of
vendor lock-in and storage reliability. • d1 = (dP) XOR (d2) XOR (d3)
 SERVICE- LEVEL AGREEMENTS SERVICE- LEVEL AGREEMENTS SERVICE- LEVEL AGREEMENTS
• A service-level agreement (SLA) is a negotiated contract between two parties, the customer and • An SLA records a common understanding in several areas: • The common metrics specified by an SLA are service-specific.
the service provider. The agreement specifies the services that the customer receive rather than
(i) services,
how the service provider delivers the services.
(ii) priorities, • For example, the metrics used by a call center usually are:
(iii) responsibilities, (i) abandonment rate: percentage of calls abandoned while waiting to be answered;
• The objectives of the agreement are:
(iv) guarantees, and (ii) average speed to answer: average time before the service desk answers a call;
• Identify and define customers’ needs and constraints, including the level of resources, security,
timing, and quality of service. (v) warranties. (iii) time service factor: percentage of calls answered within a definite time frame;
• Provide a framework for understanding. A critical aspect of this framework is a clear definition (iv) first-call resolution: percentage of incoming calls that can be resolved without a callback;
of classes of service and costs. and
• An agreement usually covers: services to be delivered, performance, tracking and reporting,
• Simplify complex issues; for example, clarify the boundaries between the responsibilities of the problem management, legal compliance and resolution of disputes, customer duties and (v) turnaround time: time to complete a certain task
clients and those of the provider of service in case of failures. responsibilities, security, handling of confidential information, and termination.
• Reduce areas of conflict. • The common metrics specified by an SLA are service-specific. For example, the metrics used by a
call center usually are: (i) abandonment rate: percentage of calls abandoned while waiting to
• Encourage dialogue in the event of disputes.
be answered; (ii) average speed to answer: average time before the service desk answers a call;
• Eliminate unrealistic expectations. (iii) time service factor: percentage of calls answered within a definite time frame; (iv) first-call
resolution: percentage of incoming calls that can be resolved without a callback; and (v)
turnaround time: time to complete a certain task

USER EXPERIENCE USER EXPERIENCE USER EXPERIENCE

• The main user concerns are security threats, the dependence on fast Internet connections that • The suggested solutions to these problems are as follows: • A broad set of concerns identified by the NIST working group on cloud security includes:
forced version updates, data ownership, and user behavior monitoring. All users reported that
• SLAs and tools to monitor usage should be deployed to prevent abuse of the cloud; • Potential loss of control/ownership of data.
trust in the cloud services is important.
• data encryption and security testing should enhance the API security; • Data integration, privacy enforcement, data encryption.
• an independent security layer should be added to prevent threats caused by malicious • Data remanence after deprovisioning.
• The security threats perceived by this group of users are:
insiders;
• Multitenant data isolation.
(i) abuse and villainous use of the cloud;
• strong authentication and authorization should be enforced to prevent account hijacking;
• Data location requirements within national borders.
(ii) APIs that are not fully secure;
• data decryption in a secure environment should be implemented to prevent data leakage;
• Hypervisor security.
(iii) malicious insiders; and
• Audit data integrity protection.
(iv) account hijacking; • Compart mentalization of components and firewalls should be deployed to limit the negative
effect of resource sharing. • Verification of subscriber policies through provider controls.
(iv) data leaks; and
• Certification/accreditation requirements for a given cloud service
(v) issues related to shared resources.

• Identity theft and privacy were major concerns for about half of the users questioned; availability,
liability, and data ownership and copyright were raised by a third of respondents.

SOFTWARE LICENSING CHALLENGES FOR CLOUD COMPUTING

• Software licensing for cloud computing is an enduring problem. The Software-as-a-Service (SaaS)
deployment model is gaining acceptance because it allows users to pay only for the services
CHALLENGES FOR CLOUD COMPUTING • Reliability is also a major concern;
they use.
• The challenges are greatly amplified due to • node failures are to be expected whenever a large number of nodes cooperate for the
computations.
• A commercial product is elasticLM, which provides license and billing for Web-based services. The  the scale of the system,
architecture of the elasticLM license service has several layers: coallocation, authentication, • Choosing an optimal instance from those offered by the cloud infrastructure is another critical
administration, management, business, and persistency.  its distributed nature, and factor to be considered.
 the fact that virtually all applications are data-intensive • cost considerations also play a role in the choice of the instance type.
• The authentication layer authenticates communication between the license service and the • The shared infrastructure could also have a negative impact in Performance isolation • Efficiency, consistency, and communication scalability are major concerns for an application
billing service as well as the individual applications; the persistence layer stores the usage records. developer.
 The performance of virtual machines fluctuates based on the load, the infrastructure services,
• Indeed due to shared networks and unknown topology, cloud infrastructures exhibit internode
and the environment, including the other users. latency and bandwidth fluctuations that affect application performance
• The main responsibility of the business layer is to provide the licensing service with the licenses
prices, and the management coordinates various components of the automated billing service. • Security isolation is also challenging on multitenant systems. • Data storage plays a critical role in the performance of any data-intensive application; the
organization of the storage, the storage location, and the storage bandwidth must be carefully
analyzed to lead to optimal application performance
• When a user requests a license from the license service, the terms of the license usage are
negotiated and they are part of an SLA document. The SLA describes all aspects of resource
usage, including the ID of application, duration, number of processors, and guarantees, such as
the maximum cost and deadlines
 CHALLENGES FOR CLOUD COMPUTING ARCHITECTURAL STYLES FOR CLOUD COMPUTING ARCHITECTURAL STYLES FOR CLOUD COMPUTING
• Many data-intensive applications use metadata associated with individual data records; • Cloud computing is based on the client-server paradigm. • Several other considerations must be analyzed before deciding on the architectural style of an
application.
• example, the metadata for an MPEG audio file may include the name of the song, the singer, • The vast majority of cloud applications take advantage of request/response communication between
recording information, and so on. clients and stateless servers. • The term neutrality refers to the ability of the application protocol to use different transport
protocols such as TCP or UDP and, in general, to run on top of a different protocol stack.
• Metadata should be stored for easy access, and the storage should be scalable and reliable. • A stateless server views a client request as an independent transaction and responds to it.
• For example, we shall see that SOAP can use TCP but also UDP, SMTP, 3 or JMS4 as transport
• Another important consideration for the application developer is logging. • A stateless system is simpler, more robust, and scalable. A client does not have to be concerned
vehicles.
with the state of the server.
• Performance considerations limit the amount of data logging, whereas the ability to identify the • Extensibility refers to the ability to incorporate additional functions, such as security.
source of unexpected results and errors is helped by frequent logging. • Example, a basic Web server is stateless; it responds to an HTTP request without maintaining a history of
past interactions with the client. • Independence refers to the ability to accommodate different programming styles.
• Logging is typically done using instance storage preserved only for the lifetime of the instance.
• The client, a browser, is also stateless since it sends requests and waits for responses. • The Common Object Request Broker Architecture (CORBA) was developed to allow networked
• Thus, measures to preserve the logs for a postmortem analysis must be taken. applications developed in different programming languages and running on systems with different
• The Hypertext Transfer Protocol (HTTP) used by a browser to communicate with the Web server is a
• Another challenge awaiting resolution is related to software licensing architectures and system software to work with one another
request/response application protocol. HTTP uses the Transport Control Protocol (TCP), a
connection-oriented and reliable transport protocol.
• The use of TCP ensures reliable delivery of large objects but exposes the Web servers to denial-of-
service attacks when malicious clients fake attempts to establish a TCP connection and force the
server to allocate space for the connection

ARCHITECTURAL STYLES FOR CLOUD COMPUTING THE ZOOKEEPER THE ZOOKEEPER

• The Simple Object Access Protocol (SOAP) is an application protocol developed in 1998 for Web • ZooKeeper is a distributed coordination service which uses high-throughput and low-latency • Figures show that a read operation directed to any server in the pack returns the same result,
applications; service. whereas the processing of a write operation is more involved;
• its message format is based on the Extensible Markup Language (XML). • The organization of the service is shown
• SOAP uses TCP and, more recently, UDP transport protocols. It can also be stacked above other
application layer protocols such as HTTP, SMTP, or JMS.
• The processing model of SOAP is based on a network consisting of senders, receivers,
intermediaries, message originators, ultimate receivers, and message paths. SOAP is an
underlying layer of Web Services • The servers in the pack communicate with one another and elect a leader.
• The Web Services Description Language (WSDL) was introduced in 2001 as an XML-based grammar • A database is replicated on each one of them and the consistency of the replicas is • The servers elect a leader, and any follower receiving a request from one of the clients connected to
to describe communication between endpoints of a networked application. maintained. it forwards it to the leader.
• Representational State Transfer (REST) is a style of software architecture for distributed hypermedia • A client can connect to any server of the pack. A client uses TCP to connect to a single • The leader uses atomic broadcast to reach consensus. When the leader fails, the servers elect a new
systems. server. leader.
• REST supports client communication with stateless servers. It is platform- and language • Through the TCP connection a client sends requests and receives responses and watches • The system is organized as a shared hierarchical namespace similar to the organization of a file
independent, supports data caching, and can be used in the presence of firewalls. REST almost events. A client synchronizes its clock with the server. system.
always uses HTTP to support all four Create/Read/Update/Delete (CRUD) operations. It uses GET,
• If the server fails, the TCP connections of all clients connected to it time out and the clients • A name is a sequence of path elements separated by a backslash. Every name in Zookeper’s
PUT, and DELETE to read, write, and delete the data, respectively
detect the failure of the server and connect to other servers. namespace is identified by a unique path.

THE ZOOKEEPER THE ZOOKEEPER THE ZOOKEEPER

• In ZooKeeper the znodes, the equivalent of the inodes of a file system, can have data • The ZooKeeper service guarantees:
• The messaging layer is responsible for the election of a new leader when the current leader fails.
associated with them.
1. Atomicity. A transaction either completes or fails.
• The messaging protocol uses packets (sequences of bytes sent through a FIFO channel), proposals
• Indeed, the system is designed to store state information. The data in each node includes
2. Sequential consistency of updates. Updates are applied strictly in the order in which (units of agreement), and messages (sequences of bytes atomically broadcast to all servers).
version numbers for the data, changes of Access Control Lists and time stamps.
they are received.
• A message is included in a proposal and it is agreed on before it is delivered. Proposals are agreed
• A client can set a watch on a znode and receive a notification when the znode changes.
3. Single system image for the clients. A client receives the same response regardless of on by exchanging packets with a quorum of servers,
This organization allows coordinated updates.
the server it connects to.
• An atomic messaging system keeps all the servers in a pack in synch. This system guarantees
• The data retrieved by a client also contains a version number. Each update is stamped with
4. Persistence of updates. Once applied, an update persists until it is overwritten by a client.
a number that reflects the order of the transition. (a) reliable delivery: if message m is delivered to one server, it will be eventually delivered to all
5. Reliability. The system is guaranteed to function correctly as long as the majority of
• The data stored in each node is read and written atomically. A read returns all the data servers;
servers function correctly
stored in a znode, whereas a write replaces all the data in the znode
(b) total order: if message m is delivered before message n to one server, m will be delivered before
• To reduce the response time, read requests are serviced from the local replica of the server
• Updates are logged to disk for recoverability, and writes are serialized to disk before they are
that is connected to the client. n to all servers; and
applied to the in-memory database that contains the entire tree
• When the leader receives a write request, it determines the state of the system where the (c) causal order: if message n is sent after m has been delivered by the sender of n, then m must be
write will be applied and then it transforms the state into a transaction that captures this new
ordered before n.
state
 THE MAPREDUCE PROGRAMMING MODEL
THE ZOOKEEPER THE MAPREDUCE PROGRAMMING MODEL • When a user program invokes the MapReduce
function, the following sequence of actions take place

The application programming interface (API) to the ZooKeeper service is very simple and consists of • (1) An application starts a master instance and M
• • MapReduce is based on a very simple idea for parallel processing of data-intensive
seven operations: worker instances for the Map phase and, later, R
applications supporting arbitrarily divisible load sharing.
worker instances for the Reduce phase.
• create – add a node at a given location on the tree. • First, split the data into blocks, assign each block to an instance or process, and run • (2) The master partitions the input data in M segments.
• delete – delete a node. these instances in parallel.
• (3) Each Map instance reads its input data segment
• get data – read data from a node. • Once all the instances have finished, the computations assigned to them start the and processes the data.
• set data – write data to a node. second phase: Merge the partial results produced by individual instances • (4) The results of the processing are stored on the local
• get children – retrieve a list of the children of the node. • MapReduce is a programming model inspired by the Map and the Reduce primitives of disks of the servers where the Map instances run.
the LISP programming language. • (5) When all Map instances have finished processing
• synch – wait for the data to propagate.
• It was conceived for processing and generating large data sets on computing clusters. As their data, the R Reduce instances read the results of
• The system also supports the creation of ephemeral nodes, which are nodes that are created when a the first phase and merge the partial results.
session starts and deleted when the session ends. a result of the computation, a set of input pairs <key, value> is transformed into a set of
output pairs <key, value>. • (6) The final results are written by the Reduce instances
to a shared storage server.
• For example, one can process logs of Web page requests and count the URL access
frequency. The Map function outputs the pairs <URL, 1> and the Reduce function produces • (7) The master instance monitors the Reduce instances
and, when all of them report task completion, the
the pairs <URL, totalcount>.
application is terminated.

THE MAPREDUCE PROGRAMMING MODEL

• The system is fault tolerant.
• For each Map and Reduce task, the master stores the state (idle, in progress, or completed) and the
identity of the worker machine.
• The master pings every worker periodically and marks the worker as failed if it does not respond.
• A task in progress on a failed worker is reset to idle and becomes eligible for rescheduling.
• The master writes periodic checkpoints of its control data structures and, if the task fails, it can be
restarted from the last checkpoint.
• The data is stored using GFS, the Google File System.