Architectures for Distributed

Systems
Chapter 2
Distributed System Architecture
Definition:
Organization of software components across multiple machines in a distributed environment.

Purpose:

• Manage complexity of distributed systems.

• Ensure efficient interaction among components.

• Provide distribution transparency (hide details of communication & location).

Realization Models:

• Centralized (single server, multiple clients).

• Decentralized (peer-to-peer, equal roles).

• Hybrid (combination, e.g., super peers, edge servers).

Distributed System Architecture
Types:

● Software Architectures → Logical organization (interaction of components).

● System Architectures → Physical placement on machines.

Software Architectures
● Describe the logical organization of a distributed system.

● Focus on interaction of software components (how components exchange data, coordinate, and integrate).

● Provide an abstract design blueprint (independent of hardware).

● Examples: Layered style, Object-based (RPC), Data-centered (shared repository), Event-driven

(publish/subscribe).
Software Architectures
Architectural Style
• Describes a particular way to configure a collection of components and connectors.
Component
• A module with well-defined required/provided interfaces.
• Should be reusable and replaceable as long as interfaces are respected.
Connector
• Defines the communication link between components.
• Can be procedure calls, message passing, or data streams.
Architectures suitable for distributed systems:
1. Layered architectures
2. Object-based architectures
3. Data-centered architectures
4. Event-based architectures
Layered Architectures in Distributed Systems
• Components are organized in layers stacked on top of each other.
• A layer provides services to the layer above and uses services of the layer below.
• Control generally flows downwards (requests) and upwards (responses).

Characteristics

• Strict hierarchy: A layer can only interact with its immediate neighbor.
• Abstraction: Each layer hides implementation details of lower layers.
• Modularity: Changes in one layer have minimal impact on others.
• Reusability: Lower layers can be reused in different systems.

Advantages

• Clear separation of concerns (each layer has a defined role).

• Easier maintenance and scalability.
• Promotes standardization (e.g., networking stacks).

Disadvantages

• Performance overhead due to multiple layer interactions.

• Sometimes rigid – a layer might need data from non-adjacent layers.
Layered Architectures in Distributed Systems
Example

● OSI Networking Model

○ Layer 7 – Application (HTTP, FTP)

○ Layer 6 – Presentation (Data translation, encryption)
○ Layer 5 – Session (Dialog control)
○ Layer 4 – Transport (TCP/UDP)
○ Layer 3 – Network (IP, routing)
○ Layer 2 – Data Link (Ethernet, error detection)
○ Layer 1 – Physical (Cables, signals, wireless)
Object-Based Architectures
Concept

● Looser organization than layered style.

● Each object = component (encapsulation of state + behavior).

● Components interact using Remote Procedure Calls (RPCs) or method invocations.

● Naturally maps to the Client-Server model in distributed systems.

Characteristics

● Encapsulation: Internal details hidden; interaction via interfaces.

● Flexibility: Objects can interact without strict hierarchy.

● Reusability: Modular, replaceable, and portable components.

● Transparency: Remote calls look like local calls (RPC abstraction).

Object-Based Architectures
Advantages
● Promotes modularity and reusability.

● Matches object-oriented programming principles.

● Supports distribution transparency in client-server systems.

● Well-suited for scalable enterprise applications.

Disadvantages

● Tight coupling between client and server objects.

● Network overhead due to RPC mechanisms.

● Object based is less structured
● Complex error handling in case of failures. ● component = object
connector = RPC or RMI
Examples

● CORBA – Middleware for distributed object communication.

● Java RMI – Remote method invocation for Java objects.

● DCOM, .NET Remoting – Microsoft’s distributed object frameworks.

 Data-Centered Architectures
Concept

• Communication takes place through a shared data repository (active or passive).

• Components (processes) interact only by reading/writing data to this repository.

• Repository acts as the central point of coordination.

• Widely used in distributed systems like file systems and web services.

Characteristics

• Passive repository: Data store holds information, processes fetch/update it.

• Active repository: Data store notifies processes of changes (triggers, publish/subscribe).

• Loose coupling: Processes don’t communicate directly, only via the repository.

• Centralized consistency: Repository ensures integrity and consistency of data.

 Data-Centered Architectures
Advantages

● Promotes loose coupling between components.

● Simplifies data sharing among multiple processes.

● Supports scalability by allowing many processes to access the repository.

● Centralized control improves data consistency and integrity.

Disadvantages

● Performance bottleneck: Repository can become overloaded.

● Single point of failure if repository is not replicated.

● Latency issues if repository is geographically distant from clients.

Data Centric Architecture; e.g., shared
● Complex synchronization when multiple processes update data. distributed file systems or Web-based
Examples
distributed systems
● Distributed File Systems (NFS, AFS). Combination of data-centered and event
based architectures
● Web-based Data Services (REST APIs, Cloud Databases).
Processes communicate asynchronously
● Enterprise Service Bus (ESB) in middleware.
Event-Based Architectures
Concept
• Components communicate through the propagation of events.

• A process publishes events, and other processes subscribe to receive them.

• Communication is asynchronous – sender and receiver need not interact directly.

• Provides loose coupling: components don’t need to know each other’s identity.

Characteristics

• Publish/Subscribe model: Middleware delivers events to subscribers.

• Spatial decoupling: Publisher and subscriber don’t need to be connected directly.

• Temporal decoupling: Subscriber can receive events later if system supports buffering.

• Scalable and flexible for dynamic, distributed environments.

Event-Based Architectures
Advantages

● Enables loose coupling among distributed components.

● Supports asynchronous communication (non-blocking).
● High scalability – many subscribers can listen to the same event.
● Flexible: components can join or leave dynamically.

Disadvantages

● Difficult to guarantee delivery in all scenarios.

● Ordering of events may be hard to maintain.
● Debugging and error handling are more complex.
● Middleware may add performance overhead.

Examples

● Publish/Subscribe Systems (e.g., Kafka, RabbitMQ, MQTT).

● Notification Services (email, push notifications).
● Distributed Monitoring Systems (system logs, alerts).
Architectural Styles

• Communication via event Event-based arch.

supports several
propagation, in dist. systems
communication styles:
seen often in Publish/ Subscribe;
• Publish-subscribe
e.g., register interest in market
• Broadcast
info; get email updates
• Point-to-point
• Decouples sender & receiver;
asynchronous communication

• Figure 2-2. (a) The event-based architectural style

System Architectures
• Describe the physical realization of software components on machines.

• Show where components are placed and how they communicate across a network.

• Concerned with deployment, performance, and fault tolerance.

• Examples: Client-Server, Peer-to-Peer, Multi-tiered systems.

Centralized Architectures – Client-Server
Model
Concept

● Widely accepted model for understanding distributed systems.

● Processes divided into two groups:

○ Server: Provides a specific service (e.g., file system, database).

○ Client: Requests service from the server.

● Follows request-reply interaction (client sends request → server processes → server replies).

Communication

● Implemented via protocols over a network.

● Connectionless protocol:

○ Efficient for reliable networks (e.g., LAN).

○ Client sends request → server processes → reply sent back.

○ Issue: If reply is lost, resending may cause duplicate operations.

C/S Architecture

Figure 2-3. General interaction between a client and a server.

Centralized Architectures – Reliability &
Protocols
Idempotent Operations

● Idempotent requests: Safe to repeat without side effects (e.g., balance inquiry).

● Non-idempotent requests: Repetition causes errors (e.g., money transfer).

● Handling failures depends on whether the operation is idempotent.

Reliable Connection-Oriented Protocols

● Used in wide-area networks where communication is less reliable.

● Example: TCP/IP-based protocols in Internet applications.

● Steps: Client sets up a connection → sends request → server replies → connection closed.

● Trade-off: More reliable but costly due to connection setup/teardown.

Centralized Architectures – Application
Layering
Key Idea

● Client-Server model can be organized into 3 logical layers.

● Helps manage complexity by separating UI, Processing, and Data.

● Often referred to as 3-Tier Architecture.

Layers

1. User-Interface Layer – Direct interaction with users.

2. Processing Layer – Core application logic and computations.

3. Data Layer – Persistent storage and management of data.

User-Interface Layer
Functions

● Handles all interaction with the user (input/output).

● Manages display elements (text, forms, menus, graphics).

● Provides usability features such as mouse-based controls, icons, drag & drop.

Examples

● Simple: Character-based terminals in mainframe systems.

● Modern: Graphical interfaces (X-Windows in UNIX, Windows/Mac GUIs).

● Office suites: Common UI for word processor, spreadsheet, etc.

 Processing Layer
Functions

● Contains the core functionality of the application.

● Translates user input into operations on the database or file system.

● Often the “middle tier” that coordinates between UI and Data layers.

Examples

● Search Engine: Transforms user keywords → database queries → ranks results.

● Decision Support Systems: Financial data analysis using AI/statistics.

● Office Suites: Applications with light processing but integrated functionality.

Data Layer
Functions

● Manages persistent data (stored even when apps are inactive).

● Responsible for consistency and integrity across applications.

● Provides data independence: app logic and data organization are separate.

Examples

● File System: Simplest form of data management.

● Relational Databases: Store tables, metadata, constraints.

● Object-Relational Databases: Useful for complex data (e.g., CAD systems).

● Triggers: Automated actions when certain data conditions are met (e.g., credit limit alerts in banking).
Peer-to-Peer Architectures – Concept &
Features
Concept

● Decentralized system: No fixed clients or servers.

● All nodes (peers) are equal participants, acting as both clients and servers.

● Communication paths are horizontal and less structured compared to centralized models.

Features

● Symmetry: Each peer can initiate requests and also provide services.

● Resource sharing: Bandwidth, storage, and processing power are pooled.

● Dynamic topology: Peers can join or leave the network at any time.

● Can be structured (organized overlays, e.g., Distributed Hash Tables) or unstructured (random
connections).
Peer-to-Peer Architectures – Pros, Cons &
Examples
Advantages

● No single point of failure → high fault tolerance.

● Scalable: Performance improves as more peers join.

● Encourages resource utilization and collaboration.

Disadvantages

● Security challenges: Harder to enforce trust and authentication.

● Performance variability: Depends on peers’ capability and availability.

● Search efficiency issues: Finding data can be slow in unstructured P2P.

Examples

● File sharing systems: BitTorrent, Gnutella.

● Collaboration tools: Skype (early superpeer model).

● Blockchain networks: Bitcoin, Ethereum.

Example 1: Internet Search Engine
● User-Interface Layer:

○ Simple interface (search box for entering keywords).

○ Displays results as a list of webpage titles/links.

● Processing Layer:

○ Core program that transforms keywords into database queries.

○ Ranks results using indexing and ranking algorithms.

○ Converts results into HTML pages for display.

● Data Layer:

○ A huge database of prefetched and indexed web pages.

○ Acts as the backend repository supporting queries.

 Application Layering

Figure 2-4. The simplified organization of an Internet search

engine into three different layers.
Example 2: Decision Support System (Stock
Brokerage)
User-Interface Layer:

● Front-end for brokers to input queries or analyze stock performance.

● Visualization tools for displaying trends, graphs, predictions.

Processing Layer:

● Core analysis programs applying advanced methods.

● Uses statistics, AI, and machine learning for financial predictions.

● May require high-performance computing for responsiveness.

Data Layer:

● Financial database with stock market and transaction data.

● Maintains consistency and accuracy of sensitive financial records.

● Often implemented with relational databases + triggers (e.g., alert when credit exceeds limit).
Example 3: Desktop Office Suite
User-Interface Layer:
● Integrated GUI across applications (word processor, spreadsheet, email, etc.).

● Supports compound documents (e.g., embedding a chart in a Word file).

Processing Layer:
● Each application performs basic processing (e.g., text formatting, calculations).

● Middle tier contains lightweight functionality, but integrated.

Data Layer:
● Files stored in user’s home directory (often on a remote file server in offices).

● Maintains persistence of documents, spreadsheets, and messages.

● Uses file systems or databases for shared storage.

Multitiered Architectures
Concept
Multitiered (or multilayered) architectures extend the logical separation of User Interface, Processing, and
Data into physical tiers distributed across different machines.

● The idea is to assign different layers to different machines to balance load, improve scalability, and
simplify management.

● At the simplest level, this leads to two-tier client-server systems. As applications grow, we often
move to three-tier or N-tier architectures.
Two-Tier Architectures
In a two-tiered system, there are only two physical types of machines:
● Client Machine → runs all or part of the User Interface.
● Server Machine → runs both Processing and Data management.

There are several possible organizations within this model:

(a) Thin Client (Server-driven Presentation)
● Only minimal UI functionality is on the client (keyboard/mouse interaction).
● The server controls how data is presented.

● Example: Terminal connected to a mainframe.

(b) Full UI on Client
● The client machine contains complete user-interface software.
● Client acts as a graphical front-end, while the server executes application logic and manages data.

● Example: A GUI banking client connecting to a central database server.

Two-Tier Architectures
(c) Partial Processing at Client

● Some parts of the application logic are placed on the client.

● Example:

○ A form-based application where the client validates input before sending to server.

○ A word processor where editing is done on the client but advanced features (grammar/spell check)
execute on the server.

(d) Client Applications + Server Data

● The application runs mostly on the client, but all data requests go to the server.

● Example: Banking software preparing transactions locally, then uploading them to the bank’s database server.

(e) Client with Local Data Cache

● Client has its own local storage in addition to server storage.

● Example: Web browsers caching recently visited web pages on local disk.
Thin Clients vs Fat Clients
● Fat Clients

○ Significant processing happens at client machine.

○ Examples: full-featured desktop apps with heavy local logic.

○ Issues:

■ Harder to manage software updates.

■ Higher risk of errors due to dependency on OS/resources.
■ Platform-specific limitations.

● Thin Clients

○ Minimal processing on client side (mainly UI/presentation).

○ Most computation and data management handled by servers.

○ Advantages:

■ Centralized management, easier updates.

■ More reliable with fewer compatibility issues.

○ Trade-offs:

■ Simpler interfaces, lower client-side performance.

Three-Tiered Architectures

Expands client-server model with a middle application layer.

Architecture:
● Client → User-interface functions
● Application Server → Processing/business logic
● Database Server → Data storage and management

Server as Client Concept:

● A server may request services from another server.
● Example: Transaction Processing Monitor coordinates multiple database servers.

Benefits:
● Better scalability.
● Clearer separation of concerns.
● Easier system management.
Three-Tiered Architecture in Web Systems
Widely used in modern web applications.
Layers:

● Web Server (Presentation Layer): Entry point for user requests (e.g., Apache, Nginx).
● Application Server (Logic Layer): Executes processing logic, validates requests, applies rules.
● Database Server (Data Layer): Stores structured/unstructured data.

Example – Online Bookstore:

● User searches for a book on website.

● Web server receives query → forwards to application server.
● Application server checks inventory logic → queries database.
● Database server responds with results → delivered back to client.
Decentralized Architectures
● Multitiered client-server architectures:

○ Based on splitting applications into UI, Processing, and Data levels.

○ This form of distribution is called Vertical Distribution.

● Decentralized architectures focus more on horizontal distribution:

○ Clients and servers may be split into equivalent parts.

○ Each part handles a portion of the workload.

○ Provides load balancing and scalability.

Vertical Distribution
● Definition: Logical components (UI, processing, data) placed on separate machines.

● Key feature: Functions are logically distinct and executed on specialized machines.

● Analogy:

○ Similar to vertical fragmentation in distributed databases (splitting tables column-wise).

● Advantages:

○ Clear separation of roles.

○ Easier management & specialization of servers.

○ Improved maintainability of applications.

Horizontal Distribution
Definition: Splitting equivalent components across multiple machines.
Each component operates on a share of the dataset.
Purpose:
● Balances workload.

● Improves scalability and fault tolerance.

Example:

● Multiple servers hosting different subsets of customer data.

● Queries distributed among them for parallel processing.

Peer-to-Peer (P2P) Systems
● Peer-to-peer systems are a modern form of horizontal distribution.

● Characteristics:

○ All processes are equal peers.

○ Each peer acts as both client and server (a servent).

○ Interactions are symmetric.

● Overlay Networks:

○ Logical network formed on top of physical connections.

○ Nodes = Processes, Links = Communication channels (e.g., TCP).

Communication in P2P Systems
● Peers cannot directly connect to all other peers.

● Messages are routed via available communication links.

● Overlay networks organize this communication.

● Two types of overlay networks:

○ Structured: Well-defined organization, deterministic routing.

○ Unstructured: Random links, search-based communication.

Structured vs Unstructured P2P
Structured P2P Networks:

● Use deterministic algorithms (e.g., Distributed Hash Tables – DHTs).

● Efficient lookups, predictable performance.

● Example: Chord, Pastry.

Unstructured P2P Networks:

● No fixed organization, peers randomly connect.

● Content discovery often uses flooding/search.

● Example: Gnutella, early Napster.

Advantages of Decentralized Architectures
• Scalability: System grows by simply adding more peers.

• Fault Tolerance: No single point of failure.

• Resource Sharing: Every peer contributes resources.

• Flexibility: Works across heterogeneous systems.

 Variant (c):
Variants of
distribution (Fig.
2-5):
Multitiered Architectures Example: Form
validation at client
(a) Only before sending to
terminal-depend server
ent UI at client
Example: Word
processor → editing
(b) Entire UI local,
software at client grammar/spell-check
server-side

(c) Part of
application logic Variant (d) & (e):
at client
Client runs
application logic but
data operations
handled at server

Example: Banking
apps → client
prepares transaction
→ server processes
Thin Fat Example: Web
Client Client browsing → local
cache stored on
client disk
Figure 2-5. Alternative client-server organizations (a)–(e).
Transmission Failures
• With connectionless transmissions, failure of any sort means no reply
• Possibilities:
• Request message was lost
• Reply message was lost
• Server failed either before, during or after performing the service
Idempotency
• Typical response to lost request in connectionless communication:
re-transmission
• Consider effect of re-sending a message such as “Increment X by
1000”
• If first message was acted on, now the operation has been performed twice
• Idempotent operations: can be performed multiple times without
harm
• e.g., “Return current value of X”; check on availability of a product
• Non-idempotent: “increment X”, order a product
 Layered (software) Architecture for Client-Server
Systems
• User-interface level: GUI’s (usually) for interacting with end users
• Processing level: data processing applications – the core
functionality
• Data level: interacts with data base or file system
• Data usually is persistent; exists even if no client is accessing it
• File or database system
System Architecture
• Mapping the software architecture to system hardware
• Correspondence between logical software modules and actual computers
• Multi-tiered architectures
• Layer and tier are roughly equivalent terms, but layer typically implies
software and tier is more likely to refer to hardware.
• Two-tier and three-tier are the most common
Multitiered Architectures
(3 Tier Architecture)

Figure 2-6. An example of a server acting as client.

Centralized vs. Decentralized
Centralized Systems:
● One central server controls processing and data.

● Clients only request services.

● Drawbacks: single point of failure, scalability limits.

Decentralized Systems:
● Multiple independent nodes share control.

● No single entity dominates.

● Advantages: improved reliability, load distribution, fault tolerance.

Analogy:
● Centralized = one teacher teaching 500 students (failure if teacher absent).

● Decentralized = multiple study groups; learning continues even if one fails.

Centralized v Decentralized Architectures
• Traditional client-server architectures exhibit vertical distribution. Each level serves a different purpose in
the system.
• Logically different components reside on different nodes
• Horizontal distribution (P2P): each node has roughly the same processing capabilities and stores/manages
part of the total system data.
• Better load balancing, more resistant to denial-of-service attacks, harder to manage than C/S
• Communication & control is not hierarchical; all about equal
Peer-to-Peer (P2P) Concept
❏ A special case of decentralization.
❏ Each peer is both client and server (servent).
❏ Overlay Network: Logical layer connecting peers on top of the physical Internet.
Features:
● Symmetric roles → all peers equal.
● No central coordinator.
● Robust to failures and dynamic membership.
Examples:
● File sharing: BitTorrent
● Communication: Early Skype
● Finance: Bitcoin, Ethereum
Peer-to-Peer
• Nodes act as both client and server; interaction is symmetric
• Each node acts as a server for part of the total system data
• Overlay networks connect nodes in the P2P system
• Nodes in the overlay use their own addressing system for storing and
retrieving data in the system
• Nodes can route requests to locations that may not be known by the
requester.
Types of Peer-to-Peer Systems
Structured P2P:

● Nodes are arranged using strict, deterministic algorithms.

● Data placement is rule-based (mainly through Distributed Hash Tables, DHTs).
● Lookup is efficient and predictable.

Unstructured P2P:

● Nodes connect randomly.

● Data placement is random.
● Lookup relies on flooding or random walks.
Structured P2P
● Built using deterministic overlay rules.
● Each node and data item gets an identifier from a large key space.
● Goal: Map each data item’s key to a responsible node efficiently.
● Lookup usually in O(log N) steps, where N = number of nodes.
● Popular DHT systems:

○ Chord: ring-based mapping of data to nodes.

○ CAN (Content Addressable Network): multidimensional space partitioning.

Pros: Efficient, scalable, good search guarantees.

Cons: Complex join/leave protocols, sensitive to churn.
Chord (Structured P2P)
● Identifier Ring: Nodes arranged in a logical circle using IDs.
● Data mapping: Key k → stored at node with ID ≥ k (called successor).
● Lookup:

○ Each node maintains routing shortcuts (finger tables).

○ Queries resolved in O(log N) hops.

● Joining: New node generates ID, finds successor, updates ring links, takes relevant keys.
● Leaving: Node transfers data to successor and updates neighbors.

Chord ensures efficient, scalable routing of requests even in large networks.

Characteristics of DHT
• Scalable – to thousands, even millions of network nodes
• Search time increases more slowly than size; usually Ο(log(N))
• Fault tolerant – able to re-organize itself when nodes fail
• Decentralized – no central coordinator
(example of decentralized algorithms)
 Chord Routing Algorithm
Structured P2P
• Nodes are logically arranged in a circle
• Nodes and data items have m-bit identifiers (keys) from
a 2m namespace.
• e.g., a node’s key is a hash of its IP address and a file’s key
might be the hash of its name or of its content or other
unique key.
• The hash function is consistent; which means that keys are
distributed evenly across the nodes, with high probability.
Inserting Items in the DHT
• A data item with key value k is mapped to the node with the smallest
identifier id such that id ≥ k (mod 2m)
• This node is the successor of k, or succ(k)
• Modular arithmetic is used
Structured Peer-to-Peer Architectures

Figure 2-7. The mapping of

data items onto nodes in
Chord for m = 4
Finding Items in the DHT
• Each node in the network knows the location of some
fraction of other nodes.
• If the desired key is stored at one of these nodes, ask for it
directly
• Otherwise, ask one of the nodes you know to look in its set
of known nodes.
• The request will propagate through the overlay network
until the desired key is located
• Lookup time is O(log(N))
Joining & Leaving the Network
• Join
• Generate the node’s random identifier, id, using the
distributed hash function
• Use the lookup function to locate succ(id)
• Contact succ(id) and its predecessor to insert self into
ring.
• Assume data items from succ(id)
• Leave (normally)
• Notify predecessor & successor;
• Shift data to succ(id)
• Leave (due to failure)
• Periodically, nodes can run “self-healing” algorithms
Overlay Networks
• Are logical or virtual networks, built on top of a physical network
• A link between two nodes in the overlay may consist of several
physical links.
• Messages in the overlay are sent to logical addresses, not physical (IP)
addresses
• Various approaches used to resolve logical addresses to physical.
Circles represent nodes in the
network. Blue nodes are also part
of the overlay network. Dotted
lines represent virtual links.
Actual routing is based on
TCP/IP protocols

Overlay Network Example

Overlay Networks
• Each node in a P2P system knows how to contact several other nodes.
• The overlay network may be structured (nodes and content are
connected according to some design that simplifies later lookups) or
unstructured (content is assigned to nodes without regard to the
network topology. )
Structured P2P Architectures
• A common approach is to use a distributed hash table
(DHT) to organize the nodes
• Traditional hash functions convert a key to a hash value,
which can be used as an index into a hash table.
• Keys are unique – each represents an object to store in the
table; e.g., at UAH, your A-number
• The hash function value is used to insert an object in the
hash table and to retrieve it.
Structured P2P Architectures
• In a DHT, data objects and nodes are each assigned a
key which hashes to a random number from a very
large identifier space (to ensure uniqueness)
• A mapping function assigns objects to nodes, based
on the hash function value.
• A lookup, also based on hash function value, returns
the network address of the node that stores the
requested object.
Summary
• Deterministic: If an item is in the system it will be found
• No need to know where an item is stored
• Lookup operations are relatively efficient
• DHT-based P2P systems scale well
• BitTorrent and Coral Content Distribution Network incorporate DHT
elements
http://en.wikipedia.org/wiki/Distributed_hash_table
Unstructured P2P (Overview)
● No deterministic organization.
● Each node maintains a random list of neighbors (partial view).
● Data placement is random.
● Search: Queries flood through network or follow random walks.
🟢 Pros:
● Very simple to build.
● Tolerant to frequent joins and leaves (churn).
🔴 Cons:
● Searches inefficient, may require many messages.

● Higher bandwidth usage.

Unstructured P2P
• Unstructured P2P organizes the overlay network as a random graph.
• Each node knows about a subset of nodes, its “neighbors”.
• Neighbors are chosen in different ways: physically close nodes, nodes that
joined at about the same time, etc. -
• Data items are randomly mapped to some node in the system &
lookup is random, unlike the structured lookup in Chord.
Locating a Data Object by Flooding
• Send a request to all known neighbors
• If not found, neighbors forward the request to their neighbors
• Works well in small to medium sized networks, doesn’t scale well
• “Time-to-live” counter can be used to control number of hops
• Example system: Gnutella & Freenet (Freenet uses a caching system
to improve performance)
Comparison
• Structured networks typically guarantee that if an object is in the
network it will be located in a bounded amount of time – usually
O(log(N))
• Unstructured networks offer no guarantees.
• For example, some will only forward search requests a specific number of
hops
• Random graph approach means there may be loops
• Graph may become disconnected
Membership Management in Unstructured
P2P
● Nodes use gossip protocols to maintain neighbor lists.

● Partial view exchange: Peers periodically swap parts of their neighbor list.

● Joining: New node contacts any known peer, gets neighbors, joins network.

● Leaving: Node silently disappears; others eventually detect and remove it.

● Issues:

○ Popular nodes may get overloaded (high indegree).

○ Networks can become disconnected if not managed properly.

Hybrid & Topology Management
● Many systems combine structured and unstructured ideas.

● Two-layer model:

○ Lower layer = unstructured random graph (ensures connectivity).

○ Upper layer = structured or semantic overlay (optimizes search).

● Example:

○ Semantic overlays group nodes storing similar data.

○ Grid/torus overlays maintain nearest neighbors.

This hybrid model balances flexibility with efficiency.

Hybrid Architectures
• Combine client-server and P2P architectures
• Edge-server systems; e.g. ISPs, which act as servers to their clients, but
cooperate with other edge servers to host shared content
• Collaborative distributed systems; e.g., BitTorrent, which supports parallel
downloading and uploading of chunks of a file. First, interact with C/S
system, then operate in decentralized manner.
Structured vs. Unstructured (Comparison)

Aspect Structured P2P (DHTs) Unstructured P2P (Random)

Data Placement Deterministic, Random, no rules

rule-based
Lookup O(log N) steps Flooding/random walks
Scalability Very high Limited (message overhead)
Robustness Sensitive to churn Highly tolerant to churn
Management Complex join/leave Simple join/leave
 Superpeers
• Maintain indexes to some or all nodes in the system
• Supports resource discovery
• Act as servers to regular peer nodes, peers to other
superpeers
• Improve scalability by controlling floods
• Can also monitor state of network
• Example: Napster

Figure 2-12.
Edge-Server Systems
• An important class of distributed systems that is organized according to a hybrid
architecture is formed by edge-server systems.
• These systems are deployed on the Internet where servers are placed "at the edge" of
the network. This edge is formed by the boundary between enterprise networks and
the actual Internet, for example, as provided by an Internet Service Provider (ISP).
• Likewise, where end users at home connect to the Internet through their ISP, the ISP
can be considered as residing at the edge of the Internet.
• End users, or clients in general, connect to the Internet by means of an edge server.
• The edge server's main purpose is to serve content, possibly after applying filtering
and transcoding functions. More interesting is the fact that a collection of edge servers
can be used to optimize content and application distribution.
• The basic model is that for a specific organization, one edge server acts as an origin
server from which all content originates.
Edge-Server Systems

Figure 2-13. Viewing the Internet as consisting of a collection of edge servers.

Collaborative Distributed Systems BitTorrent
• Clients contact a global directory (Web server) to
locate a .torrent file with the information needed to
locate a tracker; a server that can supply a list of
active nodes that have chunks of the desired file.
• Using information from the tracker, clients can
download the file in chunks from multiple sites in the
network. Clients must also provide file chunks to
other users.
BitTorrent
• BitTorrent is a peer-to-peer file downloading system.
• The basic idea is that when an end user is looking for a file, he
downloads chunks of the file from other users until the downloaded
chunks can be assembled together yielding the complete file.
• An important design goal was to ensure collaboration. .
 Collaborative Distributed Systems
Trackers know which nodes are active
(downloading chunks of a file)
Tells how to locate the
tracker for this file

• Figure 2-14. The principal working of BitTorrent [adapted with

permission from Pouwelse et al. (2004)].
BitTorrent - Justification
• Designed to force users of file-sharing systems to participate in
sharing.
• Simplifies the process of publishing large files, e.g. games
• When a user downloads your file, he becomes in turn a server who can
upload the file to other requesters.
• Share the load – doesn’t swamp your server
P2P v Client/Server
• P2P computing allows end users to communicate without a
dedicated server.
• Communication is still usually synchronous (blocking)
• There is less likelihood of performance bottlenecks since
communication is more distributed.
• Data distribution leads to workload distribution.
• Resource discovery is more difficult than in centralized
client-server computing & look-up/retrieval is slower
• P2P can be more fault tolerant, more resistant to denial of
service attacks because network content is distributed.
• Individual hosts may be unreliable, but overall, the system
should maintain a consistent level of service
Architecture versus Middleware
• Where does middleware fit into an architecture?
• Middleware: the software layer between user
applications and distributed platforms.
• Purpose: to provide distribution transparency
• Applications can access programs running on remote nodes
without understanding the remote environment
Architecture versus Middleware
• Middleware may also have an architecture
• e.g., CORBA has an object-oriented style.
• Use of a specific architectural style can make it easier to develop
applications, but it may also lead to a less flexible system.
• Possible solution: develop middleware that can be customized as
needed for different applications.
Processes in Distributed
Systems
Processes in Distributed Systems
Processes = core building block in distributed systems.
Origin: operating systems (program in execution).
OS concerns: scheduling & management.
In distributed systems, new issues emerge:
● Efficient client-server organization.
● Virtualization for isolation & portability.
● Process migration for flexibility and scalability.

Focus areas

1. Threads
2. Virtualization
3. Client & server organization
4. Code migration
Processes
● Definition: A process is a program in execution on a virtual processor provided by the operating system.

● Attributes: Maintained in a process table (CPU registers, memory maps, open files, privileges, accounting
info, etc.).

● Goal of OS: Ensure concurrency transparency → multiple processes share CPU/memory safely without
interfering.

● Cost of Processes:

○ Creation: Needs full address space, stack, memory initialization.

○ Context Switching: Expensive (save registers, program counter, flush MMU/TLB, switch memory
maps).

○ Swapping: Needed when more processes exist than available memory.

📌 Processes = secure, isolated execution, but heavy in overhead.

Threads
● Definition: A lightweight unit of execution within a process.

● Relation: Multiple threads share the same address space of the process but run
independently.

● Thread Context: Minimal (CPU state + thread mgmt info).

● Implications:

○ High performance (cheaper than processes).

○ No automatic protection between threads → programmer must ensure safe access.

Processes vs Threads
Process:
● Own address space, OS-managed.
● Costly creation, deletion, switching.
● Strong protection → concurrency transparency.

Thread:

● Shares process resources.

● Fast creation/switching.
● Minimal protection (risk of race conditions).

Implication:

● Threads = faster, but require careful programming.

Threads in Nondistributed Systems
Use cases:
● GUI apps (e.g., spreadsheet: 1 thread for UI, 1 for computation, 1 for backup).

● Multiprocessor machines: parallelism.

● Large apps → cooperative parts (word processors, editors).

Advantage over IPC:

● Avoid costly context switches in interprocess communication.

● Shared memory within process is faster.

Engineering benefit: Clearer structure for multitasking apps.

Implementation of Threads
User-level Threads

● Implemented in user space (thread library).

● Advantages: cheap creation/destruction, fast switching.

● Drawback: Blocking system calls block the whole process.

Kernel-level Threads

● Managed by OS kernel.
● Advantage: blocking I/O only blocks one thread.

● Drawback: costly (every operation = system call).

 Implementation of Threads
Lightweight Processes (LWP) → Hybrid

● Each LWP runs in kernel context, mapped to multiple

user-level threads.

● Advantages:
○ Blocking calls do not block all threads.
○ Good for multiprocessor systems.
○ User space handles most synchronization
cheaply.

● Related: Scheduler Activations → kernel upcalls to thread library for better scheduling.
Threads in Distributed Systems
Threads are essential in distributed environments due to blocking I/O, long network delays, and parallel
connections.

(a) Multithreaded Clients

● Example: Web Browser

○ Fetch HTML file with one thread, images with parallel threads.

○ Threads allow parallel TCP connections and hiding of communication latency.

○ Can connect to replicated servers → load balancing.

Threads in Distributed Systems
(b) Multithreaded Servers
● Most common use in distributed systems.

● Dispatcher/Worker model (Fig. 3-3):

○ Dispatcher thread listens for requests.

○ Hands off to worker threads.

○ Workers block on disk/network without

blocking the whole server.
Threads in Distributed Systems
Design Comparisons

● Single-threaded Server: Simple, but blocks during I/O → poor performance.

● Finite-State Machine: High performance (nonblocking I/O), but hard to program.

● Multithreaded Server: Best of both → simple sequential logic + parallelism.

Advantages of Threads in Distributed Systems
Improved Performance
● Cheaper creation/destruction than processes.

● Context switching faster (no MMU map / TLB flushing).

● Allows overlapping computation and I/O.

Concurrency in Applications
● Example: Spreadsheet → one thread updates cells, another handles input, another saves to disk.

● Example: Word Processor → separate threads for input, spell-check, layout, index generation.
Parallelism
● On multiprocessors: assign threads to CPUs → exploit hardware parallelism.
Simpler Application Design
● Programs can be structured into logically independent tasks.
Virtualisation
Concept of Virtualization
● Threads & Processes: Create the illusion of many tasks running at once.

● On a single CPU, only one instruction executes at a time → illusion achieved by

context switching.

● Same principle applied broadly → resource virtualization.

Definition:
Virtualization = replacing/extending an existing interface to mimic another system.
Programs then run on virtual resources instead of real ones.
Interfaces in Systems
Programs interact with systems via
interfaces:
● CPU instruction set → OS system
calls → Middleware APIs.

Virtualization:

● Extends/replaces an interface.
● Mimics another system’s behavior.

Fig. 3-5(b): System B virtualizes System

A → Programs run unchanged.
Four Interfaces in a Computer System
3. System Call Interface (OS):

● A way for user programs to request OS

services.

● Example: open(), read(), write() in

UNIX.

4. Library / API Interface:

● Application Programming Interface (API).

● Provides high-level function calls that

internally use system calls.

● Example: Java API, Windows API, POSIX

libraries.
Data structure Two Main Approaches to
Virtualization
Two Main Approaches to Virtualization
(A) Process Virtual Machine
● A runtime system provides a virtual environment for a single process.
● Applications run on an abstract instruction set, rather than directly on hardware.

● Implementation methods:
○ Interpretation – instructions are interpreted one by one.
■ Example: Java Virtual Machine (JVM).

○ Emulation – translates foreign instructions into host instructions.

■ Example: Running Windows applications on Linux using Wine.

● Scope:

○ Virtualization is only for one process at a time.

○ Does not virtualize the full hardware/OS.
Two Main Approaches to Virtualization
(B) System Virtual Machine (Full Virtualization)
● A Virtual Machine Monitor (VMM), also called a Hypervisor, is used.
● Provides a complete virtual hardware interface.
● Multiple Operating Systems (OSes) can run independently and simultaneously.

● Examples:

○ VMware Workstation, ESXi (Type-1 and Type-2 hypervisors).

○ Xen, KVM, Microsoft Hyper-V.

● Scope:

○ Virtualization of the entire system (CPU, memory, I/O, devices).

○ Allows concurrent OS + application stacks on a single machine.
 Data structure Two Main Approaches to
Virtualization
Feature Process Virtual Machine (PVM) System Virtual Machine (SVM)

Scope Single application/process Full system (OS + apps)

Example JVM, .NET CLR, Wine VMware, Xen, VirtualBox

Virtualization Instruction set + system calls Complete hardware

target
Isolation level Process-level OS-level

Use cases Portability, Cloud computing, server

platform-independence consolidation
Importance of Virtual Machine Monitors
(VMMs)
According to Rosenblum & Garfinkel (2005):

● Reliability & Security:

○ Isolates failures—if one VM crashes, others continue.

○ Protects against malware spreading across systems.

● Portability:

○ A complete environment (OS + apps) can be migrated across hardware.

○ Example: Live migration in cloud data centers.

Not for the exam
Examples of Virtualization Technologies
1. System Virtual Machines (Full Virtualization using Hypervisors)
● Provide complete hardware emulation → each VM has its own OS.
● Multiple operating systems run on the same physical machine.
Examples:
● VMware ESXi / Workstation → industry-standard hypervisors.
● Xen → open-source hypervisor (used by Amazon EC2).
● KVM (Kernel-based Virtual Machine) → built into Linux.
● Microsoft Hyper-V → native hypervisor for Windows.
● Oracle VirtualBox → desktop VM manager.

Used in cloud computing, server consolidation, data centers.

Process Virtual Machines
● Provide a virtual runtime environment for a single application.

Examples:

● Java Virtual Machine (JVM) → runs Java bytecode across platforms.

● .NET CLR (Common Language Runtime) → runs .NET applications.

● Wine → runs Windows apps on Linux by API emulation.

👉 Used in cross-platform application development.

OS-Level Virtualization (Containers)
Here is where Docker comes in

● Unlike hypervisors, containers share the same host OS kernel.

● Each container is isolated (with its own filesystem, libraries, processes), but not a full OS.
● Lightweight compared to VMs → faster startup, less resource usage.

Examples:

● Docker → most popular container runtime.

● Podman → alternative to Docker (rootless containers).
● Kubernetes (K8s) → orchestrator for managing 100s or 1000s of containers.
● LXC (Linux Containers) → early container tech, foundation of Docker.
● OpenVZ → container-based virtualization for Linux.

Used in microservices, DevOps, cloud-native apps.

Docker vs Virtual Machines
Feature Virtual Machines (VMware, Xen, KVM) Docker (Containers)

OS per Each VM runs full OS Shares host OS kernel

instance
Startup time Minutes Seconds

Resource Heavy (RAM, CPU for full OS) Lightweight

usage
Isolation Strong (separate OS) Process-level isolation

Use cases Multi-OS, legacy support, strong Microservices, CI/CD, cloud-native

isolation apps
Clients
Networked User Interfaces
● Purpose: Client machines allow users to interact with remote servers.

● Two Approaches:

○ Service-specific clients: Each service has a counterpart on the client (e.g., agenda on PDA syncing with
remote server) using application-level protocols.

○ Thin clients: Clients serve mainly as terminals; processing and storage are server-side (application-neutral).

● Example – X Window System:

○ Controls bit-mapped terminals (monitor, keyboard, mouse).

○ X Kernel: Handles hardware-dependent terminal operations.

○ Xlib: Application library for interacting with the X kernel over the X protocol.

○ Client-Server Relation: X kernel acts as server; applications act as clients.

Clients
Thin-client Network Computing Issues:

● Coupling of application logic and interface commands can hurt performance over high-latency networks.

● Solutions like NX and THINC:

○ Reduce bandwidth via caching, differential encoding, command batching.

○ THINC uses high-level device commands to simplify display servers while leveraging application
semantics.

Compound Documents

● Modern UIs support sharing a single graphical window among multiple applications.

● Operations like drag-and-drop and in-place editing integrate different document types (text, images,
spreadsheets).

● Key Idea: Seamless integration of multiple document types at the UI level.

Client-Side Software for Distribution
Transparency
Client software includes: UI, application logic, and middleware for transparency.

Distribution transparency examples:

● Access transparency: Client stubs hide server location and architecture differences.

● Replication transparency: Middleware collects responses from multiple replicas and forwards a
single response.

● Failure transparency: Middleware retries requests, switches servers, or serves cached data.

Concurrency and persistence transparency often rely more on server-side mechanisms.

Servers
General Design Issues
● Server: Process providing a specific service to clients; waits for requests, processes them, and responds.

● Server types:

○ Iterative: Handles requests sequentially.

○ Concurrent: Delegates requests to threads or processes for parallel handling.

● Client-server binding:

○ Servers listen on endpoints (ports); clients contact endpoints via global or dynamic assignments.

○ Superserver approach (e.g., inetd in UNIX): Single process listens for multiple service requests and
forks server processes on demand.
Servers

● Interrupt Handling: Use out-of-band data or separate control endpoints for urgent client commands.

● State Management:

○ Stateless servers: Do not retain client state; example – web servers.

○ Soft state: Temporary state maintained; expires if not refreshed.

○ Stateful servers: Maintain persistent state; e.g., file servers tracking client access.

○ Cookies: Allow stateless servers to maintain session information via the client.
Server Clusters
Definition: Networked collection of machines running servers; often LAN-based for low latency and high
bandwidth.

Typical 3-tier architecture:

1. Switch: Routes client requests.

2. Application servers: Handle application logic.
3. Data servers: File and database management.

Two-tier variants: Combine application and data servers on the same machine (common for streaming
media).
Load balancing: Round-robin or service-aware request forwarding; TCP handoff may require OS-level
modifications.

Distributed servers: Dynamically configurable clusters, appearing as a single machine to clients.

● Example: Use MIPv6 with a home agent and care-of addresses to provide stable contact addresses
and route optimization.
Managing Server Clusters
● Small clusters: Admin logs into nodes or uses an administration machine for updates.

● Large clusters: Centralized administration is impractical; ad hoc rules are often used.

● Cluster reliability: Probability of fault-free operation decreases with size; large-scale clusters require
fault-tolerant design.
Code Migration
In distributed systems, communication usually means passing data. However, sometimes it is more
efficient to move the code (programs or parts of programs), even while it is executing, instead of
sending large amounts of data.

This idea is called Code Migration.

Approaches to Code Migration
Reasons for Code Migration
1. Performance

○ Originally, code migration was used as process migration: moving an entire process
from one machine to another.

○ Main goal: load balancing (move processes from overloaded to lightly loaded
machines).

○ Example: If a client needs to process a large dataset stored on a server, instead of

transferring all data over the network, send the code to the server. This reduces
communication overhead.
Approaches to Code Migration
2. Flexibility

● Instead of pre-deciding where code runs, systems

can dynamically configure themselves.

● Example: A server can send the necessary

client-side library implementation only when the
client connects → reduces dependency on
pre-installed software.

Figure 3-17 shows this:

● Client first fetches the required software

(protocol/driver code) from the server.
● Then invokes the server.
● This enables dynamic, on-demand configuration.
Approaches to Code Migration
3. Parallelism
● Code migration can allow parallel execution without explicit parallel programming.

● Example: A mobile agent (a small mobile program) is sent to multiple sites to search
data simultaneously → gives speed-up.
Models for Code Migration
A process can be seen as three segments:

● Code Segment → program instructions

● Resource Segment → files, devices, ports, processes
● Execution Segment → program counter, stack, private data

Two key types of mobility:

1. Weak Mobility
○ Only code segment moves (with initialization data).
○ Execution always starts from the beginning (predefined point).
○ Example: Java applets.
○ Simple but limited.

2. Strong Mobility
○ Both code and execution segment move.
○ Running process can pause, migrate, and resume execution where it left off.
○ Powerful but hard to implement.
Further classification:
● Sender-Initiated Migration: code migration
started from the current machine (e.g.,
uploading a program to a compute server).

● Receiver-Initiated Migration: target

machine requests the code (e.g., Java
applets downloaded by a browser).

Figure 3-18 summarizes alternatives:

● Weak vs. Strong mobility

● Sender vs. Receiver initiated

● Execution in same process vs. new process

Migration and Local Resources
When code migrates, resource references in the process may break.

Process-to-Resource Bindings
1. By Identifier → needs that exact resource (e.g., specific TCP port, specific URL).

2. By Value → only needs the content (e.g., standard libraries).

3. By Type → needs some resource of a type (e.g., any printer).

Resource-to-Machine Bindings
1. Unattached Resources → easily moved (files).

2. Fastened Resources → movable, but expensive (large database, websites).

3. Fixed Resources → cannot move (local devices, local sockets).

Migration and Local Resources
Figure 3-19 (3×3 table) → shows possible strategies for each combination:

● Copy/Move resource (if unattached).

● Establish global reference (if fastened or fixed).
● Rebind to equivalent resource (if by type).
Migration in Heterogeneous Systems
Problem:
● Distributed systems are heterogeneous (different OS, architectures).
● Migrated code must still run correctly.
● Execution state must also be portable.

Solutions:
1. Intermediate Code & Virtual Machines
○ Use machine-independent code (e.g., Java bytecode).
○ Executed on a Virtual Machine (VM).
○ Similar to old Pascal portability solutions.
Migration in Heterogeneous Systems
Solution
1. Migrating Entire Environments

○ Instead of moving individual processes, move the whole environment/VM.

○ Example: Migrating containers/VMs across machines.
○ Preserves execution + resource bindings.
○ Useful in server clusters (e.g., VM migration in data centers).

○ Clark et al. (2005) Example:

■ Migrating a VM across servers with minimal downtime.

■ Pre-copy approach: copy memory pages while VM runs, then briefly pause and
transfer modified pages.
■ Downtime reduced to ~200 ms.