System
Architecture
Memi Lavi
[Link]
System Architecture
• The Big Picture
• Answers the questions:
• How will the system work under heavy load?
• What will happen if the system will crash at this exact moment in the
business flow?
• How complicated can be the update process?
• And more…
System Architecture
• Includes:
• Defining the Software Components (Services)
• Defining the way these components communicate
• Designing the system’s capabilities (scalability, redundancy,
performance, etc.)
System Architecture
• Includes:
• Defining the Software Components (Services)
• Defining the way these components communicate
• Designing the system’s capabilities (scalability, redundancy,
performance, etc.)
Our Topics
• Loose Coupling
• Stateless
• Caching
• Messaging
• Logging & Monitoring
Loose Coupling
Making sure the services are not
strongly tied to other services
Loose Coupling
• Without:
Service A
Service B Service C
Loose Coupling
• Without:
Service A
Service B Service C
Loose Coupling
• Without:
Service A
Service B Service C
Loose Coupling
• With:
Service A
Service B Service C
Loose Coupling
• With:
Service A
Flexible, Easy to
Maintain
Service B Service C
Loose Coupling in Services
• Prevents platform coupling
• Prevents URL coupling
Loose Coupling in Services
Stock Quotes Service
(Java)
Java RMI
Portfolio
PortfolioService
Service
(Java)
Loose Coupling in Services
Stock Quotes Service Stock Quotes Service
(Java) (Java)
REST API REST API
[Link] [Link]
Recommendation
Service
(Python)
Loose Coupling
Service A
Service B Service F
The Spiderweb
Service C Service E
Service D
Loose Coupling
Service A
Service B Service F
The Spiderweb
Service C Service E
Service D
Loose Coupling
Service A
Service B Service F
The Spiderweb
Service C Service E
Service D
Loose Coupling - Directory
Query for URL Yellow
Service A
Pages
Service B Service F
Services only
Use
need to know
URL
the Directory’s
Service C Service E URL
Service D
Loose Coupling - Gateway
Service A
Service B Service F
Services only
Gateway need to know
the Gateway’s
Service C Service E URL
Service D
Stateless
The application’s state is stored
in only two places – the data
store and the user interface
State = Application’s
Stateless Example
User
Interface
Send user / password
Log In Service
Store user for future use
Data stored in code =
Stateful
Retrieve user details Check user / password
exists
Database
Scalability - A Reminder
• Grow and shrink as needed
• Scale Up vs Scale Out
• Scale Out is usually preferred
Redundancy - A Reminder
• Allows the system to function properly when resource
is not working
• Example:
• A system with more than one server
• When a server goes down, the other continue
working
Scalable & Redundant Architecture
User
Interface
Distributes Load Scalability
Load Balancer
Routes to live services only Redundancy
Is Alive?
Log In Service Log In Service Log In Service
Database
Stateful Example
User
Interface
Send user / password
Load Balancer
User / password sent to
Service #1
Log In Service Log In Service Log In Service
Store user for future use
1 2 3
Retrieve user details
Check user / password
exists
Database
Stateful Example
User
Interface
Add item to shopping cart
Please
Load Balancer
Log In
Request sent to
Service #2
Cart Service Cart Service Cart Service
Stored user details
1 2 3
Database
Stateless Example
User
Interface
Add item to shopping cart
Load Balancer
Action sent to
Service #2
Cart Service Cart Service Cart Service
1 2 Check for user details 3
Database
Stored user details
Stateless
• Always use stateless architecture
• Supports Scalability and Redundancy
Caching
Bring data closer to its consumer
so that its retrieval will be faster
Browser Cache
Browser Cache
Server
Service Cache
Service
DAL • Serialize to Objects
Cache
• Compile the SQL
Database • Retrieve the data
Cache Tradeoff
Reliability Performance
Single Source Database High Good
of Truth Data is saved to disk Data is retrieved from disk,
then serialized
Cache Poor Excellent
Data is stored in memory Data is retrieved from memory
What to Cache?
Cache should hold data that is
frequently accessed
and
rarely modified
What to Cache?
Cache should hold data that is
frequently accessed Retrieval should be fast and easy
for:
and - Optimal user experience
- Minimum load
rarely modified Using cache:
- Retrieval is fast (in-memory)
- UX is optimal
What to Cache?
Cache should hold data that is
frequently accessed
and
rarely modified - Syncing cache and DB is a
challenge
- When not in sync, leads to data
corruption and bad user
experience
Example
Stock Quotes
Service
Cache
Database
Example
Load Balancer
Request
Update Stock API Update Stock API Update Stock API
Stock Quotes Stock Quotes Stock Quotes
Service Service Service
Cache Cache Cache
Stale
Database
Cache Types
Stock Quotes Stock Quotes Stock Quotes
Service Service Service
In-Memory, In-
Process Cache
Distributed Cache
• Existing libraries • External product
• Can be easily implemented using • Data is stored in separate process
static concurrent collection • Provides interface for accessing the data
• Great performance • Size virtually unlimited
• Size is limited to the process’s • Auto nodes syncing
memory • Not the best performance
• Stores only primitive types
Example
Load Balancer
Request
Update Stock API Update Stock API Update Stock API
Stock Quotes Stock Quotes Stock Quotes
Service Service Service
In Memory, Process Cache In Memory, Process Cache In Memory, Process Cache
• Develop Sync Mechanism
• Hammer the Database
Database
Example
Load Balancer
Request
Update Stock API Update Stock API Update Stock API
Stock Quotes Stock Quotes Stock Quotes
Service Service Service
Distributed Cache
• Data immediately synced
with all nodes
• All servers have the same
Database
data
Choosing Cache Type
Distributed Cache
Distribution among servers
Failover capabilities
Large Cache storage
Choosing Cache Type
Distributed Cache In-Memory, In-Process Cache
Distribution among servers Best performance possible
Failover capabilities Store complex objects
Large Cache storage
Choosing Cache Type
Distributed Cache In-Memory, In-Process Cache
Distribution among servers Best performance possible
Failover capabilities Store complex objects
Large Cache storage Very easy to use
Requires training and setup
Messaging
Means of communication
between the various services
Messaging
• Not just REST API
• Not Exclusive
Messaging Criteria
• Performance
• Message Size
• Execution Model
• Feedback & Reliability
• Complexity
REST API
• De-Facto Standard for HTTP-based systems
GET [Link]
POST [Link]
{order data…}
HTTP API
Service
REST API
Performance Very Fast
REST API
Performance Very Fast
Message Size Same as HTTP protocol limitations
(Usually Get -> 8KB, POST & PUT -> dozens MB)
REST API
Performance Very Fast
Message Size Same as HTTP protocol limitations
(Usually Get -> 8KB, POST & PUT -> dozens MB)
Execution Model Request / Response
Great for quick, short actions, not suitable for long
processes
REST API
Performance Very Fast
Message Size Same as HTTP protocol limitations
(Usually Get -> 8KB, POST & PUT -> dozens MB)
Execution Model Request / Response
Great for quick, short actions, not suitable for long
processes
Feedback & Immediate feedback via Response
Reliability Codes
REST API
Performance Very Fast
Message Size Same as HTTP protocol limitations
(Usually Get -> 8KB, POST & PUT -> dozens MB)
Execution Model Request / Response
Great for quick, short actions, not suitable for long
processes
Feedback & Immediate feedback via Response
Reliability Codes
Complexity Extremely easy to implement
REST API
Performance Very Fast
Message Size Same as HTTP protocol limitations
(Usually Get -> 8KB, POST & PUT -> dozens MB)
Execution Model Request / Response
Great for quick, short actions, not suitable for long
processes
Feedback & Immediate feedback via Response
Reliability Codes
Complexity Extremely easy to implement
Useful For Traditional Web Apps
HTTP Push Notifications
Client Real-Time Communication:
Subscribe Notify
Service
HTTP Push Notifications
• Uses advanced web techniques (ie. Web Sockets)
• Very popular in chats
HTTP Push Notification
Performance Excellent
HTTP Push Notification
Performance Excellent
Message Size Limited
Usually no more than a few KB
HTTP Push Notification
Performance Excellent
Message Size Limited
Usually no more than a few KB
Execution Model Web Socket connection / Long Polling
HTTP Push Notification
Performance Excellent
Message Size Limited
Usually no more than a few KB
Execution Model Web Socket connection / Long Polling
Feedback & None (Fire & Forget)
Reliability (Can be implemented, quite complex)
HTTP Push Notification
Performance Excellent
Message Size Limited
Usually no more than a few KB
Execution Model Web Socket connection / Long Polling
Feedback & None (Fire & Forget)
Reliability (Can be implemented, quite complex)
Complexity Extremely easy to implement
HTTP Push Notification
Performance Excellent
Message Size Limited
Usually no more than a few KB
Execution Model Web Socket connection / Long Polling
Feedback & None (Fire & Forget)
Reliability (Can be implemented, quite complex)
Complexity Extremely easy to implement
Useful For Chat, Monitoring
Queue
Place Pull message
message in… from…
Service #1 Queue Service #2
• Messages will be handled once and only once
• Messages will be handled in order
Queue
Performance Not so good
(Push / Poll, DB Persistence)
Queue
Performance Not so good
(Push / Poll, DB Persistence)
Message Size Technically almost not limited
But use small messages
Queue
Performance Not so good
(Push / Poll, DB Persistence)
Message Size Technically almost not limited
But use small messages
Execution Model Polling
Queue
Performance Not so good
(Push / Poll, DB Persistence)
Message Size Technically almost not limited
But use small messages
Execution Model Polling
Feedback & Very reliable
Reliability
Queue
Performance Not so good
(Push / Poll, DB Persistence)
Message Size Technically almost not limited
But use small messages
Execution Model Polling
Feedback & Very reliable
Reliability
Complexity Requires training and setup
Queue
Performance Not so good
(Push / Poll, DB Persistence)
Message Size Technically almost unlimited
But use small messages
Execution Model Polling
Feedback & Very reliable
Reliability
Complexity Requires training and setup
Useful For Complex system with lots of data, when
order and reliability are top priority
File-based & Database-based
Place Pull message
message in… File Folder from…
Service #1 Service #2
Database
File-based & Database-based
Performance Not so good
(Push / Poll, DB Persistence)
Message Size Unlimited
Execution Model Polling
Feedback & Very reliable
Reliability
Complexity Requires training and setup
Useful For Complex system with lots of data.
Better use queues
File-based & Database-based
Service #2
Instance #1
Problems:
Service #2
1. File locked File Folder Service #2
Instance #2
2. Duplicate processing
Service #2
Instance #3
Messaging Summary
REST API HTTP Push Queue File- & DB-based
Performance Very fast Excellent Not so good Not so good
Message Size Same as Limited Technically Unlimited
HTTP unlimited
limitations
Execution Request / Web Socket / Long Polling Polling
Model Response Polling
Feedback & Immediate None Very reliable Very reliable
Reliability feedback
Complexity Extremely Extremely easy Requires training Requires training
easy and setup and setup
Useful For Traditional Chat, Monitoring Complex system Complex system
web apps with lots of data, with lots of data.
where order and Better use queues
reliability are top
priority
Logging & Monitoring
Central Logging Service
Service #1 Service #2 Service #3
Log Engine #1 Log Engine #2 Log Engine #3
Database NoSQL
Central Logging Service
Service #1 Service #2 Service #3
Log Engine #1 Log Engine #2 Log Engine #3
Logging Service
Database
Central Logging Service
• Implementation:
• API
• Watch folders
Correlation ID
Service #1 Service #2 Service #3
Correlation ID
Recommendation
Log
X
Stock Quotes
Log
Recommendation Stock Quotes
Log Log
CorID 17 CorID 17
CorID 18 CorID 18
System Architecture- Summary
• Use these concepts to design a fast ,secure, reliable
and easy to maintain system
• Make the choice as early as possible
• They are not exclusive…
• But they are the most important ones
