A Database Management System (DBMS) is a software system that is
designed to manage and organize data in a structured manner. It allows
users to create, modify, and query a database, as well as manage the
security and access controls for that database.
What is DSMS Architecture ?
DSMS stands for data stream management system. It is nothing but a
software application just like DBMS (database management system) but it
involves processing and management of continuously flowing data stream
rather than static data like excel or pdf or other files. It generally use for deal
data stream from with various sources which included sensors data, social
media field, financial report etc.
Just like DBMS, DSMS also provides the wide range of operations likes
storage, processing, analyzing, integration also helps to generate the
visualization and report only use for data streams.
There are wide range of DSMS application available in the market among
them Apache flint, Apache Kafka, Apache storm, Amazon kinesis etc. DSMS
process 2 types of queries standard queries and ad hoc queries.
DSMS consists of various layer which are dedicated to perform particular
operation which are as follows:
1. Data source Layer
The first layer of DSMS is data source layer as it name suggest it is
comprises of all the data sources which includes sensors, social media
feeds, financial market, stock markets etc. In the layer capturing and parsing
of data stream happens. Basically it is the collection layer which collects the
data.
2. Data Ingestion Layer
You can consider this layer as bridge between data source layer and
processing layer. The main purpose of this layer is to handle the flow of data
i.e., data flow control, data buffering and data routing.
3. Processing Layer
This layer consider as heart of DSMS architecture it is functional layer of
DSMS applications. It process the data streams in real time. To perform
processing it is uses processing engines like Apache flink or Apache storm
etc., The main function of this layer is to filter, transform, aggregate and
enriching the data stream. This can be done by derive insights and detect
patterns.
4. Storage Layer
Once data is process we need to store the processed data in any storage
unit. Storage layer consist of various storage like NoSQL database,
distributed database etc., It helps to ensure data durability and availability of
data in case of system failure.
5. Querying Layer
As mentioned above it support 2 types of query ad hoc query and standard
query. This layer provides the tools which can be used for querying and
analyzing the stored data stream. It also have SQL like query languages or
programming API. This queries can be question like how many entries are
done? which type of data is inserted? etc.,
6. Visualization and Reporting Layer
This layer provides tools for perform visualization like charts, pie chart,
histogram etc., On the basis of this visual representation it also helps to
generate the report for analysis.
7. Integration Layer
This layer responsible for integrating DSMS application with traditional
system, business intelligence tools, data warehouses, ML application, NLP
applications. It helps to improve already present running applications.
The layers are responsible for working of DSMS applications. It provides
scalable and fault tolerance application which can handle huge volume of
streaming data. These layer can change according to the business
requirements some may include all layer some may exclude layers.
