Thandar Aung

Followers

Following

Public Views

John Barry

Queen's University Belfast

Laurel Heydir

Universitas Sriwijaya

David Seamon

Kansas State University

Mustafa Kemal Bayırbağ

Middle East Technical University

Timothy Morton

Rice University

Nikos Katsikis

Delft University of Technology

Thomas Marois

McMaster University

Siti Nurchasanah

Jenderal Soedirman University

Darwin H . Pangaribuan

The University Of Lampung

Carlos Manuel Faísca

University of Coimbra

Interests

Uploads

Papers by Thandar Aung

Performance Evaluation for Real-Time Messaging System in Big Data Pipeline Architecture

2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), 2018

Nowadays, Real time messaging system is the essential thing in enabling time-critical decision ma... more Nowadays, Real time messaging system is the essential thing in enabling time-critical decision making in many applications where it is important to deal with real-time requirements and reliability requirements simultaneously. For dependability reasons, we intend to maximize the reliability requirement of real time messaging system. To develop real time messaging system, we create real time big data pipeline by using Apache Kafka and Apache Storm. This paper focuses on analyzing the performance of producer and consumer in Apache Kafka processing. The performance of Kafka processing modify to be more reliable on the pipeline architecture. Then, the experiment will be conducted the processing time in the performance of the producer and consumer on various partitions. The performance evaluation of Kafka can impact on messaging system in real time big data pipeline architecture.

Evaluation of Apache Kafka in Real-Time Big Data Pipeline Architecture

Today, many applications based on real-time analytics need to enable time-critical decision with ... more Today, many applications based on real-time analytics need to enable time-critical decision with real-time requirements and process with reliability requirements. Modern distributed systems are growing exponentially as far as performance and scale. The main purpose of Big Data real-time processing is to realize an entire system that can process such mesh data in a short time. And the performance of processing time can be guaranteed in a satisfactory range. To develop a distributed data pipeline, the system proposes real-time big data pipeline by using Apache Kafka and Apache Storm. Apache Kafka is currently the most popular framework used to ingest the data streams into the processing platforms. However, there are many challenges how to send reliable messages on many servers. This paper focuses on the comparison of the processing time between successful processes and failed processes on many servers. The experimental results show the performance impact on both producer and consumer ...

Analytics of Reliability for Real-Time Big Data Pipeline Architecture

Nowadays, many applications need high reliability pipeline architecture to get faster process and... more Nowadays, many applications need high reliability pipeline architecture to get faster process and reliable data within short time. Kafka has emerged as one of the important components of real-time processing pipelines in combination with Storm. This paper focuses to develop the real-time big data analytics pipeline architecture for reliability. Real-time data pipelines can be implemented in many ways and it will look different for every business. To develop the pipeline architecture, we create real time big data pipeline by using Apache Kafka and Apache Storm. Kafka and Storm naturally complement each other and their powerful cooperation enables real-time streaming analytics for fast-moving big data. Then, the experiment will be conducted how the processing time decreases with the same messages on the different partitions. KeywordsMessaging, Real-time processing, Apache

CIMLA: Checkpoint Interval Message Logging Algorithm in Kafka Pipeline Architecture

2020 International Conference on Advanced Information Technologies (ICAIT), 2020

In big data era, the reliability of real-time messages is the most important point for many busin... more In big data era, the reliability of real-time messages is the most important point for many business needs. Real-time messaging system establishes to handle real-time requirements and reliability requirements in business applications. Apache Kafka is a highly scalable publish-subscribe messaging system. The effect of reliability in Apache Kafka impact on the different use cases. Fault tolerance becomes the main factor to improve the reliability of Apache Kafka. Apache Kafka needs to handle the challenges of the message loss rate in server failure. To improve the message recovery, Kafka requires to manage the message delivery process. This paper introduces CIMLA (Checkpoint Interval Message Logging Algorithm) to enhance the message delivery in Apache Kafka. The system evaluates the message loss rate, throughput and latency in the server failure case. The system achieves the message recovery process by using CIMLA which effectively controls the lost messages with identification. The e...

Coordinate Checkpoint Mechanism on Real-Time Messaging System in Kafka Pipeline Architecture

2019 International Conference on Advanced Information Technologies (ICAIT), 2019

The real-time messaging system is the critical thing in computing based on time-critical decision... more The real-time messaging system is the critical thing in computing based on time-critical decision making in many organizations. In the real-time messaging system, fault tolerance is the key challenge of developing reliability requirements. Apache Kafka is a popular framework for consuming data stream into the processing platforms. However, there are many challenges in the replication process because of a server failure in Apache Kafka. To develop fault tolerance in Apache Kafka, this paper focuses on defining a fixed checkpoint interval to reduce the recovery time and lost messages in server failure. Then, we measure the cost of checkpointing, cost of rollback and total time cost of overheads due to the fixed checkpointing intervals. The system shows the drawback of real-time processing depend on the number of lost messages on various partitions and server failure processes. The experimental results emphasize the checkpoint interval method to reduce recovery time and lost messages. Conforming to the experimental results, the performance of the total time cost with checkpoint interval saves time approximately 30% than without checkpoint interval.

Comparative analysis of real-time messages in big data pipeline architecture

International Journal of High Performance Computing and Networking, 2019

Nowadays, real-time messaging system is the essential thing in enabling time-critical decision ma... more Nowadays, real-time messaging system is the essential thing in enabling time-critical decision making in many applications where it is important to deal with real-time requirements and reliability requirements simultaneously. For dependability reasons, we intend to maximise the reliability requirement of the real-time messaging system. To develop a real-time messaging system, we create real-time big data pipeline by using Apache Kafka and Apache Storm. This paper focuses on analysing the performance of producer and consumer in Apache Kafka processing. Apache Kafka is the most popular framework used to ingest the data streams into the processing platforms. The comparative analysis of Kafka processing is more efficient to get reliable data on the pipeline architecture. Then, the experiment will be conducted the processing time in the performance of the producer and consumer on various partitions and many servers. The performance analysis of Kafka can impact on messaging systems in real-time big data pipeline architecture.

Download

Enhancement of Fault Tolerance in Kafka Pipeline Architecture

Proceedings of the 11th International Conference on Advances in Information Technology

Fault tolerance act as the most important factor to improve the reliability of the real-time mess... more Fault tolerance act as the most important factor to improve the reliability of the real-time messaging system. Checkpointing and replication are the main factors to promote the performance of fault tolerance. The paper emphasizes the message recovery scheme based on checkpointing. Apache Kafka is a highly scalable publishing-subscribing messaging system. However, the message recovery process becomes one of the challenges when server failure occurred. Apache Kafka needs to catch up with the lost messages in the failure case. To overcome the problem, Kafka requires to check the failure case by identifying the optimal checkpoint interval in Kafka configuration. The paper focuses on editing the default checkpoint interval (DCI) to recover the lost messages. Kafka needs to estimate the overall total overhead cost after defining the optimal checkpoint interval. The proposed system intends to recover lost messages and reduce the total overhead cost, according to the evaluation of Kafka. The paper measures the total overhead cost by using a fixed checkpoint interval (FCI) method. Conforming to the experimental results, the system achieves the problems of message loss and improve the main factor of fault tolerance in Kafka. The system also measures the total overhead cost by using a fixed checkpoint interval (FCI) method. Experimental results outperform the message recovery and save time approximately 50% than the original system.