Zirije Hasani

Followers

Following

Public Views

Greg Simons

Daffodil International University(DIU)

David Seamon

Kansas State University

Armando Marques-Guedes

UNL - New University of Lisbon

Hacer Topaktas Ustuner

Istanbul University

Florin Curta

University of Florida

Narek S. Galstyan Նարեկ Ս. Գալստյան

Yerevan State University

Nikolaos Tzifakis

University of the Peloponnese

Arolda Elbasani

European University Institute

António Raimundo

Universidade do Minho

jelena bogdanovic

Vanderbilt University

Interests

Uploads

Papers by Zirije Hasani

Proposal of model for prediction of grape processing and spraying time by using IoT smart agriculture sensor data

International Journal on Information Technologies and Security, Mar 1, 2024

Download

Calibration Techniques and Analyzing the Website Design with Eye Tracking Glasses

Lecture notes in networks and systems, 2024

Comparison of Predictive Algorithms for IOT Smart Agriculture Sensor Data

International journal of interactive mobile technologies, Nov 14, 2023

Download

Proposed Model for Real-Time Anomaly Detection in Big IoT Sensor Data for Smart City

International journal of interactive mobile technologies, Feb 9, 2024

A smart city represents an advanced urban environment that utilizes digital technologies to impro... more A smart city represents an advanced urban environment that utilizes digital technologies to improve the well-being of residents, efficiently manage urban operations, and prioritize long-term sustainability. These technologically advanced cities collect significant data through various Internet of Things (IoT) sensors, highlighting the crucial importance of detecting anomalies to ensure both efficient operation and security. However, real-time identification of anomalies presents challenges due to the sheer volume, rapidity, and diversity of the data streams. This manuscript introduces an innovative framework designed for the immediate detection of anomalies within extensive IoT sensor data in the context of a smart city. Our proposed approach integrates a combination of unsupervised machine learning techniques, statistical analysis, and expert feature engineering to achieve real-time anomaly detection. Through an empirical assessment of a practical dataset obtained from a smart city environment, we demonstrate that our model outperforms established techniques for anomaly detection.

Download

Robust anomaly detection algorithms for real-time big data: Comparison of algorithms

2017 6th Mediterranean Conference on Embedded Computing (MECO), 2017

Most of the today's world data are streaming, time-series data, where anomalies detection giv... more Most of the today's world data are streaming, time-series data, where anomalies detection gives significant information of possible critical situations. Yet, detecting anomalies in big streaming data is a difficult task, requiring detectors to acquire and process data in a real-time, as they occur, even before they are stored and instantly alarm on potential threats. Suitable to the need for real-time alarm and unsupervised procedures for massive streaming data anomaly detection, algorithms have to be robust, with low processing time, eventually at the cost of the accuracy. In this work we explore several such fast algorithms like MAD, RunMAD, Boxplot, Twitter ADVec, DBSCAN, Moving Range Technique, Statistical Control Chart Techniques, ARIMA and Moving Average. The algorithms are tested and results are visualized in the system R, on the three Numenta datasets, with known anomalies and own e-dnevnik dataset with unknown anomalies. Evaluation is done by comparing achieved results (the algorithm execution time, CPU usage and the number of anomalies found) with Numenta HTM algorithm that detects all the anomalies in their datasets. Our interest is monitoring of the streaming log data that are generating in the national educational network (e-dnevnk) that acquires a massive number of online queries and to detect anomalies in order to scale up performance, prevent network downs, alarm on possible attacks and similar.

An Adaptive Anomaly Detection Algorithm for Periodic Data Streams

Intelligent Data Engineering and Automated Learning – IDEAL 2018, 2018

Real Time Analytic of SQL Queries Based on Log Analytic

Analyzing huge amount of data are a big challenge. On one hand we are faced with the problem of s... more Analyzing huge amount of data are a big challenge. On one hand we are faced with the problem of storing a large amount of data, and on another to process it in a reasonable or even real time. Real time analytics can be defined as the capacity to use all available enterprise data and sources in the moment they arrive or happen in the system. In this paper, we present an infrastructure that we have implemented in order to analyze data from big log files in real time. The main components of the infrastructure are Redis, Logstash, Elasticsearch and Kibana. Redis is used for temporary buffering of the log data, Logstash utilizes different filters to manipulate and analyze the data, Elasticsearch is used for indexing and storing the data and Kibana is a user interface used to visualize the results. We explore implementation of several filters in order to post-process the log information and produce various statistics that suit our needs in analyzing log files containing SQL queries from a...

Download

Lambda Architecture for Real Time Big Data Analytic

Defining the environment for analyzing streamed big data in real time is not an easy task. There ... more Defining the environment for analyzing streamed big data in real time is not an easy task. There are many architecture proposals for real time big data analytic, but the most interesting one for our problem is Lambda Architecture. In this paper we are presenting motivation for developing such architecture, how it works and our practical work for implementing it. Lambda Architecture is comprised by three layers batch, speed and serving layer. Thus far we have implemented the batch layer employing Hadoop framework. We also briefly review the other two layers in order to implement them in the next phase of our work, where for serving and speed layer we conclude that Storm is the best choice. Practical example demonstrates the analytical process in Hadoop for analyzing Wikipedia text data.

Download

Implementation of Infrastructure for Streaming Outlier Detection in Big Data

In recent years the amount of data available at World Wide Web grew drastically. Therefore the ne... more In recent years the amount of data available at World Wide Web grew drastically. Therefore the need to analyze these large amount of data is a problem faced by researchers in this field. The problem that occurs is analyzing this data in real time before they are stored. Analyzing network log data in order to find system anomalies is another problem that researchers face. There are many tools that are used in order to detect anomalies in real time big data but in this research Fluentd is used. Submission of questionnaires in such large quantities of data is a way to take our decisions in various businesses and organizations. Another challenge is how to monitor the data generated in the network and to detect errors in order to scale up performance. To enable the infrastructure to detect anomalies in streaming data outlier detection plugin for Fluentd is implemented. Another important thing is how to visualize the result in order to have a legible result. In this paper we show the visu...

Real-Time Anomaly Detection Over Massive Data Streams

Real-time anomaly detection of massive data streams is one of the important research topics nowad... more Real-time anomaly detection of massive data streams is one of the important research topics nowadays due to the fact that the most of the world data is generated in continuous temporal processes. It addresses various problems in a lot of domains such as health, education, finance, government, etc. In this work, we propose an enhancement of this approach implemented in HW and TDHW forecasting models. The Genetic Algorithm (GA) is applied to periodically optimize HW and TDHW smoothing parameters in addition to the two sliding windows parameters that improve Hyndman's MASE measure of deviation and value of the threshold parameter that defines no anomaly confidence interval. We also propose a new optimization function based on the input training datasets with the annotated anomaly intervals to detect the right anomalies and reduce the number of false ones.

Improvement of Implemented Infrastructure for Streaming Outlier Detection in Big Data with ELK Stack

Nowadays the usage of internet is constantly increasing the amount of data. As a result the need ... more Nowadays the usage of internet is constantly increasing the amount of data. As a result the need for analyzing this data has recently emerged as we need to face a new phenomena known as the Big Data. This research is focused in finding appropriate architecture for real-time big data analytics and its main task is to detect anomalies in this real-time data. There are some tools that are used and analyzed by us in order to find the best one, but in this paper we use Timeline and compare it with Fluentd which is the tool we used in previous research [12]. Here we are going to show the reasons why Timelion is better than Fluentd. Anomaly detection in real-time big data is a problem that faces many organizations and it is a challenge for researchers as well. Our research deals with developing infrastructure for monitoring e-dnevnik (education national system in Macedonia) application server and to detect errors in order to scale up the performance. In order to enable this infrastructure ...

Survey and Proposal of an Adaptive Anomaly Detection Algorithm for Periodic Data Streams

Journal of Computer and Communications, 2019

Real-time anomaly detection of massive data streams is an important research topic nowadays due t... more Real-time anomaly detection of massive data streams is an important research topic nowadays due to the fact that a lot of data is generated in continuous temporal processes. There is a broad research area, covering mathematical, statistical, information theory methodologies for anomaly detection. It addresses various problems in a lot of domains such as health, education, finance, government, etc. In this paper, we analyze the state-of-the-art of data streams anomaly detection techniques and algorithms for anomaly detection in data streams (time series data). Critically surveying the techniques' performances under the challenge of real-time anomaly detection of massive high-velocity streams, we conclude that the modeling of the normal behavior of the stream is a suitable approach. We evaluate Holt-Winters (HW), Taylor's Double Holt-Winters (TDHW), Hierarchical temporal memory (HTM), Moving Average (MA), Autoregressive integrated moving average (ARIMA) forecasting models, etc. Holt-Winters (HW) and Taylor's Double Holt-Winters (TDHW) forecasting models are used to predict the normal behavior of the periodic streams, and to detect anomalies when the deviations of observed and predicted values exceeded some predefined measures. In this work, we propose an enhancement of this approach and give a short description about the algorithms and then they are categorized by type of prediction as: predictive and non-predictive algorithms. We implement the Genetic Algorithm (GA) to periodically optimize HW and TDHW smoothing parameters in addition to the two sliding windows parameters that improve Hyndman's MASE measure of deviation, and value of the threshold parameter that defines no anomaly confidence interval [1]. We also propose a new optimization function based on the input training datasets with the annotated anomaly intervals, in order to detect the right anomalies and minimize the number of false ones. The proposed method is evaluated on the known anomaly detection benchmarks NUMENTA and Yahoo datasets with annotated anomalies and real log data generated by the National education information system

Download

Infrastructure with r package for anomaly detection in real time big log data

Pressacademia, 2017

Analyzing and detecting anomalies in huge amount of data are a big challenge. On one hand we are ... more Analyzing and detecting anomalies in huge amount of data are a big challenge. On one hand we are faced with the problem of storing a large amount of data, on the other to process it and detect anomalies in reasonable or even real time. Real time analytics can be defined as the capacity to use all available enterprise data and sources in the moment they arrive or happen in the system. In this paper, we present an infrastructure that we have implemented in order to analyze data from big log files in real time. Also we present algorithms that are used for anomaly detection in big data. The algorithms are implemented in R language. The main components of the infrastructure are Redis, Logstash, Elasticsearch, elastic-R client and Kibana. We explore implementation of several filters in order to post-process the log information and produce various statistics that suit our needs in analyzing log files containing SQL queries from a big national system in education. The post-processing of the SQL queries is mainly focused on preparing the log information in adequate format and information extraction. The other interesting part of the paper is to compare the anomaly detection algorithms and to conclude which of them is better to us for our needs. Also we add the elastic-R client to the infrastructure we develop for big data analytic in order to detect anomalies. The purpose of the analysis is to monitor performance and detect anomalies in order to prevent possible problems in real time.

Download

Performance measurement with high performance computer of HW-GA anomaly detection algorithms for streaming data

Computer Science

Anomaly detection is very important in every sector as health, education, business, etc. Knowing ... more Anomaly detection is very important in every sector as health, education, business, etc. Knowing what is going wrong with data/digital system help peoples from every sector to take decision. Detection anomalies in real time Big Data is nowadays very crucial. Dealing with real time data requires speed, for this reason the aim of this paper is to measure the performance of our previously proposed HW-GA algorithm compared with other anomaly detection algorithms. Many factors will be analyzed which may affect the performance of HW-GA as visualization of result, amount of data and performance of computers. Algorithm execution time and CPU usage are the parameters which will be measured to evaluate the performance of HW-GA algorithm. Also, another aim of this paper is to test the HW-GA algorithm with large amount of data to verify if it will find the possible anomalies and the result to compare with other algorithms. The experiments will be done in R with different datasets as real data C...

Download

Anomaly Detection Algorithms for Streaming Data: Performance Comparison

Journal of Computer Science

Real Time Anomaly Detection in Massive Data Streams with ELK Stack

Journal of Computer Science

Real time anomaly detection is very popular topic nowadays this because the number of data genera... more Real time anomaly detection is very popular topic nowadays this because the number of data generated every day is larger and larger. Facing with the phenomena of Big Data is not an easy task. The main aim of this research is to fine appropriate architecture for real-time big data analytic and its main task is to detect anomalies in this real-time data. In this paper we show the implementation of anomaly detection algorithm in real time infrastructure in order to find anomalies as soon as possible. We have proposed architecture for real time anomaly detection by adding some new components and the main part of the infrastructure is Timelion which enable implementation of different algorithms for anomaly detection. The research is focused to develop infrastructure to monitor ednevnik (education national system in Macedonia) application server and to detect errors in order to scale up the performance.

Download

Performance Comparison of Anomaly Detection Algorithms for Streaming Data

Most of the today's world data are streaming, time-series data, where anomalies detection giv... more Most of the today's world data are streaming, time-series data, where anomalies detection gives significant information of possible critical situations. Yet, detecting anomalies in big streaming data is a difficult task, requiring detectors to acquire and process data in a real-time, as they occur, even before they are stored and instantly alarm on potential threats. Suitable to the need for real-time alarm and unsupervised procedures for massive streaming data anomaly detection, algorithms have to be robust, with low processing time, eventually at the cost of the accuracy. In this work we compare the performance of our proposed anomaly detection algorithm HW-GA[1] with other existing methods as ARIMA [10], Moving Average [11] and Holt Winters [12]. The algorithms are tested and results are visualized in the system R, on the three Numenta datasets, with known anomalies and own e-dnevnik dataset with unknown anomalies. Evaluation is done by comparing achieved results (the algorit...

Proposal of model for prediction of grape processing and spraying time by using IoT smart agriculture sensor data

International Journal on Information Technologies and Security, Mar 1, 2024

Download

Calibration Techniques and Analyzing the Website Design with Eye Tracking Glasses

Lecture notes in networks and systems, 2024

Comparison of Predictive Algorithms for IOT Smart Agriculture Sensor Data

International journal of interactive mobile technologies, Nov 14, 2023

Download

Proposed Model for Real-Time Anomaly Detection in Big IoT Sensor Data for Smart City

International journal of interactive mobile technologies, Feb 9, 2024

Download

Robust anomaly detection algorithms for real-time big data: Comparison of algorithms

2017 6th Mediterranean Conference on Embedded Computing (MECO), 2017

An Adaptive Anomaly Detection Algorithm for Periodic Data Streams

Intelligent Data Engineering and Automated Learning – IDEAL 2018, 2018

Real Time Analytic of SQL Queries Based on Log Analytic

Download

Lambda Architecture for Real Time Big Data Analytic

Download

Implementation of Infrastructure for Streaming Outlier Detection in Big Data

Real-Time Anomaly Detection Over Massive Data Streams

Improvement of Implemented Infrastructure for Streaming Outlier Detection in Big Data with ELK Stack

Survey and Proposal of an Adaptive Anomaly Detection Algorithm for Periodic Data Streams

Journal of Computer and Communications, 2019

Download

Infrastructure with r package for anomaly detection in real time big log data

Pressacademia, 2017

Download

Performance measurement with high performance computer of HW-GA anomaly detection algorithms for streaming data

Computer Science

Download

Anomaly Detection Algorithms for Streaming Data: Performance Comparison

Journal of Computer Science

Real Time Anomaly Detection in Massive Data Streams with ELK Stack

Journal of Computer Science

Download

Performance Comparison of Anomaly Detection Algorithms for Streaming Data

Most of the today's world data are streaming, time-series data, where anomalies detection giv... more Most of the today's world data are streaming, time-series data, where anomalies detection gives significant information of possible critical situations. Yet, detecting anomalies in big streaming data is a difficult task, requiring detectors to acquire and process data in a real-time, as they occur, even before they are stored and instantly alarm on potential threats. Suitable to the need for real-time alarm and unsupervised procedures for massive streaming data anomaly detection, algorithms have to be robust, with low processing time, eventually at the cost of the accuracy. In this work we compare the performance of our proposed anomaly detection algorithm HW-GA[1] with other existing methods as ARIMA [10], Moving Average [11] and Holt Winters [12]. The algorithms are tested and results are visualized in the system R, on the three Numenta datasets, with known anomalies and own e-dnevnik dataset with unknown anomalies. Evaluation is done by comparing achieved results (the algorit...

Zirije Hasani

Related Authors

Uploads

Papers by Zirije Hasani

Log In