Zhaojie Niu

Followers

Following

Public Views

James Elkins

School of the Art Institute of Chicago

David Kirsh

University of California, San Diego

Marc Champagne

Kwantlen Polytechnic University

John Sutton

Macquarie University

Ibrahim Cahit

Near East University

Fernando H Fukuda

UVA Universidade Veiga de Almeida

Viacheslav Kuleshov

Stockholm University

Mark Lewis

Trinity University, Texas

Angela Lima Peres

UNIVERSIDADE ESTADUAL DE CIENCIAS DA SAÚDE DE ALAGOAS - UNCISAL

PALIMOTE JUSTICE

RIVERS STATE POLYTECHNIC

Interests

Uploads

Papers by Zhaojie Niu

A Study of Big Data Computing Platforms: Fairness and Energy Consumption

Download

An Adaptive Efficiency-Fairness Meta-Scheduler for Data-Intensive Computing

IEEE Transactions on Services Computing, Nov 1, 2019

In data-intensive cluster computing platforms such as Hadoop YARN, efficiency and fairness are tw... more In data-intensive cluster computing platforms such as Hadoop YARN, efficiency and fairness are two important factors for system design and optimizations. Previous studies are either for efficiency or for fairness solely, without considering the tradeoff between efficiency and fairness. Recent studies observe that there is a tradeoff between efficiency and fairness because of resource contention between users/jobs. By leveraging the existing schedulers, a meta-scheduler is able to dynamically choose one of them for job/task scheduling at runtime. In this paper, we propose a meta-scheduler called FLEX to realize the tradeoff between system efficiency and fairness in Hadoop YARN. FLEX combines multiple existing schedulers into a single aggregated view without any modification on the original schedulers. Equipped with these candidate schedulers, FLEX utilizes machine learning approach to adaptively choose the most proper scheduler according to the characteristic of current running workload and user-defined SLA (Service Level Agreement). We implement FLEX in Hadoop YARN. We conduct experiments with real deployment in a local cluster and perform simulation studies with production traces. Experimental results show that the FLEX outperforms the state-of-the-art approach in two aspects: 1) Given a predefined threshold on the fairness loss, the FLEX reduces the makespan by up to 22% and 24% in real deployment and the large-scale simulation, respectively; 2) Given the predefined threshold on the makespan reduction, the FLEX reduces the fairness loss by up to 75% and 73% in real deployment and the large-scale simulation, respectively.

Download

JouleMR: Towards Cost-Effective and Green-Aware Data Processing Frameworks

IEEE Transactions on Big Data, Jun 1, 2018

Interests have been growing in energy management of the cluster effectively in order to reduce th... more Interests have been growing in energy management of the cluster effectively in order to reduce the energy consumption as well as the electricity cost. Renewable energy and dynamic pricing schemes in smart grids are two major emerging trends in energy markets. However, current data processing frameworks are not aware of the efficiency of each joule consumed by the data center workloads in the context of these two major trends. In fact, not all joules are equal in the sense that the amount of work that can be done by a joule can vary significantly in data centers. Ignoring this fact leads to significant energy waste (by 25% of the total energy consumption in Hadoop YARN on a Facebook production trace according to our study). In this paper, we propose JouleMR, a cost-effective and green-aware data processing framework. Specifically, we investigate how to exploit such joule efficiency to maximize the benefits of renewable energy as well as dynamic pricing schemes for MapReduce framework. We develop job/task scheduling algorithms with a particular focus on the factors on joule efficiency in the data center, including the energy efficiency of MapReduce workloads, renewable energy supply, dynamic pricing and the battery usage. We further develop a simple yet effective performance-energy consumption model to guide our scheduling decisions. We have implemented JouleMR on top of Hadoop YARN. The experiments demonstrate the accuracy of our models, and the effectiveness of our cost-effective and green-aware optimizations outperform the state-of-the-art implementations over Hadoop YARN.

Download

Long-Term Multi-Resource Fairness for Pay-as-you Use Computing Systems

IEEE Transactions on Parallel and Distributed Systems, May 1, 2018

Many current computing systems such as clouds and supercomputers charge users for their resource ... more Many current computing systems such as clouds and supercomputers charge users for their resource usages. A user's demand is often changing over time, indicating that it is difficult to keep the high resource utilization all the time for cost efficiency. Resource sharing is a classical and effective approach for high resource utilization. In view of the heterogeneous resource demands of users' workloads, multi-resource allocation fairness is a must for resource sharing in such pay-as-you-use computing systems. However, we find that, existing multi-resource fair policies such as Dominant Resource Fairness (DRF), implemented in currently popular resource management systems such as Apache YARN [4] and Mesos [23], are not suitable for the pay-as-you-use computing systems. We show that this is because of their memoryless characteristic that can cause the following problems in the pay-as-you-use computing systems: 1). users can get resource benefits by cheating; 2). users might not be able to get the total amount of resources that they are entitled to in terms of their resource contributions. In this paper, we propose a new policy called H-MRF, which generalizes DRF and Asset Fairness with the long-term notion. We show that it can address these problems and is suitable for pay-as-you-use computing systems. We have implemented it into YARN by developing a prototype called MRYARN. Finally, we evaluate H-MRF using both testbed and simulated experiments. The experimental results show that there are about

Download

Not All Joules are Equal: Towards Energy-Efficient and Green-Aware Data Processing Frameworks

Interests have been growing in integrating renewable energy into data centers, which attracts man... more Interests have been growing in integrating renewable energy into data centers, which attracts many research efforts in developing green-aware algorithms and systems. However, little attention was paid to the efficiency of each joule consumed by data center workloads. In fact, not all joules are equal in the sense that the amount of work that can be done by a joule can vary significantly in data centers. Ignoring this fact leads to significant energy waste (by 25% of the total energy consumption in Hadoop YARN on a Facebook production trace according to our study). In this paper, we investigate how to exploit such joule efficiency to maximize the benefits of renewable energy for MapReduce framework. We develop job/task scheduling algorithms with a particular focus on the factors on joule efficiency in the data center, including the energy efficiency of MapReduce workloads, renewable energy supply and the battery usage. We further develop a simple yet effective performance-energy consumption model to guide our scheduling decisions. We have implemented GreenMR, an energy-efficient and green-aware MapReduce framework, on top of Hadoop YARN. The experiments demonstrate the accuracy of our models, and the effectiveness of our energy-efficient and green-aware optimizations over Hadoop YARN and a state-ofthe-art green-aware Hadoop YARN implementation.

Download

Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing

In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are t... more In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are two important concerns for users. Existing studies show that, because of the resource contention between users/jobs, there is a tradeoff between the performance and fairness. In our work, we observe that such trade-off is related to the resource demand of the workload and is changing with the variation of multi-resource demand of submitted jobs during the computation. We also find that having an algorithm to be aware of the resource demand variation is important for the bi-criteria optimization between performance and fairness. However, most previous studies are not aware of this and design their heuristic algorithms with the assumption of fixed trade-off. In this paper, we propose a adaptive scheduler called Gemini for Hadoop YARN. For Gemini, it first develops a regression approach to construct a model which can estimate the performance improvement and the fairness loss under the sharing computation compared to the exclusive non-sharing scenario. Next, it leverages the model to guide the resource allocation for pending tasks to optimize the performance of the cluster given the user-defined fairness level. Instead of using a static scheduling policy, Gemini adaptively decides the proper scheduling policy according to the current running workload. We implement Gemini in Hadoop YARN. Experimental results show that Gemini outperforms the state-ofthe-art work in two aspects. 1) For the same fairness loss, Gemini increases the performance improvement up to 225% and 200% in real deployment and the large-scale simulation, respectively; 2) For the same performance improvement, Gemini reduces the fairness loss up to 70% and 62.5% in real deployment and the large-scale simulation, respectively.

Download

Storage Technologies for Big Data

Springer eBooks, 2019

Two Dimensional Overlay Peer-to-Peer Networks

... objects and connections in a primary group G whose objects OID satisfying h'(h(OID)) = (... more ... objects and connections in a primary group G whose objects OID satisfying h'(h(OID)) = (h'(h(NID)) + 1) % M (denoted as PG'(h(NID)) + 1) % M for ... The setting of our simulation experiments is; we use SHA-1 as the first hash function, and the hash space is set to 2128, we use a ...

Multi-objective Optimizations in Geo-Distributed Data Analytics Systems

In geographically distributed data centers, data analytics systems have recently been developed a... more In geographically distributed data centers, data analytics systems have recently been developed and optimized for such geo-distributed environments. With respect to various system operators’ requirements on data analytics, existing studies have optimized systems for individual goals such as resource efficiency, per-job latency and fairness. However, the optimizations with multiple objectives simultaneously have been overlooked. Even worse, some objectives can be translated to discordant actions and their relationship can be impacted by the unique features of geo-distributed data analytics systems. For example, we have observed clear trade-off between fairness and resource efficiency. In this paper, we develop an efficient framework for multi-objective optimizations on geo-distributed data analytics systems. Specifically, we develop GeoSpark, an extension to Spark, which automatically performs a multi-objective optimization according to the system operators’ preferences on different objectives. The multi-objective optimization is inherently intractable especially for large-scale workloads. Therefore, we propose an efficient online heuristic to approximate the optimal scheduling plan while achieving a lower bound guarantee in the worst case. Evaluation using synthetic workload shows that GeoSpark effectively performs the multi-objective optimizations based on system operators’ preferences on different objectives. GeoSpark achieves up to 30% makespan reduction, 28% job latency reduction and better fairness guarantee compared with existing schedulers in Apache Spark in the geo-distributed setting.

Elastic multi-resource fairness: balancing fairness and efficiency in coupled CPU-GPU architectures

IEEE International Conference on High Performance Computing, Data, and Analytics, Nov 13, 2016

Fairness and efficiency are two important concerns for users in a shared computer system, and the... more Fairness and efficiency are two important concerns for users in a shared computer system, and there tends to be a tradeoff between them. Heterogeneous computing poses new challenging issues on the fair allocation of computational resources among users due to the availability of different kinds of computing devices (e.g., CPU and GPU). Prior work either considers the fair resource allocation separately for each computing device or is unable to balance flexibly the tradeoff between the fairness and system utilization. In this work, we consider an emerging heterogeneous computing system with coupled CPU and GPU into a single chip. We first show that it is essential to have a new fair policy for coupled CPU-GPU architectures that is capable of considering both the CPU and the GPU as a whole in fair resource allocation and being aware of the system utilization maximization. We then propose a fair policy called Elastic Multi-Resource Fairness (EMRF) for coupled CPU-GPU architectures, by modeling CPU and GPU as two resource types and viewing the resource fairness problem as a multi-resource fairness problem. It extends DRF by adding a knob that allows users to tune and balance fairness and performance flexibly, and considers the fair allocation of computational resources as a whole for CPU and GPU devices. We show that EMRF satisfies fairness properties of sharing incentive, envy-freeness and pareto efficiency. Finally, we evaluate EMRF using real experiments, and the results show that EMRF can achieve better performance and fairness.

Download

弾性資源公平性:結合CPU‐GPUアーキテクチャにおける平衡公平性と効率【Powered by NICT】

IEEE Conference Proceedings, 2016

Storage Technologies for Big Data

Encyclopedia of Big Data Technologies

SilverWisdom : Towards a Knowledge Base for Elderly People

Active aging has become a hot research topic for both academia and industry. A lot of initiatives... more Active aging has become a hot research topic for both academia and industry. A lot of initiatives have been set up and delivered fruitful results and applications towards this emerging research area. As the society and population develops, many elderly people have high education background and knowledge, which enable various opportunities in knowledge sharing and learning, and even creating new knowledge. Relatively little attention has been paid to knowledge management for elderly people. In this position paper, we present a system named SilverWisdom towards a comprehensive knowledge base for elderly people. SilverWisdom manages the entire life cycle of knowledge, and fosters an infrastructure for multi-disciplinary research. Particularly, we present the system architecture of SilverWisdom and outline the open research problems. We believe that the success of SilverWisdom empowers active aging with knowledge and makes elderly people feel more fulfilled and socialized. Keyword: Know...

Download

A Study of Big Data Computing Platforms: Fairness and Energy Consumption

2016 IEEE International Conference on Cloud Engineering Workshop (IC2EW), 2016

Improving the performance is the common sense on those large-scale data processing frameworks and... more Improving the performance is the common sense on those large-scale data processing frameworks and fruitful studies are proposed in this direction. In contrast, the fairness and energy consumption of those frameworks need further exploration and how the performance, fairness and energy consumption interact each other on big data computing frameworks is not well addressed. In our research, we study the fairness and the energy consumption of those big data computing systems. We find that there are tradeoff between these factors. We conduct detailed studies on the factors which impact the tradeoff between different factors. Based on the observations in our study, we propose workload aware, energy-efficient and green-aware optimizations and implement them into Hadoop YARN. Particularly, in this thesis proposal, we propose to explore the following research problems. First, we explore the tradeoff between fairness and performance, and improve the performance of the state-of the-art approac...

Download

Elastic Multi-resource Fairness: Balancing Fairness and Efficiency in Coupled CPU-GPU Architectures

SC16: International Conference for High Performance Computing, Networking, Storage and Analysis

Download

Multi-objective Optimizations in Geo-Distributed Data Analytics Systems

2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS)

Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing

2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom)

In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are t... more In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are two important factors for system design and optimizations. Many previous studies are either for performance or for fairness solely, without considering the tradeoff between performance and fairness. Recent studies observe that there is a tradeoff between performance and fairness because of resource contention between users/jobs. However, their scheduling algorithms for bi-criteria optimization between performance and fairness are static, without considering the impact of different workload characteristics on the tradeoff between performance and fairness. In this paper, we propose an adaptive scheduler called Gemini for Hadoop YARN. We first develop a model with the regression approach to estimate the performance improvement and the fairness loss under the sharing computation compared to the exclusive non-sharing scenario. Next, we leverage the model to guide the resource allocation for pending tasks to optimize the performance of the cluster given the user-defined fairness level. Instead of using a static scheduling policy, Gemini adaptively decides the proper scheduling policy according to the current running workload. We implement Gemini in Hadoop YARN. Experimental results show that Gemini outperforms the state-of-the-art approach in two aspects. 1) For the same fairness loss, Gemini improves the performance by up to 225% and 200% in real deployment and the large-scale simulation, respectively; 2) For the same performance improvement, Gemini reduces the fairness loss up to 70% and 62.5% in real deployment and the large-scale simulation, respectively.

Download

A study of big data computing platforms : performance, fairness and energy consumption

Today, population aging is a trend spreading across the world. It is estimated that the number of... more Today, population aging is a trend spreading across the world. It is estimated that the number of people aged 60 years or over, is increasing rapidly. Although the elderly has wisdom and wealth gathered from their life experience, they often require long-term assistance from others. Ambient assisted living (AAL) systems open up a new opportunity to address the needs of aged by utilizing information and communication techniques. Multiple sensors and actuators coexist in ambient assisted living systems and are deployed everywhere surrounding elderly. A huge volume of sensed data is collected continuously from elderly, storage of these massive raw data and efficient processing on them to infer knowledge from the data and also guide the actuators to meet the needs of elderly is a problem we face. In the thesis, we mainly focus on the efficient large-scale data processing, we investigate whether and how we can build an effective large-scale data processing system to effectively excavate knowledge, wisdom and skills from elderly people to impact the development of the entire society. A lot of large-scale big data analytics systems have thereby emerged to process the massive data effectively. Efficient job scheduling and resource management for these data analytic frameworks are nontrivial. Modern job schedulers and resource coordinators in data processing frameworks often need to consider multiple objectives simultaneously due to various system operators requirements on data analytics. Currently, resource efficiency (throughput), job latency (per-job performance), fairness (isolation guarantee) and energy consumption are important concerns for the job scheduler in modern large-scale multiple-tenant environments. Resource efficiency is de facto the very important factor for the big data analytic framework. Job latency reflects the waiting time of the application. Fairness is a key building block of any multiple-tenant computing system that allows resource sharing effectively. Energy usage of the data center has reached 3% of the global electricity consumption while generating 200 million metric tons of CO2 in 2014. In order to reduce carbon emission and financial burden on the electricity, a lot of data centers have been redesigned and powered with multiple energy sources, including renewable (green) energy from non-polluting sources and brown energy from traditional polluting sources. Improving the resource efficiency and reducing the per-job latency are the common sense on those large-scale data processing frameworks and fruitful studies are proposed in these directions. In i Firstly, I express my thanks to my previous supervisor Dr. Bingsheng He, my current supervisor Dr. Lau Chiew Tong, my co-supervisor Dr. Ming Jian and my mentor Dr. Zhenchang Xing for their continuous, remarkable and patient guidance throughout my four year's study. Their immense knowledge, invaluable advice and unstinting encouragement have helped me a lot in my research. I would like to thank my friends and colleagues in IGS-LILY and PDSL. I will always remember all the great memories we created together in the part four years. I gratefully acknowledge the scholarship provided by LILY, IGS(Interdisciplinary Graduate School) and Nanyang Technological University for my doctorial program. Last but not least, I want to thank my family and friends for their unwavering support throughput my PhD.

Download

Appreciation to distributed and parallel databases reviewers

Distributed and Parallel Databases

Ailidani Ailijiang Murat Ali Bayir Nihat Altiparmak Douglas Alves Peixoto Vaibhav Arora Ira Assen... more Ailidani Ailijiang Murat Ali Bayir Nihat Altiparmak Douglas Alves Peixoto Vaibhav Arora Ira Assent Manos Athanassoulis Erman Ayday Samira Babalou Mehdi Bahrami Madhushi Bandara Fuat Basik Kaustubh Beedkar Ladjel Bellatreche Carsten Binnig Klemens Böhm Angela Bonifati Vanessa Braganholo Guadalupe Canahuate Alberto Cano Lei Cao Nicholas Car Fabio Casati Aniket Chakrabarti Lijun Chang Aleksey Charapko Shimin Chen Ling Chen Yu Cheng Fei Chiang Byron Choi Bin Cui Khuzaima Daudjee Engin Demir Murat Demirbas Anton Dignoes Bailu Ding Xiaofeng Ding Jaeyoung Do Bin Dong Hai Dong Laurent D’orazio Ahmed Eldawy Iman Elghandour Mohammed Eunus Ali Liyue Fan

Long-Term Multi-Resource Fairness for Pay-as-you Use Computing Systems

IEEE Transactions on Parallel and Distributed Systems

Download

A Study of Big Data Computing Platforms: Fairness and Energy Consumption

Download

An Adaptive Efficiency-Fairness Meta-Scheduler for Data-Intensive Computing

IEEE Transactions on Services Computing, Nov 1, 2019

Download

JouleMR: Towards Cost-Effective and Green-Aware Data Processing Frameworks

IEEE Transactions on Big Data, Jun 1, 2018

Download

Long-Term Multi-Resource Fairness for Pay-as-you Use Computing Systems

IEEE Transactions on Parallel and Distributed Systems, May 1, 2018

Download

Not All Joules are Equal: Towards Energy-Efficient and Green-Aware Data Processing Frameworks

Download

Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing

Download

Storage Technologies for Big Data

Springer eBooks, 2019

Two Dimensional Overlay Peer-to-Peer Networks

Multi-objective Optimizations in Geo-Distributed Data Analytics Systems

Elastic multi-resource fairness: balancing fairness and efficiency in coupled CPU-GPU architectures

IEEE International Conference on High Performance Computing, Data, and Analytics, Nov 13, 2016

Download

弾性資源公平性:結合CPU‐GPUアーキテクチャにおける平衡公平性と効率【Powered by NICT】

IEEE Conference Proceedings, 2016

Storage Technologies for Big Data

Encyclopedia of Big Data Technologies

SilverWisdom : Towards a Knowledge Base for Elderly People

Download

A Study of Big Data Computing Platforms: Fairness and Energy Consumption

2016 IEEE International Conference on Cloud Engineering Workshop (IC2EW), 2016

Download

Elastic Multi-resource Fairness: Balancing Fairness and Efficiency in Coupled CPU-GPU Architectures

SC16: International Conference for High Performance Computing, Networking, Storage and Analysis

Download

Multi-objective Optimizations in Geo-Distributed Data Analytics Systems

2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS)

Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing

2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom)

In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are t... more In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are two important factors for system design and optimizations. Many previous studies are either for performance or for fairness solely, without considering the tradeoff between performance and fairness. Recent studies observe that there is a tradeoff between performance and fairness because of resource contention between users/jobs. However, their scheduling algorithms for bi-criteria optimization between performance and fairness are static, without considering the impact of different workload characteristics on the tradeoff between performance and fairness. In this paper, we propose an adaptive scheduler called Gemini for Hadoop YARN. We first develop a model with the regression approach to estimate the performance improvement and the fairness loss under the sharing computation compared to the exclusive non-sharing scenario. Next, we leverage the model to guide the resource allocation for pending tasks to optimize the performance of the cluster given the user-defined fairness level. Instead of using a static scheduling policy, Gemini adaptively decides the proper scheduling policy according to the current running workload. We implement Gemini in Hadoop YARN. Experimental results show that Gemini outperforms the state-of-the-art approach in two aspects. 1) For the same fairness loss, Gemini improves the performance by up to 225% and 200% in real deployment and the large-scale simulation, respectively; 2) For the same performance improvement, Gemini reduces the fairness loss up to 70% and 62.5% in real deployment and the large-scale simulation, respectively.

Download

A study of big data computing platforms : performance, fairness and energy consumption

Download

Appreciation to distributed and parallel databases reviewers

Distributed and Parallel Databases

Long-Term Multi-Resource Fairness for Pay-as-you Use Computing Systems

IEEE Transactions on Parallel and Distributed Systems

Download

Zhaojie Niu

Related Authors

Uploads

Papers by Zhaojie Niu

Log In