Skip to main content

Yanbin Liu

Followers

0

Following

1

Public Views

University of California, Santa Barbara

University of British Columbia

National University of "Kyiv-Mohyla Academy"

Javier Díaz Noci

Pompeu Fabra University

Kansas State University

Armando Marques-Guedes

UNL - New University of Lisbon

Università di Bologna

Andrew W Wilkins

Goldsmiths, University of London

Tulane University

Guilherme Moerbeck

UERJ - Universidade do Estado do Rio de Janeiro / Rio de Janeiro State University

Interests

Uploads

Papers by Yanbin Liu

Run-time performance optimization of a BigData query language

Proceedings of the 5th ACM/SPEC international conference on Performance engineering, 2014

JAQL is a query language for large-scale data that connects BigData analytics and MapReduce frame... more JAQL is a query language for large-scale data that connects BigData analytics and MapReduce framework together. Also an IBM product, JAQL's performance is critical for IBM In-foSphere BigInsights, a BigData analytics platform. In this paper, we report our work on improving JAQL performance from multiple perspectives. We explore the parallelism of JAQL, profile JAQL for performance analysis, identify I/O as the dominant performance bottleneck, and improve JAQL performance with an emphasis on reducing I/O data size and increasing (de)serialization efficiency. With TPCH benchmark on a simple Hadoop cluster, we report up to 2x performance improvements in JAQL with our optimization fixes.

Looking for an Evolution of Grid Scheduling: Meta-Brokering

Grid Middleware and Services, 2008

A Grid Resource Broker for a Grid domain, or also known as meta-scheduler, is a middleware compon... more A Grid Resource Broker for a Grid domain, or also known as meta-scheduler, is a middleware component used for matching works to available Grid resources from one or more IT organizations. A Grid meta-scheduler usually has its own interfaces for the functionalities it provides and its own job scheduling objectives. This situation causes two main problems: the user uniform access to the Grid is lost, and the scheduling decisions are taken separately while they should be done in coordination. These problems have been observed in different efforts such as the HPC-Europa project but they are still open problems. In this paper we discuss the requirements to achieve a more uniform access to the Grids through a new approach to global brokering. As the results of these discussions on brokering requirements, we propose a meta-brokering design, so called metameta-scheduler design, and discuss how it can be realized as a centralized model for the HPC-Europa project, and as a distributed model for the LA Grid project.

Efficiency Assessment of Parallel Workloads on Virtualized Resources

2011 Fourth IEEE International Conference on Utility and Cloud Computing, 2011

In cloud computing, virtual containers on physical resources are provisioned to requesting users.... more In cloud computing, virtual containers on physical resources are provisioned to requesting users. Resource providers may pack as many containers as possible onto each of their physical machines, or may pack specific types and quantities of virtual containers based on user or system QoS objectives. Such elastic provisioning schemes for resource sharing may present major challenges to scientific parallel applications that require task synchronization during execution. Such elastic schemes may also inadvertently lower utilization of computing resources. In this paper, we describe the elasticity constraint effect and ripple effect that cause a negative impact to application response time and system utilization. We quantify the impact using real workload traces through simulation. Then, we demonstrate that some resource scheduling techniques can be effective in mitigating the impacts. We find that a tradeoff is needed among the elasticity of virtual containers, the complexity of scheduling algorithms, and the response time of applications.

New Metrics for Scheduling Jobs on Cluster of Virtual Machines

2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum, 2011

As the virtualization of resources becomes popular, the scheduling problem of batch jobs on virtu... more As the virtualization of resources becomes popular, the scheduling problem of batch jobs on virtual machines requires new approaches. The dynamic and sharing aspects of virtual machines introduce unique challenges and complexity for the scheduling problems of batch jobs. In this paper, we propose a new set of metrics, called potential capacity (PC) and equilibrium capacity (EC), of resources that incorporate these dynamic, elastic, and sharing aspects of co-located virtual machines. We then show that we mesh this set of metrics smoothly into traditional scheduling algorithms. We evaluate the performance in using the metrics in a widely used greedy scheduling algorithm and show that the new scheduler improves job speedup for various configurations when compared to a similar algorithm using traditional physical machine metrics such as available CPU capacity.

Cloud federation in a layered service model

Journal of Computer and System Sciences, 2012

We show how a layered Cloud service model of software (SaaS), platform (PaaS), and infrastructure... more We show how a layered Cloud service model of software (SaaS), platform (PaaS), and infrastructure (IaaS) leverages multiple independent Clouds by creating a federation among the providers. The layered architecture leads naturally to a design in which inter-Cloud federation takes place at each service layer, mediated by a broker specific to the concerns of the parties at that layer. Federation increases consumer value for and facilitates providing IT services as a commodity. This business model for the Cloud is consistent with broker mediated supply and service delivery chains in other commodity sectors such as finance and manufacturing. Concreteness is added to the federated Cloud model by considering how it works in delivering the Weather Research and Forecasting service (WRF) as SaaS using PaaS and IaaS support. WRF is used to illustrate the concepts of delegation and federation, the translation of service requirements between service layers, and inter-Cloud broker functions needed to achieve federation.

P4p: provider portal for applications

ACM SIGCOMM Conference, 2008

As peer-to-peer (P2P) emerges as a major paradigm for scalable network application design, it als... more As peer-to-peer (P2P) emerges as a major paradigm for scalable network application design, it also exposes significant new chal- lenges in achieving efficient and fair utilization of Internet network resources. Being largely network-oblivious, many P2P applications may lead to inefficient network resource usage and/or low applica- tion performance. In this paper, we propose a simple architecture called P4P to allow

Enabling Interoperability among Grid Meta-Schedulers

Journal of Grid Computing, 2013

commonly referred to as the meta-scheduler or meta-broker. In this paper, we present an approach ... more commonly referred to as the meta-scheduler or meta-broker. In this paper, we present an approach to a meta-scheduler architecture, combining hierarchical and peer-to-peer models for flexibility and extensibility. Interoperability is further promoted through the introduction of a set of protocols, allowing meta-schedulers to maintain sessions and exchange job and resource state using Web Services. Our architecture also incorporates a resource model that enables an efficient resource matching across multiple Virtual Organizations, especially where the compute resources and state are dynamic. Experiments demonstrate these new functional features across three distributed organizations (BSC, FIU, and IBM), that internally use different job scheduling technologies, computing infrastructure and security mechanisms. Performance evaluations through actual system measurements and simulations provide the insights on the architecture's effectiveness and scalability.

Active and real-time functionalities for electronic brokerage design

Proceedings of International Workshop on Advance Issues of E-Commerce and Web-Based Information Systems. (Cat. No.PR00334)

In this position paper, we discuss an architecture for developing next generation electronic brok... more

An experimental system for grid meta-broker evaluation

Proceedings of the 1st ACM workshop on Large-Scale system and application performance - LSAP '09, 2009

Grid meta-broker is a key enabler in realizing the full potential of inter-operating grid computi... more Grid meta-broker is a key enabler in realizing the full potential of inter-operating grid computing systems. A challenge to properly evaluate the effectiveness of meta-brokers is the complexity of developing a realistic grid experimental environment. In this paper, this challenge is addressed by a unique combination of two approaches: using reduced workload traces to demonstrate the resource matching and scheduling functions of the meta-broker, and using emulation to provide a flexible and scalable modeling and management for local resources of a grid environment. Real workload traces are reduced while preserving their key workload characteristics to allow exploration of various dimensions of meta-broker functions in reasonable time. Evaluation of roundrobin, queue-length, and utilization based meta-broker scheduling algorithms shows that they have different effects on various workloads.

Enabling Autonomic Meta-Scheduling in Grid Environments

2008 International Conference on Autonomic Computing, 2008

Grid computing supports workload execution on computing resources that are shared across a set of... more Grid computing supports workload execution on computing resources that are shared across a set of collaborative organizations. At the core of workload management for Grid computing is a software component, called meta-scheduler or Grid resource broker, that provides a virtual layer on top of heterogeneous Grid middleware, schedulers, and resources. Meta-schedulers typically enable end-users and applications to compete over distributed shared resources through the use of one or more instances of the same meta-scheduler, in a centralized or distributed manner, respectively. We propose an approach to enabling autonomic meta-scheduling through the use of a new communication protocol that-if adopted by different meta-schedulers or by the applications using themcan improve the workload execution while avoiding potential chaos, which can be resulted from blind competition over resources. This can be made possible by allowing the metaschedulers and/or their applications to engage in a process to negotiate their roles (e.g., consumer, provider, or both), scheduling policies, service-level agreement, etc. To show the feasibility of our approach, we developed a prototype that enables some preliminary autonomic management among three different meta-schedulers, namely, GridWay, eNANOS, and TDWB.

An integrated approach for applying dynamic voltage scaling to hard real-time systems

The 9th IEEE Real-Time and Embedded Technology and Applications Symposium, 2003. Proceedings.

Wireless and portable devices depend on the limited power supplied by the battery. Dynamic Voltag... more Wireless and portable devices depend on the limited power supplied by the battery. Dynamic Voltage Scaling (DVS) is an effective method to reduce CPU power consumption. For real-time systems, DVS algorithms must not only provide enough CPU cycles, but also guarantee that no job misses its deadline. In this paper, we propose an integrated approach for applying DVS to real-time systems. We define two functions, the available cycle function (ACF) and the required cycle function (RCF), to capture the CPU workload of the real-time tasks. We then formulate the DVS scheduling problem for real-time systems as a nonlinear optimization problem and propose an optimal off-line algorithm to solve this problem. We also propose a novel online algorithm with time complexity O(1) to further reduce power consumption when a job uses fewer execution cycles than the worst-case budget. The algorithms in this paper are based solely on ACF and RCF, and may be applied to different scheduling policies. We illustrate the generality of our approach over previous research by applying our method to EDF and RM scheduling policies and deriving the optimal off-line DVS algorithms for them. Our simulation results show significant improvement over previous work.

Run-time performance optimization of a BigData query language

Proceedings of the 5th ACM/SPEC international conference on Performance engineering, 2014

JAQL is a query language for large-scale data that connects BigData analytics and MapReduce frame... more JAQL is a query language for large-scale data that connects BigData analytics and MapReduce framework together. Also an IBM product, JAQL's performance is critical for IBM In-foSphere BigInsights, a BigData analytics platform. In this paper, we report our work on improving JAQL performance from multiple perspectives. We explore the parallelism of JAQL, profile JAQL for performance analysis, identify I/O as the dominant performance bottleneck, and improve JAQL performance with an emphasis on reducing I/O data size and increasing (de)serialization efficiency. With TPCH benchmark on a simple Hadoop cluster, we report up to 2x performance improvements in JAQL with our optimization fixes.

Looking for an Evolution of Grid Scheduling: Meta-Brokering

Grid Middleware and Services, 2008

A Grid Resource Broker for a Grid domain, or also known as meta-scheduler, is a middleware compon... more A Grid Resource Broker for a Grid domain, or also known as meta-scheduler, is a middleware component used for matching works to available Grid resources from one or more IT organizations. A Grid meta-scheduler usually has its own interfaces for the functionalities it provides and its own job scheduling objectives. This situation causes two main problems: the user uniform access to the Grid is lost, and the scheduling decisions are taken separately while they should be done in coordination. These problems have been observed in different efforts such as the HPC-Europa project but they are still open problems. In this paper we discuss the requirements to achieve a more uniform access to the Grids through a new approach to global brokering. As the results of these discussions on brokering requirements, we propose a meta-brokering design, so called metameta-scheduler design, and discuss how it can be realized as a centralized model for the HPC-Europa project, and as a distributed model for the LA Grid project.

Efficiency Assessment of Parallel Workloads on Virtualized Resources

2011 Fourth IEEE International Conference on Utility and Cloud Computing, 2011

In cloud computing, virtual containers on physical resources are provisioned to requesting users.... more In cloud computing, virtual containers on physical resources are provisioned to requesting users. Resource providers may pack as many containers as possible onto each of their physical machines, or may pack specific types and quantities of virtual containers based on user or system QoS objectives. Such elastic provisioning schemes for resource sharing may present major challenges to scientific parallel applications that require task synchronization during execution. Such elastic schemes may also inadvertently lower utilization of computing resources. In this paper, we describe the elasticity constraint effect and ripple effect that cause a negative impact to application response time and system utilization. We quantify the impact using real workload traces through simulation. Then, we demonstrate that some resource scheduling techniques can be effective in mitigating the impacts. We find that a tradeoff is needed among the elasticity of virtual containers, the complexity of scheduling algorithms, and the response time of applications.

New Metrics for Scheduling Jobs on Cluster of Virtual Machines

2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum, 2011

As the virtualization of resources becomes popular, the scheduling problem of batch jobs on virtu... more As the virtualization of resources becomes popular, the scheduling problem of batch jobs on virtual machines requires new approaches. The dynamic and sharing aspects of virtual machines introduce unique challenges and complexity for the scheduling problems of batch jobs. In this paper, we propose a new set of metrics, called potential capacity (PC) and equilibrium capacity (EC), of resources that incorporate these dynamic, elastic, and sharing aspects of co-located virtual machines. We then show that we mesh this set of metrics smoothly into traditional scheduling algorithms. We evaluate the performance in using the metrics in a widely used greedy scheduling algorithm and show that the new scheduler improves job speedup for various configurations when compared to a similar algorithm using traditional physical machine metrics such as available CPU capacity.

Cloud federation in a layered service model

Journal of Computer and System Sciences, 2012

We show how a layered Cloud service model of software (SaaS), platform (PaaS), and infrastructure... more We show how a layered Cloud service model of software (SaaS), platform (PaaS), and infrastructure (IaaS) leverages multiple independent Clouds by creating a federation among the providers. The layered architecture leads naturally to a design in which inter-Cloud federation takes place at each service layer, mediated by a broker specific to the concerns of the parties at that layer. Federation increases consumer value for and facilitates providing IT services as a commodity. This business model for the Cloud is consistent with broker mediated supply and service delivery chains in other commodity sectors such as finance and manufacturing. Concreteness is added to the federated Cloud model by considering how it works in delivering the Weather Research and Forecasting service (WRF) as SaaS using PaaS and IaaS support. WRF is used to illustrate the concepts of delegation and federation, the translation of service requirements between service layers, and inter-Cloud broker functions needed to achieve federation.

P4p: provider portal for applications

ACM SIGCOMM Conference, 2008

As peer-to-peer (P2P) emerges as a major paradigm for scalable network application design, it als... more As peer-to-peer (P2P) emerges as a major paradigm for scalable network application design, it also exposes significant new chal- lenges in achieving efficient and fair utilization of Internet network resources. Being largely network-oblivious, many P2P applications may lead to inefficient network resource usage and/or low applica- tion performance. In this paper, we propose a simple architecture called P4P to allow

Enabling Interoperability among Grid Meta-Schedulers

Journal of Grid Computing, 2013

commonly referred to as the meta-scheduler or meta-broker. In this paper, we present an approach ... more commonly referred to as the meta-scheduler or meta-broker. In this paper, we present an approach to a meta-scheduler architecture, combining hierarchical and peer-to-peer models for flexibility and extensibility. Interoperability is further promoted through the introduction of a set of protocols, allowing meta-schedulers to maintain sessions and exchange job and resource state using Web Services. Our architecture also incorporates a resource model that enables an efficient resource matching across multiple Virtual Organizations, especially where the compute resources and state are dynamic. Experiments demonstrate these new functional features across three distributed organizations (BSC, FIU, and IBM), that internally use different job scheduling technologies, computing infrastructure and security mechanisms. Performance evaluations through actual system measurements and simulations provide the insights on the architecture's effectiveness and scalability.

Active and real-time functionalities for electronic brokerage design

Proceedings of International Workshop on Advance Issues of E-Commerce and Web-Based Information Systems. (Cat. No.PR00334)

In this position paper, we discuss an architecture for developing next generation electronic brok... more

An experimental system for grid meta-broker evaluation

Proceedings of the 1st ACM workshop on Large-Scale system and application performance - LSAP '09, 2009

Grid meta-broker is a key enabler in realizing the full potential of inter-operating grid computi... more Grid meta-broker is a key enabler in realizing the full potential of inter-operating grid computing systems. A challenge to properly evaluate the effectiveness of meta-brokers is the complexity of developing a realistic grid experimental environment. In this paper, this challenge is addressed by a unique combination of two approaches: using reduced workload traces to demonstrate the resource matching and scheduling functions of the meta-broker, and using emulation to provide a flexible and scalable modeling and management for local resources of a grid environment. Real workload traces are reduced while preserving their key workload characteristics to allow exploration of various dimensions of meta-broker functions in reasonable time. Evaluation of roundrobin, queue-length, and utilization based meta-broker scheduling algorithms shows that they have different effects on various workloads.

Enabling Autonomic Meta-Scheduling in Grid Environments

2008 International Conference on Autonomic Computing, 2008

Grid computing supports workload execution on computing resources that are shared across a set of... more Grid computing supports workload execution on computing resources that are shared across a set of collaborative organizations. At the core of workload management for Grid computing is a software component, called meta-scheduler or Grid resource broker, that provides a virtual layer on top of heterogeneous Grid middleware, schedulers, and resources. Meta-schedulers typically enable end-users and applications to compete over distributed shared resources through the use of one or more instances of the same meta-scheduler, in a centralized or distributed manner, respectively. We propose an approach to enabling autonomic meta-scheduling through the use of a new communication protocol that-if adopted by different meta-schedulers or by the applications using themcan improve the workload execution while avoiding potential chaos, which can be resulted from blind competition over resources. This can be made possible by allowing the metaschedulers and/or their applications to engage in a process to negotiate their roles (e.g., consumer, provider, or both), scheduling policies, service-level agreement, etc. To show the feasibility of our approach, we developed a prototype that enables some preliminary autonomic management among three different meta-schedulers, namely, GridWay, eNANOS, and TDWB.

An integrated approach for applying dynamic voltage scaling to hard real-time systems

The 9th IEEE Real-Time and Embedded Technology and Applications Symposium, 2003. Proceedings.

Wireless and portable devices depend on the limited power supplied by the battery. Dynamic Voltag... more Wireless and portable devices depend on the limited power supplied by the battery. Dynamic Voltage Scaling (DVS) is an effective method to reduce CPU power consumption. For real-time systems, DVS algorithms must not only provide enough CPU cycles, but also guarantee that no job misses its deadline. In this paper, we propose an integrated approach for applying DVS to real-time systems. We define two functions, the available cycle function (ACF) and the required cycle function (RCF), to capture the CPU workload of the real-time tasks. We then formulate the DVS scheduling problem for real-time systems as a nonlinear optimization problem and propose an optimal off-line algorithm to solve this problem. We also propose a novel online algorithm with time complexity O(1) to further reduce power consumption when a job uses fewer execution cycles than the worst-case budget. The algorithms in this paper are based solely on ACF and RCF, and may be applied to different scheduling policies. We illustrate the generality of our approach over previous research by applying our method to EDF and RM scheduling policies and deriving the optimal off-line DVS algorithms for them. Our simulation results show significant improvement over previous work.