Clusters, multi-cluster systems and the grid are becoming popular high performance computing infr... more Clusters, multi-cluster systems and the grid are becoming popular high performance computing infrastructures. How jobs are scheduled on them greatly influence the performance. Since jobs are online in most computer systems, they are scheduled when the entire information about them is unknown. We propose a greedy scheduling algorithm that uses both arrival order and how hard a job is to schedule while prioritizing. We compare its performance with that of the Fit processor First Served (FPFS) algorithm. We deduce performance dominance on different job streams.
Performance (and sensitivity) studies in parallel job scheduling mostly use average values of the... more Performance (and sensitivity) studies in parallel job scheduling mostly use average values of the measurement metrics over the entire job stream. This does not give an idea of relative job performance (hence starvation) and sensitivity to scheduler parameters. Some jobs can therefore be easily starved without being detected. We investigate the influence of jobs' physical characteristics on their schedulability in a multi cluster system using the Fit Processor First Served (FPFS) scheduler. We also investigate their relative strength in determining schedulability as well as their sensitivity to scheduler parameters. We deduce the implication of the findings in a practical scheduling scenario.
Query Optimization is an important process in Relational databases. With the current load on data... more Query Optimization is an important process in Relational databases. With the current load on databases increasing, the need to optimize queries in batches is a promising way out. Studies have shown that sharing among common sub-expressions can as well be beyond the optimal plans of the constituent queries. But challenges of excessively large sample space, buffer management, establishment of optimal order of optimization, and identification of disjoint queries remain in place. In this dissertation, We propose how We can efficiently establish the extent of inter-query shareability and exploit it so as to compute common sub-expressions once and share the output among the queries. We also propose the optimal order of optimization so that the sharing is done in a more cost saving and time conserving manner. v
Insufficient computational power, caused by limited budgets is a big challenge to high quality re... more Insufficient computational power, caused by limited budgets is a big challenge to high quality research and education in developing countries. Some institutions suffer acute shortages while others enjoy comparatively more resources. Many institutions therefore source for funds to procure more resources while paying less attention to optimally utilizing what they have or creating facilities for sharing with other institutions for mutual benefit. Current developments in Grid computing technology provide for mechanisms of consolidating isolated computing resources to provide a high power computing environment. In this project, we aim at studying ways (limited) computer resources can be optimally used in a financially constrained setting as well as ways for building an environment for providing high computing power for learning and research in a cost effective way.
Performance evaluation in multi-cluster processor co-allocation -like in many other parallel job ... more Performance evaluation in multi-cluster processor co-allocation -like in many other parallel job scheduling problems-is mostly done by computing the average metric value for the entire job stream. This does not give a comprehensive understanding of the relative performance of the different jobs grouped by their characteristics. It is however the characteristics that affect how easy/hard jobs are to schedule. We, therefore, do not get to understand scheduler performance at job type level. In this paper, we study the performance of multi-cluster processor co-allocation for different job groups grouped by their size, components and widest component. We study their relative performance, sensitivity to parameters and how their performance is affected by the heuristics used to break them up into components. We show that the widest component us characteristic that most affects job schedulability. We also show that to get better performance, jobs should be broken up in such a way that the width of the widest component is minimized.
Processor co-allocation can be of performance benefit. This is because breaking jobs into compone... more Processor co-allocation can be of performance benefit. This is because breaking jobs into components reduces overall cluster fragmentation. However, the slower inter-cluster communication links increase job execution times. This leads to performance deterioration which can make co-allocation unviable. We use intra-cluster to inter-cluster communication speed ratio and job communication intensity to model the job execution time penalty due to co-allocation. We then study viability of co-allocation in selected job and system based instances. We also study performance variation with selected job stream parameters. We observe that co-allocation is viable so long as the execution time penalty caused is relatively low. We also observe that the negative performance effect due to co-allocation is felt by the entire job stream rather than only the (few) co-allocated jobs. Finally, we observe that for every value of communication time penalty, there is a job size s * , where if all jobs whose size is greater than s * are co-allocated, we get the best performance.
Most schedulers in parallel job scheduling do not put (job) schedulability into consideration whe... more Most schedulers in parallel job scheduling do not put (job) schedulability into consideration when prioritizing jobs. Performance evaluation is mostly done using average values of the measurement metric. Using the average metric value may conceal relative job starvation details hence giving a shallower understanding of scheduler performance. We propose a greedy multi-cluster scheduler that uses the (estimate of) job schedulability and the time a job has spent in the queue to compute its priority. We compare the performance of our scheduler with that of Fit Processor First Served (FPFS) scheduler. We also study the sensitivity of its performance to parameter changes. We observe that (i) within some parameter ranges, our scheduler outperforms FPFS; (ii) for big jobs, our scheduler outperforms FPFS; for small jobs, FPFS outperforms our scheduler and (iii) our scheduler is fairer than FPFS.
Parallel job schedulers are mostly evaluated using performance metrics. Deductions however can be... more Parallel job schedulers are mostly evaluated using performance metrics. Deductions however can be misleading due to selective job starvation (unfairness). To choose a better scheduler, therefore, there is a need to compare schedulers for fairness as well. Performance and fairness, however, have mostly been studied independently. We examine characteristics of three approaches to fairness evaluation in parallel job scheduling. We examine how they represent job starvation and other aspects of discrimination. We show that the implied unfairness is not always starvation/discrimination in practice. We use simultaneous consideration of performance and fairness and compare deductions with scheduler effectiveness derived from group-wise performance evaluation. We observe that due to possible misrepresentation of starvation by fairness metrics, schedulers shown as superior may not be so in practice.
Fairness is an important aspect in queuing systems. Several fairness measures have been proposed ... more Fairness is an important aspect in queuing systems. Several fairness measures have been proposed in queuing systems in general and parallel job scheduling in particular. Generally, a scheduler is considered unfair if some jobs are discriminated while others are favored. Some of the metrics used to measure fairness for parallel job schedulers can imply unfairness where there is no discrimination (and vice versa). This makes them inappropriate. In this paper, we show how the existing misrepresents fairness in practice. We then propose a new approach for measuring fairness for parallel job schedulers. Our approach is based on two principles: (i) since jobs have different resource requirements and find different queue/system states, they need not to have the same performance for the scheduler to be fair and (ii) to compare two schedulers for fairness, we make comparisons of how the schedulers favor/discriminate individual jobs. We use performance and discrimination trends to validate our approach. We observe that our approach can deduce discrimination more accurately. This is true even in cases where the most discriminated jobs are not the worst performing jobs.
Clusters, multi-cluster systems and the grid are becoming popular high performance computing infr... more Clusters, multi-cluster systems and the grid are becoming popular high performance computing infrastructures. How jobs are scheduled on them greatly influence the performance. Since jobs are online in most computer systems, they are scheduled when the entire information about them is unknown. We propose a greedy scheduling algorithm that uses both arrival order and how hard a job is to schedule while prioritizing. We compare its performance with that of the Fit processor First Served (FPFS) algorithm. We deduce performance dominance on different job streams.
Performance (and sensitivity) studies in parallel job scheduling mostly use average values of the... more Performance (and sensitivity) studies in parallel job scheduling mostly use average values of the measurement metrics over the entire job stream. This does not give an idea of relative job performance (hence starvation) and sensitivity to scheduler parameters. Some jobs can therefore be easily starved without being detected. We investigate the influence of jobs' physical characteristics on their schedulability in a multi cluster system using the Fit Processor First Served (FPFS) scheduler. We also investigate their relative strength in determining schedulability as well as their sensitivity to scheduler parameters. We deduce the implication of the findings in a practical scheduling scenario.
Query Optimization is an important process in Relational databases. With the current load on data... more Query Optimization is an important process in Relational databases. With the current load on databases increasing, the need to optimize queries in batches is a promising way out. Studies have shown that sharing among common sub-expressions can as well be beyond the optimal plans of the constituent queries. But challenges of excessively large sample space, buffer management, establishment of optimal order of optimization, and identification of disjoint queries remain in place. In this dissertation, We propose how We can efficiently establish the extent of inter-query shareability and exploit it so as to compute common sub-expressions once and share the output among the queries. We also propose the optimal order of optimization so that the sharing is done in a more cost saving and time conserving manner. v
Insufficient computational power, caused by limited budgets is a big challenge to high quality re... more Insufficient computational power, caused by limited budgets is a big challenge to high quality research and education in developing countries. Some institutions suffer acute shortages while others enjoy comparatively more resources. Many institutions therefore source for funds to procure more resources while paying less attention to optimally utilizing what they have or creating facilities for sharing with other institutions for mutual benefit. Current developments in Grid computing technology provide for mechanisms of consolidating isolated computing resources to provide a high power computing environment. In this project, we aim at studying ways (limited) computer resources can be optimally used in a financially constrained setting as well as ways for building an environment for providing high computing power for learning and research in a cost effective way.
Performance evaluation in multi-cluster processor co-allocation -like in many other parallel job ... more Performance evaluation in multi-cluster processor co-allocation -like in many other parallel job scheduling problems-is mostly done by computing the average metric value for the entire job stream. This does not give a comprehensive understanding of the relative performance of the different jobs grouped by their characteristics. It is however the characteristics that affect how easy/hard jobs are to schedule. We, therefore, do not get to understand scheduler performance at job type level. In this paper, we study the performance of multi-cluster processor co-allocation for different job groups grouped by their size, components and widest component. We study their relative performance, sensitivity to parameters and how their performance is affected by the heuristics used to break them up into components. We show that the widest component us characteristic that most affects job schedulability. We also show that to get better performance, jobs should be broken up in such a way that the width of the widest component is minimized.
Processor co-allocation can be of performance benefit. This is because breaking jobs into compone... more Processor co-allocation can be of performance benefit. This is because breaking jobs into components reduces overall cluster fragmentation. However, the slower inter-cluster communication links increase job execution times. This leads to performance deterioration which can make co-allocation unviable. We use intra-cluster to inter-cluster communication speed ratio and job communication intensity to model the job execution time penalty due to co-allocation. We then study viability of co-allocation in selected job and system based instances. We also study performance variation with selected job stream parameters. We observe that co-allocation is viable so long as the execution time penalty caused is relatively low. We also observe that the negative performance effect due to co-allocation is felt by the entire job stream rather than only the (few) co-allocated jobs. Finally, we observe that for every value of communication time penalty, there is a job size s * , where if all jobs whose size is greater than s * are co-allocated, we get the best performance.
Most schedulers in parallel job scheduling do not put (job) schedulability into consideration whe... more Most schedulers in parallel job scheduling do not put (job) schedulability into consideration when prioritizing jobs. Performance evaluation is mostly done using average values of the measurement metric. Using the average metric value may conceal relative job starvation details hence giving a shallower understanding of scheduler performance. We propose a greedy multi-cluster scheduler that uses the (estimate of) job schedulability and the time a job has spent in the queue to compute its priority. We compare the performance of our scheduler with that of Fit Processor First Served (FPFS) scheduler. We also study the sensitivity of its performance to parameter changes. We observe that (i) within some parameter ranges, our scheduler outperforms FPFS; (ii) for big jobs, our scheduler outperforms FPFS; for small jobs, FPFS outperforms our scheduler and (iii) our scheduler is fairer than FPFS.
Parallel job schedulers are mostly evaluated using performance metrics. Deductions however can be... more Parallel job schedulers are mostly evaluated using performance metrics. Deductions however can be misleading due to selective job starvation (unfairness). To choose a better scheduler, therefore, there is a need to compare schedulers for fairness as well. Performance and fairness, however, have mostly been studied independently. We examine characteristics of three approaches to fairness evaluation in parallel job scheduling. We examine how they represent job starvation and other aspects of discrimination. We show that the implied unfairness is not always starvation/discrimination in practice. We use simultaneous consideration of performance and fairness and compare deductions with scheduler effectiveness derived from group-wise performance evaluation. We observe that due to possible misrepresentation of starvation by fairness metrics, schedulers shown as superior may not be so in practice.
Fairness is an important aspect in queuing systems. Several fairness measures have been proposed ... more Fairness is an important aspect in queuing systems. Several fairness measures have been proposed in queuing systems in general and parallel job scheduling in particular. Generally, a scheduler is considered unfair if some jobs are discriminated while others are favored. Some of the metrics used to measure fairness for parallel job schedulers can imply unfairness where there is no discrimination (and vice versa). This makes them inappropriate. In this paper, we show how the existing misrepresents fairness in practice. We then propose a new approach for measuring fairness for parallel job schedulers. Our approach is based on two principles: (i) since jobs have different resource requirements and find different queue/system states, they need not to have the same performance for the scheduler to be fair and (ii) to compare two schedulers for fairness, we make comparisons of how the schedulers favor/discriminate individual jobs. We use performance and discrimination trends to validate our approach. We observe that our approach can deduce discrimination more accurately. This is true even in cases where the most discriminated jobs are not the worst performing jobs.
Uploads
Papers by John Ngubiri