Efficient load rebalancing for distributed file system in Clouds

Cloud computing is an upcoming era in software industry. It's a very vast and developing technology. Distributed file systems play an important role in cloud computing applications based on map reduce techniques. While making use of distributed file systems for cloud computing, nodes serves computing and storage functions at the same time. Given file is divided into small parts to use map reduce algorithms in parallel. But the problem lies here since in cloud computing nodes may be added, deleted or modified any time and also operations on files may be done dynamically. This causes the unequal load distribution of load among the nodes which leads to load imbalance problem in distributed file system. Newly developed distributed file system mostly depends upon central node for load distribution but this method is not helpful in large-scale and where chances of failure are more. Use of central node for load distribution creates a problem of single point dependency and chances of performance of bottleneck are more. As well as issues like movement cost and network traffic caused due to migration of nodes and file chunks need to be resolved. So we are proposing algorithm which will overcome all these problems and helps to achieve uniform load distribution efficiently.

IOSR Journal of Computer Engineering, 2014

Cloud Computing is an emerging technology, it is based on demand service in which shared resources, information, software and other devices are provided according to the clients to the requirements at specific time with the availability of internet. Load balancing is one of the challenging issue in cloud computing. An efficient load balancing makes cloud computing more efficient and improves user satisfaction. It includes fault tolerance, high availability, scalability, flexibility, reduced overhead for users, reduced cost of ownership, on demand services etc. Distributed file systems are key building blocks for cloud computing applications based on the Map Reduce programming paradigm. In such file systems, nodes at the same time serve computing and storage functions. Files can be created, deleted, and appended dynamically. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed uniformly as possible among the nodes.

For cloud computing applications the Distributed file system is used as a key building block which is simply a classical model. In such file system a file is partitioned into a number of chunks allocated in distinct nodes .Each chunk allocates to separate node to perform MapReduce function parallel over each node. In cloud, the central node (master in MapReduce) becomes bottleneck if number of storage nodes, number of files and assesses to that file increases. In this survey paper to overcome the above load imbalance problem the fully distributed load rebalancing algorithm is used to exclude the load on central node and also the movement cost is reduced. In this paper the load misbalancing problem is overcome.

Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment.

For cloud computing applications the Distributed file system is used as a key building block which is simply a classical model. In such file system a file is partitioned into a number of chunks allocated in distinct nodes .Each chunk allocates to separate node to perform MapReduce function parallel over each node. In cloud, the central node (master in MapReduce) becomes bottleneck if number of storage nodes, number of files and assesses to that file increases. In this survey paper to overcome the above load imbalance problem the fully distributed load rebalancing algorithm is used to exclude the load on central node and also the movement cost is reduced. In this paper the load misbalancing problem is overcome.

2016

Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, the nodes are simultaneously serve this computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that this MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. In this paper, a fully distributed load rebalancing algorithm is presented to solve with the load imbalance problem. This algorithm is compared against a centralized approach in...

Contemporary Engineering Sciences, 2015

File storage load can be balanced in the storage nodes avail in the cloud system by using totally distributed load rebalancing algorithm. Large level distributed systems such as cloud applications come with rising challenges on how to transfer and where to store compute data. Cloud computing is a distributed computing over a network. Node concurrently serves as a computing and storage task. In cloud computing environment, files can also be dynamically created, deleted and append. The file chunks are not distributed as equally among the nodes. Lead to load inequity in a distributed file system. The existing distributed file system depends on single node to manage almost all operations such as chunk reallocation of every data block in the file system. As a result it can be bottleneck resource and a single point of failure. A new technique Random Linear Network Coding (RLNC) is employed in the proposed system. RLNC is performed at the opening when the file is stored in the cloud. Using this strategy, a file will be split into different parts and send distinct parts to each chunk server. RSA algorithm applied to calculate the response time file size and deadlock detection. RSA algorithm used to detect the anomalies of the system. Dynamic scheduling algorithm present in the proposed system to overcome to load inequity problem. This algorithm is compared against a centralized approach in a production system. The results indicate that our proposal is considerably outperforms the prior distributed in terms of load inequity factor, movement cost, and algorithmic overhead.

2013

Distributed file systems are key building blocks for cloud computing applications based on the Map Reduce programming paradigm. Load balance among storage nodes is a critical function in clouds. In a load-balanced cloud, the resources can be well utilized and provisioned, maximizing the performance of Map Reduce-based applications. In such a distributed file system, the load of a node is typically proportional to the number of file chunks the node possesses. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. The proposed algorithm is compared against a centralized approach in a production system and strives to balance the loads of nodes and reduce the demanded movement cost as much as possible, while taking advantage of physical network locality and node heterogeneity.

2014

Map Reduce programming paradigm plays a vital role in the development of cloud computing application using the Distributed file system where nodes concurrently provide computing as well as storage functions. Initially a file is partitioned into number of chunks allocated into different nodes so that Map Reduce technique can be performed in the nodes. Since cloud computing is a dynamic environment upgrading, replacing and adding new nodes to the environment is a frequent concern. This confidence is obviously insufficient in a large-scale, failure-prone atmosphere since the central load balancer is put under significant workload that is linearly scaled with the structure of the system range, and may lead to a performance bottleneck the single point of failure. To overcome the failure in this paper, a fully distributed load rebalancing algorithm is presented to handle the load imbalance problem. The proposed algorithm is compared alongside a centralized approach in a production system ...

International Journal of Computer Applications, 2015

Distributed Systems are useful for computation and storage of large scale data at dispersed location. Distributed File System (DFS) is a subsystem of Distributed System. DFS is a means of sharing of storage space and data. Servers, Storage devices and Clients are on dispersed location in DFS. Fault tolerance and Scalability are two main features of distributed file system. Performance of DFS is measured by response time. Apart from response time there are also other dimensions such as transparency in which performance of DFS is viewed. DFS provides file services with scalability, fault tolerance, availability, minimum response time. The truth behind the minimum response time is good design of load balance algorithm. To improve the minimum response time and utilization of all nodes in DFS cluster it is found static as well as dynamic load balance strategies. In this survey paper Self acting, load balancing for parallel file system, Adaptive loading data migration in distributed file system, Load balancing in distributed multi agent computing systems, Self organizing storage clusters for data intensive applications, User centric data migration in networked storage systems are discussed to study the different load balancing schemes. Adaptive loading data migration is one of the latest solution found in literature survey. Self acting, load balancing (SALB) for parallel file system is for load balancing uses online load prediction methods and is distributed architecture.