Approximation algorithms for data placement on parallel disks

2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom), 2013

We consider the problem of distributing media files for streaming on a distributed storage network, where servers have heterogeneous capacities and bandwidths. Regarding networking the servers' bandwidths are the bottlenecks for streaming. We present an algorithm that computes an assignment of n files to m servers for distributing media files such that the streaming speed requirements and capacity constraints are kept. As an additional feature this assignment algorithm works online, i.e. it can assign each file without files to be stored later on. Our algorithm computes the data assignment in time O(nm+m log m) outperforming linear program solvers.

Theoretical Computer Science, 2012

Video-on-Demand (VoD) services require frequent updates in file configuration on the storage subsystem, so as to keep up with the frequent changes in movie popularity. This defines a natural reconfiguration problem in which the goal is to minimize the cost of moving from one file configuration to another. The cost is incurred by file replications performed throughout the transition. The problem shows up also in production planning, preemptive scheduling with setup costs, and dynamic placement of Web applications. We show that the reconfiguration problem is NP-hard already on very restricted instances. We then develop algorithms which achieve the optimal cost by using servers whose load capacities are increased by O(1), in particular, by factor 1 + δ for any small 0 < δ < 1 when the number of servers is fixed, and by factor of 2 + ε for arbitrary number of servers, for some ε ∈ [0, 1). To the best of our knowledge, this particular variant of the data migration problem is studied here for the first time.

2007

In this thesis we address three problems related to self-management of storage networksdata placement, data reconfiguration and data monitoring. Examples of such storage networks include centrally managed systems like Storage Area Networks and Network Attached Storage devices, or even highly distributed systems like a P2P network or a Sensor Network. One of the crucial functions of a storage system is that of deciding the placement of data within the system. This data placement is dependent on the demand pattern for the data and subject to constraints of the storage system. For instance, if a particular data item is very popular the storage system might want to host it on a disk with high bandwidth or make multiple copies of the item. We present new results for some of these data placement problems.

Computer Networks, 2002

The delivery of large files to individual users, such as video on demand or application programs to the envisioned network computers is expected by many to be one of the main tasks of broadband communication networks. This requires high bandwidth capacity as well as fast and dense storage servers. This motivates multimedia service providers to optimize the delivery network, as well as the electronic content allocation.

Journal of Computer and System Sciences, 2006

The effectiveness of a distributed system hinges on the manner in which tasks and data are assigned to the underlying system resources. Moreover, today's large-scale distributed systems must accommodate heterogeneity in both the offered load and in the makeup of the available storage and compute capacity. The ideal resource assignment must balance the utilization of the underlying system against the loss of locality incurred when individual tasks or data objects are fragmented among several servers. In this paper we describe this locality-maximizing placement problem and show that an optimal solution is NP-hard. We then describe a polynomial-time algorithm that generates a placement within an additive constant of two from optimal.

2005

The addition of storage capacity in network nodes for the caching or replication of popular data objects results in reduced end-user delay, reduced network traffic, and improved scalability. The problem of allocating an available storage budget to the nodes of a hierarchical content distribution system is formulated; optimal algorithms, as well as fast/efficient heuristics, are developed for its solution. An innovative aspect of the presented approach is that it combines all relevant subproblems, concerning node locations, node sizes, and object placement, and solves them jointly in a single optimization step. The developed algorithms may be utilized in content distribution networks that employ either replication or caching/replacement.

Very Large Data Bases, 1997

Recently, technological advances have resulted in the wide availability of commercial prod- ucts offering near-line, robot-based, tertiary storage libraries. Thus, such libraries have become a crucial component of modern large- scale storage servers, given the very large stor- age requirements of modern applications. Al- though the subject of optimal data placement (ODP) strategies has received considerable at- tention for other

Parallel and Distributed Processing Techniques and Applications, 2000

Media-On-Demand (MOD) servers cater to users' needs for data and information such as news, movies, interactive games, music, merchandise catalogs, etc. This requires the storage, management, and delivery of huge amounts of multimedia data. One of the models of a MOD server is general network of computers. Some of the nodes in the network are storage nodes which contain data repositories, and the others are interface nodes which obtain data from storage nodes and deliver them to the users (clients) in a timely fashion. Determining the locations of the storage nodes to minimize the data tra c between them and the interface nodes is a global optimization problem. This paper presents an o ine heuristic for choosing the locations of storage nodes and clustering interface nodes with storage nodes so as to minimize the trafc from the storage nodes. The proposed algorithm has been validated by simulations.

Proceedings International Parallel and Distributed Processing Symposium, 2003

As storage systems scale to thousands of disks, data distribution and load balancing become increasingly important. We present an algorithm for allocating data objects to disks as a system as it grows from a few disks to hundreds or thousands. A client using our algorithm can locate a data object in microseconds without consulting a central server or maintaining a full mapping of objects or buckets to disks. Despite requiring little global configuration data, our algorithm is probabilistically optimal in both distributing data evenly and minimizing data movement when new storage is added to the system. Moreover, our algorithm supports weighted allocation and variable levels of object replication, both of which are needed to permit systems to efficiently grow while accommodating new technology.

The Journal of Supercomputing, 2009

In this paper we investigate the composition of cheap network storage resources to meet specific availability and capacity requirements. We show that the problem of finding the optimal composition for availability and price requirements can be reduced to the knapsack problem, and propose three techniques for efficiently finding approximate solutions. The first algorithm uses a dynamic programming approach to find mirrored storage resources for high availability requirements, and runs in the pseudopolynomial O(n 2 c) time where n is the number of sellers' resources to choose from and c is a capacity function of the requested and minimum availability. The second technique is a heuristic which finds resources to be agglomerated into a larger coherent resource, with complexity of O(n log n). The third technique finds a compromise between capacity and availability (which in our phrasing is a complex integer programming problem) using a genetic algorithm. The algorithms can be implemented on a broker that intermediates between buyers and sellers of storage resources. Finally, we show that a broker in an open storage market, using the combination of the three algorithms can more frequently meet user requests and lower the cost of requests that are met compared to a broker that simply matches single resources to requests.

1996

A Video-on-Demand (VOD) server needs to store hundreds of movie titles and to support thousands of concurrent accesses. This, technically and economically, imposes a great challenge on the design of the disk storage subsystem of a VOD server. Due to di erent demands for di erent movie titles, the numbers of concurrent accesses to di erent movie titles can di er a lot. We de ne access pro le as the number of concurrent accesses to each movie title that should be supported by a VOD server. The access pro le is derived based on the popularity of each movie title and thus serves as a major design goal for the disk storage subsystem. Since some popular (hot) movie titles may be concurrently accessed by hundreds of users and a current high-end magnetic disk array (disk) can only support tens of concurrent accesses, it is necessary to replicate and/or stripe the hot movie les over multiple disk arrays. The consequence of replication and striping for hot movie titles is the potential increase on the required number of disk arrays. Therefore, how to replicate, stripe, and place the movie les over a minimum number of magnetic disk arrays such that a given access pro le can be supported is an important problem. In this paper, we formulate the problem of the video le allocation over disk arrays, demonstrate that it is a NP-hard problem, and present some heuristic algorithms to nd the near-optimal solutions. The result of this study can be applied to the design of the storage subsystem of a VOD server to economically minimize the cost or to maximize the utilization of disk arrays.

2010 IEEE International Symposium on Information Theory, 2010

We consider the problem of distributing a file in a network of storage nodes whose storage budget is limited but at least equals the size file. We first generate T encoded symbols (from the file) which are then distributed among the nodes. We investigate the optimal allocation of T encoded packets to the storage nodes such that the probability of reconstructing the file by using any r out of n nodes is maximized. Since the optimal allocation of encoded packets is difficult to find in general, we find another objective function which well approximates the original problem and yet is easier to optimize. We find the optimal symmetric allocation for all coding redundancy constraints using the equivalent approximate problem. We also investigate the optimal allocation in random graphs. Finally, we provide simulations to verify the theoretical results.

2015 IEEE Global Communications Conference (GLOBECOM), 2014

In this paper we consider distributed allocation problems with memory constraint limits. Firstly, we propose a tractable relaxation to the problem of optimal symmetric allocations from [1]. The approximated problem is based on the Q-error function, and its solution approaches the solution of the initial problem, as the number of storage nodes in the network grows. Secondly, exploiting this relaxation, we are able to formulate and to solve the problem for storage allocations for memory-limited DSS storing and arbitrary memory profiles. Finally, we discuss the extension to the case of multiple data objects, stored in the DSS.

Journal of Computer and System Sciences, 1989

The design and analysis of algorithms for on-line dynamic storage allocation has been a fundamental problem area in computer science for many years. In this paper we study the stochastic behavior of dynamic allocation algorithms under the natural assumption that liles enter and leave the system according to a Poisson process. In particular, we prove that for any dynamic allocation algorithm and any distribution of file sizes, the expected wasted space (or fragmentation) in the system at any time is sZ (& Jw), where N is the expected number of items (or used space) in the system. This result is known to be tight in the special case when all files have the same size. More importantly, we also construct a dynamic allocation algorithm which for any distribution of file sizes wastes only O(filog3'4N) space with very high probability. This bound is also shown to be tight for a wide variety of file-size distributions, including for example the uniform and normal distributions. The results are significant because they show that the cumulative wasted space in the holes formed by the continual arrival and departure of items is a vanishingly small portion of the used space, at least on the average. This fact is in striking contrast with Knuth's well-known 50% rule which states that the number of these holes is linear in the used space. Moreover, the proof techniques establish a surprising connection between stochastic processes, such as dynamic allocation, and static problems such as bin-packing and planar matching. We suspect that the techniques will also prove useful in analyzing other stochastic processes which might otherwise prove intractable. Lastly, we present experimental data in support of the theoretical proofs, and as a basis for postulating several conjectures. DYNAMIC STORAGE ALLOCATION 3 setting, items (records, files, etc.) of varying sizes enter and leave a storage device in a sequence not known in advance. The storage device is represented by a set of consecutive locations or addresses. At its time of arrival, an item is allocated storage space consisting of a contiguous sequence of unoccupied locations equal in length to the item's size. Some time later, the item may depart, thereby making its space available to other items yet to arrive. An item cannot be moved prior to departure, and thus wasted space builds up over time in the form of interior holes alternating with regions of occupied space. This phenomenon is commonly known as fragmentation. The basic problem is to design algorithms which minimize, at least on the average, the cumulative wasted space in the interior holes.

2007

We present a randomized block-level storage virtualiza- tion for arbitrary heterogeneous storage systems that can distribute data in a fair and redundant way and can adapt this distribution in an efficient way as storage devices en- ter or leave the system. More precisely, our virtualization strategies can distribute a set of data blocks among a set of storage devices of arbitrary non-uniform capacities so that a storage device representing x% of the capacity in the system will get x% of the data (as long as this is in princi- ple possible) and the different copies of each data block are stored so that no two copies of a data block are located in the same device. Achieving these two properties is not easy, and no virtualization strategy has been presented so far that has been formally shown to satisfy fairness and redundancy while being time- and space-efficient and allowing an effi- cient adaptation to a changing set of devices.