PVFS: A Parallel File System for Linux Clusters

2006 IEEE International Performance Computing and Communications Conference, 2006

This paper presents the design and implementation of a new I/O networking infrastructure, named PaScal 1 (Parallel and Scalable I/O Networking Framework). PaScal is used to support high data bandwidth IP based global storage systems for large scale Linux clusters. PaScal has several unique properties. It employs (1) Multi-level switch-fabric interconnection network by combining high speed interconnects for computing Inter-Process Communication (IPC) requirements and low-cost Gigabit Ethernet interconnect for global IP based storage/file access, (2) A bandwidth on demand scaling I/O networking architecture, (3) Open-standard IP networks (routing and switching), (4) Multipath routing for load balancing and failover, (5) Open Shortest Path First (OSPF) routing software, and (6) Supporting a global file system in multi-cluster and multiplatform environments. We describe both the hardware and software components of our proposed PaScal.

Cluster Computing and the …, 2002

In this paper, we report on the experiences in designing a portable parallel file system for clusters. The file system offers to the applications an interface compliant with MPI-IO, the I/O interface of the MPI-2 standard. The file system implementation relies upon MPI for internal coordination and communication. This guarantees high performance and portability over a wide range of hardware and software cluster platforms. The internal architecture of the file system has been designed to allow rapid prototyping and experimentation of novel strategies for managing parallel I/O in a cluster environment. The discussion of the file system design and early implementation is completed with basic performance measures confirming the potential of the approach.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2010

pCFS is a highly available parallel, symmetrical (where nodes perform both compute and I/O work) cluster file system that we have designed to run in medium-sized clusters. In this paper, using exactly the same hardware and Linux version across all nodes we compare pCFS with two distinct configurations of PVFS: one using internal disks, and therefore not able to provide any tolerance against disk and/or I/O node failures, and another where PVFS I/O servers access LUNs in a disk array and thus provide high availability (in the following named HA-PVFS). We start by measuring I/O bandwidth and CPU consumption of PVFS and HA-PVFS setups; then, the same set of tests is performed with pCFS. We conclude that, when using the same hardware, pCFS compares very favourably with HA-PVFS, offering the same or higher I/O bandwidths at a much lower CPU consumption.

2009

Today, clusters are the de facto cost effective platform both for high performance computing (HPC) as well as IT environments. HPC and IT are quite different environments and differences include, among others, their choices on file systems and storage: HPC favours parallel file systems geared towards maximum I/O bandwidth, but which are not fully POSIX-compliant and were devised to run on top of (fault prone) partitioned storage; conversely, IT data centres favour both external disk arrays (to provide highly available storage) and POSIX compliant file systems, (either general purpose or shared-disk cluster file systems, CFSs). These specialised file systems do perform very well in their target environments provided that applications do not require some lateral features, e.g., no file locking on parallel file systems, and no high performance writes over cluster-wide shared files on CFSs. In brief, we can say that none of the above approaches solves the problem of providing high levels of reliability and performance to both worlds. Our pCFS proposal makes a contribution to change this situation: the rationale is to take advantage on the best of both-the reliability of cluster file systems and the high performance of parallel file systems. We don't claim to provide the absolute best of each, but we aim at full POSIX compliance, a rich feature set, and levels of reliability and performance good enough for broad usage-e.g., traditional as well as HPC applications, support of clustered DBMS engines that may run over regular files, and video streaming. pCFS' main ideas include:

Proceedings of the 2001 ACM/IEEE conference on Supercomputing (CDROM) - Supercomputing '01, 2001

A critical but often ignored component of system performance is the I/O system. Today's applications demand a great deal from underlying storage systems and software, and both high-performance distributed storage and high level interfaces have been developed to fill these needs.

MOSIX is a cluster computing enhancement of Linux that supports preemptive process migration. This paper presents the MOSIX Direct File System Access (DFSA), a provision that can improve the performance of cluster le systems by migrating the process to the le, rather then the traditional way of bringing the le's data to the process. DFSA is suitable for clusters that manage a pool of shared disks among multiple machines. With DFSA, it is possible to migrate parallel processes from a client node to le servers enabling parallel access to di erent les. DFSA can work with any le system that maintains cache consistency. Since no such le system is currently available for Linux, we implemented the MOSIX File-System (MFS) as a rst prototype using DFSA. The paper describes DFSA and presents the performance of MFS with and without DFSA.

Lecture Notes in Computer Science, 2006

The technology advances made in supercomputers and high performance computing clusters over the past few years have been tremendous. Clusters are the most common solution for high performance computing at the present time. In this kind of systems, an important subject is the parallel I/O subsystem design. Parallel file systems (GPFS, PVFS, Lustre, etc) have been the solution used to obtain high performance I/O. Parallel file systems increase performance by distributing data file across several I/O nodes. However, cluster's size is increasing continuously, specially for compute nodes, becoming the I/O nodes in a possible bottleneck of the system. In this paper, we propose a new architecture that solves the problem pointed out before: new hierarchical I/O architecture based on parallel I/O proxies. Those I/O proxies execute on the compute nodes offering an intermediate parallel file system between the applications and the storage system of the cluster. That architecture reduces the load on the I/O nodes increasing the global performance. This paper shows the design of the proposed solution and a preliminary evaluation, using a cluster located in the Stuttgart HLRS center.

2004

This paper presents the integration of two collective I/O techniques into the Clusterfile parallel file system : diskdirected I/O and two-phase I/O. We show that global cooperative cache management improves the collective I/O performance. The solution focuses on integrating disk parallelism with other types of parallelism: memory (by buffering and caching on several nodes), network (by parallel I/O scheduling strategies) and processors (by redistributing the I/O related computation over several nodes). The performance results show considerable throughput increases over ROMIO's extended two-phase I/O.

2018 IEEE International Conference on Cluster Computing (CLUSTER), 2018

We present GekkoFS, a temporary, highly-scalable burst buffer file system which has been specifically optimized for new access patterns of data-intensive High-Performance Computing (HPC) applications. The file system provides relaxed POSIX semantics, only offering features which are actually required by most (not all) applications. It is able to provide scalable I/O performance and reaches millions of metadata operations already for a small number of nodes, significantly outperforming the capabilities of general-purpose parallel file systems.

There are no high performance file systems that allow sharing of data between different clusters presently. In this paper, we address this issue by developing FICUS (File System for Inter Cluster Unified Storage). FICUS provides the convenience of file system level access to remote data while preserving the performance of striped file systems such as PVFS. We achieve a bandwidth of 80 MB/s for local access using four I/O nodes, and 75 MB/s in accessing the same number of I/O nodes on a remote storage cluster. A parallel data intensive application achieves comparable performance when accessing 6 GB of data in local and remote storage. We show that careful pipelining of data transfer and proper integration with underlying file system and communication layers are crucial for preserving the performance of remote access.