Dynamic buffer management with extensible replacement policies
Zhiyong Huang2005, The VLDB Journal
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
The objective of extensible DBMSs is to ease the construction of specialized DBMSs for nontraditional applications. Although much work has been done in providing various levels of extensibility (e.g., extensibility of data types and operators, query language extensibility, and query optimizer extensibility), there has been very limited research in providing extensibility at the buffer management level. Supporting extensibility at the buffer management level is important as it can contribute significantly to overall system performance. This paper addresses the problem of supporting extensibility of buffer replacement policies. The main contribution is the proposal of a framework for modeling buffer replacement policies. This work is novel in two aspects. First, by providing a uniform and generic specification of buffer replacement policies, the proposed framework unifies existing work in this area. Second, our work introduces a new level of extensibility. None of the existing extensible DBMSs, to our knowledge, provides extensibility at the buffer management level. The proposed framework provides a basis for the construction of an extensible buffer manager as part of a 100% Java-based storage manager. We conducted an extensive performance study to investigate the performance of the proposed framework. The experimental results demonstrate that the proposed framework is indeed feasible for existing DBMSs and improves system performance significantly without costing significant overhead.
Related papers
A framework for modeling buffer replacement strategies
Proceedings of the ninth international conference on Information and knowledge management - CIKM '00, 2000
An e ective bu er management system is crucial for any database management system. While much work has be e n expended to provide extensible data t ypes,extensible query languages and even extensible optimizers, there is very limited research in providing extensibility at the bu er management level. Supporting extensibility at the bu er management level is equally, if not more, important as no single strategy can perform well in all applications e ciently. In this paper, we present a uniform framework for modeling bu er replacement policies. The framework allows the bu er manager to beeasily extended to provide support for and ne-tuning of di erent replacement policies. Our work is novel in t wo aspects. First, the proposed framework uni es existing work in this area. Second, our work introduces a new level of extensibility. To our knowledge, none of the existing extensible DBMSs and storage managers provide extensibility at the bu er management level. We implemented an extensible bu er manager and experimented with di erent bu er replacement polices. The experimental study illustrates the ease of use and e ciency of the propo s e dframework.
Goal-oriented dynamic buffer pool management for data base systems
… Systems, 1995. Held …, 1995
This paper presents a technique for performing dynamic goal oriented bu er pool management for database management systems. To dynamically adjust the bu er pool sizes for the multiple bu er pools provided by database management systems is a complex constrained optimization problem. In the goal oriented approach, the user speci es each bu er pool's random access response time goal and the total available number of bu ers for all bu er pools. The problem is to dynamically expand or contract the bu er pool sizes based on the database workload to achieve these pre-de ned response time goals for each bu er pool while maintaining the same total number of bu ers in the database system. Our goal satisfaction algorithm monitors goal satisfaction of each bu er pool and periodically changes bu er pool sizes to improve goal satisfaction. The expanding and contracting process does not allocate new or free up existing virtual storage. We demonstrate that dynamic tuning can greatly improve bu er pool goal satisfaction through trace driven simulations.
A programmable buffer management platform
2017 IEEE 25th International Conference on Network Protocols (ICNP), 2017
Buffering architectures and policies for their efficient management constitute one of the core ingredients of a network architecture. However, despite strong incentives to experiment with, and deploy, new policies, the opportunities for alterating anything beyond minor elements of such policies are limited. In this work we introduce a new specification language, OpenQueue, that allows users to specify entire buffering architectures and policies conveniently through several comparators and simple functions. We show examples of buffer management policies in OpenQueue and empirically demonstrate its direct impact on performance in various settings.
A cost-based buffer replacement algorithm for object-oriented database systems
Information Sciences, 2001
Many object-oriented database systems manage object buers to provide fast access to objects. Traditional buer replacement algorithms based on ®xed-length pages simply assume that the cost incurred by operating a buer is proportional to the number of buer faults. However, this assumption no longer holds in an object buer where objects are of variable-lengths and the cost of replacing an object varies for each object.
The oracle universal server buffer manager
PROCEEDINGS OF THE …, 1997
The buffer manager is integral to the performance, scalability, and reliability of Oracle's Universal Dam Server, a high performance object-relational database manager that provides robust data-management services for a variety of applications and tools. The rich ...
Flexible and scalable storage management for data-intensive stream processing
Proceedings of the 12th International Conference on Extending Database Technology Advances in Database Technology - EDBT '09, 2009
Data Stream Management Systems (DSMS) operate under strict performance requirements. Key to meeting such requirements is to efficiently handle time-critical tasks such as managing internal states of continuous query operators, traffic on the queues between operators, as well as providing storage support for shared computation and archived data. In this paper, we introduce a general purpose storage management framework for DSMSs that performs these tasks based on a clean, loosely-coupled, and flexible system design that also facilitates performance optimization. An important contribution of the framework is that, in analogy to buffer management techniques in relational database systems, it uses information about the access patterns of streaming applications to tune and customize the performance of the storage manager. In the paper, we first analyze typical application requirements at different granularities in order to identify important tunable parameters and their corresponding values. Based on these parameters, we define a general-purpose storage management interface. Using the interface, a developer can use our SMS (Storage Manager for Streams) to generate a customized storage manager for streaming applications. We explore the performance and potential of SMS through a set of experiments using the Linear Road benchmark.
An Alternative approach to Temporary Memory Management in Databases using Object Oriented Systems
2010
Regardless of the supremacy of relational database management systems (RDBMS) in the databases, object-oriented database management systems (OODBMS) continue to play crucial role in the management of data. Generally, the complex data are often found in telecommunications, business, engineering and web based applications. The most common approach of accessing such data is navigation. However, the approach of navigational access of data has the potential of generating excessive disk IO because objects in the path of navigation may be placed in different disk pages. Excessive disk IO is becoming increasingly undesirable because disk IO performance improves at only 5-8% per year whereas CPU performance doubles approximately every 18 months. Thus disk IO is likely to be a bottleneck in an increasing number of OODB applications. This paper focuses on reducing disk IO effects to improve OODBMS performance. In database environment effective buffer management of the main memory is the key in increasing efficiency through reducing the disk IO bottleneck in OODBMSs. There has been much existing work, namely in the areas of: static clustering; dynamic clustering; buffer replacement; and pre-fetching. All of these techniques can be used together in a complimentary manner. Most existing research has focused on finding the best solution for each area with little regard on how solutions from the different areas affect each other. We believe synergy exists between the areas, and that exploiting the synergy leads to the best overall solution. This paper focuses on exploring whether synergistic techniques are both feasible to implement and outperform their non-synergistic counterparts.
The LRU-K page replacement algorithm for database disk buffering
ACM SIGMOD Record, 1993
This paper introduces a new approach to database disk buffering, called the LRU-K method. The basic idea of LRU-K is to keep track of the times of the last K references to popular database pages, using this information to statistically estimate the interarrival time of such references on a page by page basis. Although the LRU-K approach performs optimal statistical inference under relatively standard assumptions, it is fairly simple and incurs little bookkeeping overhead. As we demonstrate with simulation experiments, the LRU-K algorithm surpasses conventional buffering algorithms in discriminating between frequently and infrequently referenced pages. In fact, LRU-K can approach the behavior of buffering algorithms in which page sets with known access frequencies are manually assigned to different buffer pools of specifically tuned sizes. Unlike such customized buffering algorithms however, the LRU-K method is self-tuning, in the sense that it does not rely on external hints about workload characteristics. Furthermore, the LRU-K algorithm adapts in real time to changing patterns of access.
An Extensible Storage Manager for Mobile DBMS
2002
The increasing usage of mobile devices like PDAs, laptops, or embedded devices results in a new type of application which must especially consider the strict limitations of the used mobile hardware. One aspect of the application development is the storage and retrieval of data. For non-mobile application this is often efficiently realized with database management systems, which offer standardized interfaces and can be easily integrated into the applications. For mobile devices DBMS are also already available. But existing solutions are not extensible, and therefore, limited to the builtin functionality. That means also, that they include functions which are not always necessary. The optimal DBMS for mobile database systems must allow for the special requirements of its applications in order to reduce the hardware requirements. Thus, it must offer core funtionality which can be extended by additional required features. In this paper, we present a core component of such a customizable DBMS -the storage manager -and describe the architecture as well as the main modules. Furthermore, we show how this modules can be combined in order to address different requirements.
The end of an architectural era:(it's time for a complete rewrite)
2007
In previous papers [SC05, SBC+07], some of us predicted the end of "one size fits all" as a commercial relational DBMS paradigm. These papers presented reasons and experimental evidence that showed that the major RDBMS vendors can be outperformed by 1-2 orders of magnitude by specialized engines in the data warehouse, stream processing, text, and scientific database markets. Assuming that specialized engines dominate these markets over time, the current relational DBMS code lines will be left with the business data processing (OLTP) market and hybrid markets where more than one kind of capability is required. In this paper we show that current RDBMSs can be beaten by nearly two orders of magnitude in the OLTP market as well. The experimental evidence comes from comparing a new OLTP prototype, H-Store, which we have built at M.I.T., to a popular RDBMS on the standard transactional benchmark, TPC-C. We conclude that the current RDBMS code lines, while attempting to be a "one size fits all" solution, in fact, excel at nothing. Hence, they are 25 year old legacy code lines that should be retired in favor of a collection of "from scratch" specialized engines. The DBMS vendors (and the research community) should start with a clean sheet of paper and design systems for tomorrow's requirements, not continue to push code lines and architectures designed for yesterday's needs.
Related papers
LFU-K: An Effective Buffer Management Replacement Algorithm
Database Systems for Advances Applications, 9th International Conference, DASFAA 2004, 2004
This paper introduces a new approach to database disk buffering, called the LFU-K method. The LFU-K page replacement algorithm is an improvement to the Least Frequently Used (LFU) algorithm. The paper proposes a theoretical-probability model for formal description of LFU-K algorithm. Using this model we evaluate estimations for the LFU-K parameters. This paper also describes an implementation of LFU-2 policy. As we demonstrate by trace-driven simulation experiments, the LFU-2 algorithm provides significant improvement over conventional buffering algorithms for the shared-nothing database systems.
A new approach to dynamic self-tuning of database buffers
ACM Transactions on Storage, 2008
Current businesses rely heavily on efficient access to their databases. Manual tuning of these database systems by performance experts is increasingly infeasible: For small companies, hiring an expert may be too expensive; for large enterprises, even an expert may not fully understand the interaction between a large system and its multiple changing workloads. This trend has led major vendors to offer tools that automatically and dynamically tune a database system. Many database tuning knobs concern the buffer pool for caching data and disk pages. Specifically, these knobs control the buffer allocation and thus the cache miss probability, which has direct impact on performance. Previous methods for automatic buffer tuning are based on simulation, black-box control, gradient descent, and empirical equations. This article presents a new approach, using calculations with an analytically-derived equation that relates miss probability to buffer allocation; this equation fits four buffer r...
P2: An Extensible Lightweight DBMS1
1995
A lightweight database system (LWDB) is a high-performance, application-specific DBMS. It differs from a general-purpose (heavyweight) DBMS in that it omits one or more features and specializes the implementation of its features to maximize performance. Although heavyweight monolithic and extensible DBMSs might be able to emulate LWDB capabilities, they cannot match LWDB performance. In this paper, we explore LWDB applications,
The EXODUS Extensible DBMS Project: An Overview
1990
This paper presents an overview of EXODUS, an extensible database system project that is addressing data management problems posed by a variety of challenging new applications. The goal of the project is to facilitate the fast development of high-performance, application-specific database systems. EXODUS provides certain kernel facilities, including a versatile storage manager. In addition, it provides an architectural framework for building application-specific database systems; powerful tools to help automate the generation of such systems, including a rule-based query optimizer generator and a persistent programming language; and libraries of generic software components (e.g., access methods) that are likely to be useful for many application domains. We briefly describe each of the components of EXODUS in this paper, and we also describe a next-generation DBMS that we are now building using the EXODUS tools.
Reconciling Buffer Management with Persistence Optimisations
1998
The so-called 'read' and 'write' barriers present an obstacle to the efficient execution of persistent programs that use a volatile object buffer, nonvolatile object store memory model. The barriers, implemented as checks added to the code, are required to ensure that objects are moved from store to buffer before being used and, if updated in the buffer, that they are written back on periodic checkpoints.
Ql/mrp: A buffer management mechanism for qos support in a multimedia dbms
1998
Multimedia database systems (MMDBSs) have to be capable to handle efficiently time-dependent and timeindependent data, and to support Quality-of-Service (QoS). To support continuous playout of time-dependent data, reservations of the limited resources disk I/O bandwidth and network bandwidth have to be combined with appropriate buffer management. Based on the special requirements of the DEDICATION pre-study, i.e., building a cheap prototype system for asynchronous distance education, we have designed the buffer management mechanism Q-L/MRP. Q-L/MRP is a buffer preloading and page replacement mechanism for multimedia applications with heterogeneous QoS requirements. Q-L/MRP extends L/MRP with two features: (1) it supports multiple concurrent users, and (2) it supports QoS with a dynamic prefetching daemon. This dynamic prefetching daemon is able to dynamically adapt to the changes in network and disk I/O load. Furthermore, QoS requirements from the users like frame rate are mapped into the buffer mechanism. Our performance analysis shows that Q-L/MRP is very suitable for the special environment in DEDICATION and outperforms other buffer management mechanisms. Since we have implemented Q-L/MRP in software only, it is also suitable for other multimedia applications on other systems with different hardware configurations and workloads.
Panel: The effect of large main memory on database systems
ACM SIGMOD Record, 1986
The Architecture of the EXODUS Extensible DBMS
1986
With non-traditional application areas such as engineering design, image/voice data management, scientific/statistical applications, and artificial intelligence systems all clamoring for ways to store and efficiently process larger and larger volumes of data, it is clear that traditional database technology has been pushed to its limits. It also seems clear that no single database system will be capable of simultaneously meeting the functionality and performance requirements of such a diverse set of applications. In this paper we describe the initial design of EXODUS, an extensible database system that will facilitate the fast development of high-performance, applicationspecific database systems. EXODUS provides certain kernel facilities, including a versatile storage manager and a type manager. In addition, it provides an architectural framework for building application-specific database systems, tools to partially automate the generation of such systems, and libraries of software components (e.g., access methods) that are likely to be useful for many application domains. Recently, a number of new database system research projects have been initiated to address the needs of this emerging class of applications: EXODUS 1 at the University of Wisconsin [Care85a, Care86], PROBE at CCA [Daya85, Mano86], POSTGRES [Ston86b, Ston86c] at Berkeley, GEMSTONE at Servio Logic Corporation [Cope84, Maie86], STARBURST at IBM Almaden Research Center [Schw86], and GENESIS [Bato86] at the
L/MRP: A Buffer Management Strategy for Interactive Continuous Data Flows in a Multimedia DBMS
Very Large Data Bases, 1995
Multimedia applications demand specific support from database management systems due to the characteristics of multimedia data and their interactive usage. This includes integrated support for high-volume and time-dependent (continuous) data types like audio and video. One critical issue is to provide handling of continuous data streams including buffer management as needed for multimedia presentations. Buffer management strategies for continuous data
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.