Compression and query execution within column oriented databases

2014

Domain encoding is a common technique to compress the columns of a column store and to accelerate many types of queries at the same time. It is based on the assumption that most columns contain a relatively small set of distinct values, in particular string columns. In this paper, we argue that domain encoding is not the end of the story. In real world systems, we observe that a substantial amount of the columns are of string types. Moreover, most of the memory space is consumed by only a small fraction of these columns. To address this issue, we make three main contributions: First we survey several approaches and variants for dictionary compression, i. e., data structures that store the dictionary of domain encoding in a compressed way. As expected, there is a trade-off between size of the data structure and its access performance. This observation can be used to compress rarely accessed data more than frequently accessed data. Furthermore the question which approach has the best ...

1998

Journal of Computers, 2009

Abstract-Loss-less data compression is potentially attractive in database application for storage cost reduction and performance improvement. The existing compression architectures work well for small to large database and provide good performance. But these systems can ...

The Computer Journal, 1998

Future database applications will require significant improvements in performance beyond the capabilities of conventional disk based systems. This paper describes a new approach to database systems architecture, which is intended to take advantage of solid-state memory in combination with data compression to provide substantial performance improvements. The compressed data representation is tailored to the data manipulation operations requirements. The architecture has been implemented and measurements of performance are compared to those obtained using other high-performance database systems. The results indicate from one to five orders of magnitude speed-up in retrieval, equivalent or slightly faster performance during insertion (and compression) of data, while achieving approximately one order of magnitude compression in data volume. The resultant architecture is thus capable of greater cost/effectiveness than conventional approaches.

Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, 2009

Column-oriented database systems [19, 23] perform better than traditional row-oriented database systems on analytical workloads such as those found in decision support and business intelligence applications. Moreover, recent work [1, 24] has shown that lightweight compression schemes significantly improve the query processing performance of these systems. One such a lightweight compression scheme is to use a dictionary in order to replace long (variable-length) values of a certain domain with shorter (fixedlength) integer codes. In order to further improve expensive query operations such as sorting and searching, column-stores often use order-preserving compression schemes. In contrast to the existing work, in this paper we argue that orderpreserving dictionary compression does not only pay off for attributes with a small fixed domain size but also for long string attributes with a large domain size which might change over time. Consequently, we introduce new data structures that efficiently support an order-preserving dictionary compression for (variablelength) string attributes with a large domain size that is likely to change over time. The main idea is that we model a dictionary as a table that specifies a mapping from string-values to arbitrary integer codes (and vice versa) and we introduce a novel indexing approach that provides efficient access paths to such a dictionary while compressing the index data. Our experiments show that our data structures are as fast as (or in some cases even faster than) other stateof-the-art data structures for dictionaries while being less memory intensive.

Security and Privacy, 2019

A digital watermark is a secret pattern of bits inserted in the data objects to verify the integrity of the contents or to identify ownership. It is an effective technique for protecting various databases against data copyright infringement, piracy, and data tampering in both relational and not-only-Structured Query Language (NoSQL) databases including columnar database. Unfortunately, most of the existing digital watermarking schemes do not take into consideration the side effects that watermarking might have on the database's important characteristics such as data compression and overall performance. Thus, we are in need of database watermarking schemes that are tailored to the target database systems' specific architecture so that it does not interfere with the database's characteristics and performance. Columnar database is a type of NoSQL database where its data are stored in a column-wise manner. This facilitates better query processing (by leaving the unrelated attributes on the disk) as well as higher compression ratio. In this research, we propose a distortion-free fragile watermarking scheme for the columnar database architecture without interfering its underlying data compression scheme and its overall performance. The proposed scheme is flexible and can be adapted to various distributions of data. We tested our proposed scheme on both synthetic and real-world data, and proved its effectiveness. K E Y W O R D S columnar databases, data integrity, database security, digital watermarking, information hiding 1 INTRODUCTION Databases have served the business intelligence industry well over the last few decades. However, in this current era of the "Big Data," where the increasingly large volumes of data are added to the business organizations' databases at a high speed, fast analysis of data is becoming more difficult. Columnar database (DB) is a type of not-only-Structured Query Language (NoSQL) database where data is stored in a column-wise manner, in which all values of an attribute are clustered together. In contrast to row-wise relational databases, columnar DB boosts performance of data analytic transactions by reducing the amount of data that needs to be read from the disk, thus decreasing the bottleneck between Random Access Memory (RAM) and hard drive. For instance, in the traditional relational database, answering a range query would require loading the set of records that falls in the range query and then executing projection query to discard the unwanted attributes. In columnar DB, in contrast, answering range queries would require loading only the columns that contain the attributes that are associated with query. Therefore, the query performance is often increased, especially when the database is very large. Another crucial advantage of storing the data in columns is that it lends itself naturally to compression. Columnar DB achieves a much higher compression ratio and allows many queries to be answered on the compressed data.

Proceedings 2003 VLDB Conference, 2003

The Oracle RDBMS recently introduced an innovative compression technique for reducing the size of relational tables.

Soft Computing, 2018

Structured data are one of the most important segments in the realm of big data analysis that have undeniably prevailed over the years. In recent years, column-oriented design has become a frequent practice to organize structured data in analytical systems. The storage systems that organize data in a column-wise manner are often referred to as column stores. Column-oriented databases or warehouses and spreadsheet applications in particular have recently become a popular and a convenient tool for column-wise data processing and analysis. At the same time, the volume of data is increasing at an extreme rate, which despite the decrease in pricing of storage systems stresses the importance of data compression. Apart from resounding performance gain in large read-mostly data repositories, column-oriented data are easily compressible, which enables efficient query processing and pushes the peak of the overall performance. Many compression algorithms, including the Run Length Encoding (RLE), exploit the similarity among the column values, where repetitions of the same value form columnar runs that can be found in most database systems. This paper presents a comprehensive analysis and comparison of common and well-known meta-heuristics for columnar run minimization, based on standard implementations by using real datasets. We have analyzed genetic algorithms, simulated annealing, cuckoo search, particle swarm optimization, Tabu search, and the bat algorithm. The first three being the most efficient have undergone sensitivity analysis on synthetic datasets to fine-Communicated by V. Loia.

International Journal of Data Mining & Knowledge Management Process, 2013

Column-Stores database stores data column-by-column. The need for Column-Stores database arose for the efficient query processing in read-intensive relational databases. Also, for read-intensive relational databases,extensive research has performed for efficient data storage and query processing. This paper gives an overview of storage and performance optimization techniques used in Column-Stores.

IEEE Transactions on Knowledge and Data Engineering, 1997

Disk I/O has long been a performance bottleneck for very large databases. Database compression can be used to reduce disk I/O bandwidth requirements for large data transfers. In this paper, we explore the compression of large statistical databases and propose techniques for organizing the compressed data such that standard database operations such as retrievals, inserts, deletes and modifications are supported. We examine the applicability and performance of three methods. Two of these are adaptations of existing methods, but the third, called Tuple Differential Coding (TDC) [16], is a new method that allows conventional access mechanisms to be used with the compressed data to provide efficient access. We demonstrate how the performance of queries that involve large data transfers can be improved with these database compression techniques.