Academia.eduAcademia.edu

Fast Dictionary-Based Compression for Inverted Indexes

Profile image of Giulio Ermanno PibiriGiulio Ermanno Pibiri

Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining

Abstract

Dictionary-based compression schemes provide fast decoding operation, typically at the expense of reduced compression effectiveness compared to statistical or probability-based approaches. In this work, we apply dictionary-based techniques to the compression of inverted lists, showing that the high degree of regularity that these integer sequences exhibit is a good match for certain types of dictionary methods, and that an important new trade-off balance between compression effectiveness and compression efficiency can be achieved. Our observations are supported by experiments using the document-level inverted index data for two large text collections, and a wide range of other index compression implementations as reference points. Those experiments demonstrate that the gap between efficiency and effectiveness can be substantially narrowed.

Figures (14)
Table 1: Two examples of plurally parsable dictionaries of width € = 4 over the alphabet {a,b,c,d} where symbol “a” is highly probable, symbol “b” is moderately probable, and “-” entries indicate don’t-care values. The last column provides the length of each string and is also stored as part of the dictionary. In (b), index zero is used as the code for rare symbol exceptions.
Table 1: Two examples of plurally parsable dictionaries of width € = 4 over the alphabet {a,b,c,d} where symbol “a” is highly probable, symbol “b” is moderately probable, and “-” entries indicate don’t-care values. The last column provides the length of each string and is also stored as part of the dictionary. In (b), index zero is used as the code for rare symbol exceptions.
Table 3: Performance counts reported by the Linux perf tool, com- paring variable-length copying and constant-length copying for € = 16 when decoding the index sequences of Gov2 using a rectan- gular dictionary. Table 2: Total index size in GiB for a complete document-level index (docids and freqs combined, including block-access overhead and dictionary space) for the Gov2 collection using a block size of B = 256 items and the DSV dictionary construction approach.
Table 3: Performance counts reported by the Linux perf tool, com- paring variable-length copying and constant-length copying for € = 16 when decoding the index sequences of Gov2 using a rectan- gular dictionary. Table 2: Total index size in GiB for a complete document-level index (docids and freqs combined, including block-access overhead and dictionary space) for the Gov2 collection using a block size of B = 256 items and the DSV dictionary construction approach.
Figure 3: Packed layout for the dictionary shown in Table 1(b), with € = 4 and b = 3. The first number in each element in start[] is the sequence length. All trailing don’t-care symbols have been trimmed, and dictionary sequences have been removed if they are a prefix o another longer sequence. As an example, when the input codeword is 3 the four integers in dictionary[4 ...7] are copied to the output, anc then the output pointer is incremented by two. In this illustration no provision has been made for frequent symbol exceptions.
Figure 3: Packed layout for the dictionary shown in Table 1(b), with € = 4 and b = 3. The first number in each element in start[] is the sequence length. All trailing don’t-care symbols have been trimmed, and dictionary sequences have been removed if they are a prefix o another longer sequence. As an example, when the input codeword is 3 the four integers in dictionary[4 ...7] are copied to the output, anc then the output pointer is incremented by two. In this illustration no provision has been made for frequent symbol exceptions.
Table 4: Average number of decoded integers per codeword, m, when decoding the sequences of Gov2, using DINT-DSF with a rectangular dictionary and b = 16; and predicted and actual de- coding times measured in nanoseconds per integer, based on c, the measured decoding time per codeword.
Table 4: Average number of decoded integers per codeword, m, when decoding the sequences of Gov2, using DINT-DSF with a rectangular dictionary and b = 16; and predicted and actual de- coding times measured in nanoseconds per integer, based on c, the measured decoding time per codeword.
Figure 4: Example in which optimal parsing requires fewer codewords than greedy parsing. The sequence “aadbaaaa” is being represented relative to the dictionary shown in Table 1(b). The cost below each node is the length of the shortest path from the origin to that point. The parse shown at the bottom of the graph has a cost of 5; whereas the greedy parse shown above requires 6 codewords. In both cases it is assumed that “d” requires a rare symbol exception (rse) codeword, followed by a patch codeword that identifies the symbol required.
Figure 4: Example in which optimal parsing requires fewer codewords than greedy parsing. The sequence “aadbaaaa” is being represented relative to the dictionary shown in Table 1(b). The cost below each node is the length of the shortest path from the origin to that point. The parse shown at the bottom of the graph has a cost of 5; whereas the greedy parse shown above requires 6 codewords. In both cases it is assumed that “d” requires a rare symbol exception (rse) codeword, followed by a patch codeword that identifies the symbol required.
Table 5: Number of lists, postings and documents for the Gov2 and CCNEWS collections.
Table 5: Number of lists, postings and documents for the Gov2 and CCNEWS collections.
Table 6: Total index size (GiB) and compression rate (bits per in- teger) for docids and freqs for two test collections, using a range of compression techniques. The two DINT implementations both make use of b = 16 and ¢ = 16, single dictionaries, and greedy parsing. The rows are ordered by decreasing total index size.
Table 6: Total index size (GiB) and compression rate (bits per in- teger) for docids and freqs for two test collections, using a range of compression techniques. The two DINT implementations both make use of b = 16 and ¢ = 16, single dictionaries, and greedy parsing. The rows are ordered by decreasing total index size.
Table 7: Sequential decoding throughput in nanoseconds per in- teger, measured over the complete index. Both DINT rows make use of b = 16 and a packed dictionary. The rows are ordered by increasing speed of docids decoding on Gov?2.
Table 7: Sequential decoding throughput in nanoseconds per in- teger, measured over the complete index. Both DINT rows make use of b = 16 and a packed dictionary. The rows are ordered by increasing speed of docids decoding on Gov?2.
Figure 5: Percentage of integers, codewords, and dictionary entries associated with each target size for the docids and freqs of Gov2, using the DSF approach and parameters b = 16, € = 16.
Figure 5: Percentage of integers, codewords, and dictionary entries associated with each target size for the docids and freqs of Gov2, using the DSF approach and parameters b = 16, € = 16.
Table 8: Total index size (GiB) and compression rate (bits per inte ger) for docids and freqs using b = 16 and ¢ = 16. The first row uses DINT-DSF with greedy parsing; four enhancements are then added and in the penultimate row a total of 12 contexts are used with optimal parsing and an exhaustive search to identify the cheapest context for each block. In the last row, the dictionary indices are then assumed to be input to a set of 12 optimal entropy coders Except for the last row (gray numbers), which contains values that are calculated rather than measured, these results can be directly compared with Table 6.
Table 8: Total index size (GiB) and compression rate (bits per inte ger) for docids and freqs using b = 16 and ¢ = 16. The first row uses DINT-DSF with greedy parsing; four enhancements are then added and in the penultimate row a total of 12 contexts are used with optimal parsing and an exhaustive search to identify the cheapest context for each block. In the last row, the dictionary indices are then assumed to be input to a set of 12 optimal entropy coders Except for the last row (gray numbers), which contains values that are calculated rather than measured, these results can be directly compared with Table 6.
Table 9: Dictionary space (MiB) for different schemes and corre- sponding decoding speeds, for Gov2. In the three “x1” rows, one dictionary is used for the docids and another for the freqs. In the “x6” rows, six dictionaries are used for each stream, with (in the last row) all six of them combined into a single dictionary[] array, and six start[] arrays maintained.
Table 9: Dictionary space (MiB) for different schemes and corre- sponding decoding speeds, for Gov2. In the three “x1” rows, one dictionary is used for the docids and another for the freqs. In the “x6” rows, six dictionaries are used for each stream, with (in the last row) all six of them combined into a single dictionary[] array, and six start[] arrays maintained.
Figure 6: Final effectiveness-efficiency graph for Gov2. The vertical scale sums the per-posting docids and freqs times for each method; the horizontal scale shows total index size in GiB. The two DINT points both use a single packed dictionary per stream, b = 16, and optimal parsing. Both scales are logarithmic.
Figure 6: Final effectiveness-efficiency graph for Gov2. The vertical scale sums the per-posting docids and freqs times for each method; the horizontal scale shows total index size in GiB. The two DINT points both use a single packed dictionary per stream, b = 16, and optimal parsing. Both scales are logarithmic.

Related papers

Compressed inverted files with reduced decoding overheads
Anh Vo

Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '98, 1998

Compressed inverted files are the most compact way of indexing large text databases, typically occupying around 10% of the space of the collection they index. The drawback of compression is the need to decompress the index lists during query processing. Here we describe an improved implementation of compressed inverted lists that eliminates almost all redundant decoding and allows extremely fast processing of conjunctive Boolean queries and ranked queries. We also describe a pruning method to reduce the number of candidate documents considered during the evaluation of ranked queries. Experimental results with a database of 510 Mb show that the new mechanism can reduce the CPU and elapsed time for Boolean queries of 4-10 terms to one tenth and one fifth respectively of the standard technique. For ranked queries, the new mechanism reduces both CPU and elapsed time to one third and memory usage to less than one tenth of the standard algorithm, with no degradation in retrieval effectiveness. Permission to make digital/hard copy of all or part of this work for personal or classroom we is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is given that copying is by permission of ACM. Inc. To COW otherwise. to reoublish. to oost on servers or I" to redistribute to lists, requires prior specific permission and/or fee. SIGIR'98,

View PDFchevron_right
Revisiting dictionary-based compression
Szymon Grabowski

Software - Practice and Experience, 2005

One of the attractive ways to increase text compression is to replace words with references to a text dictionary given in advance. Although there exist a few works in this area, they do not fully exploit the compression possibilities or consider alternative preprocessing variants for various compressors in the latter phase. In this paper, we discuss several aspects of dictionary-based compression, including compact dictionary representation, and present a PPM/BWCA oriented scheme, word replacing transformation, achieving compression ratios higher by 2-6% than stateof-the-art StarNT (2003) text preprocessor, working at a greater speed. We also present an alternative scheme designed for LZ77 compressors, with the advantage over StarNT reaching up to 14% in combination with gzip.

View PDFchevron_right
Hybrid compression of inverted lists for reordered document collections
Mauricio Oyarzún

Information Processing & Management, 2018

Text search engines are a fundamental tool nowadays. Their efficiency relies on a popular and simple data structure: inverted indexes. They store an inverted list per term of the vocabulary. The inverted list of a given term stores, among other things, the document identifiers (docIDs) of the documents that contain the term. Currently, inverted indexes can be stored efficiently using integer compression schemes. Previous research also studied how an optimized document ordering can be used to assign docIDs to the document database. This yields important improvements in index compression and query processing time. In this paper we show that using a hybrid compression approach on the inverted lists is more effective in this scenario, with two main contributions: • First, we introduce a document reordering approach that aims at generating runs of consecutive docIDs in a properly-selected subset of inverted lists of the index. • Second, we introduce hybrid compression approaches that combine gap and run-length encodings within inverted lists, in order to take advantage not only from small gaps, but also from long runs of consecutive docIDs generated by our document reordering approach. Our experimental results indicate a reduction of about 10%-30% in the space usage of the whole index (just regarding docIDs), compared with the most efficient state-of-the-art results. Also, decompression speed is up to 1.22 times faster if the runs of consecutive docIDs must be explicitly decompressed, and up to 4.58 times faster if implicit decompression of these runs is allowed (e.g., representing the runs as intervals in the output). Finally, we also improve the query processing time of AND queries (by up to 12%), WAND queries (by up to 23%), and full (non-ranked) OR queries (by up to 86%), outperforming the best existing approaches.

View PDFchevron_right
Survey of Dictionary Based Compression
Aarti Parekh

International Journal of Engineering Research and, 2015

Dictionary Based Compression is a useful technique through which we can encode variable-length strings of symbols as single tokens. There are number of algorithms available for Dictionary Based Compression. It uses less computing resources so it is very effective compression technique. The purpose of this paper is to present and analyze a variety of dictionary based algorithms.

View PDFchevron_right
Compression of inverted indexes for fast query evaluation
Falk Scholer

Proceedings of the 25th …, 2002

Compression reduces both the size of indexes and the time needed to evaluate queries. In this paper, we revisit the compression of inverted lists of document postings that store the position and frequency of indexed terms, considering two approaches to improving retrieval efficiency: better implementation and better choice of integer compression schemes. First, we propose several simple optimisations to well-known integer compression schemes, and show experimentally that these lead to significant reductions in time. Second, we explore the impact of choice of compression scheme on retrieval efficiency.

View PDFchevron_right

Related topics

  • Computer Science

    • Related papers

    Efficient Inverted Index Compression Algorithm Characterized by Faster Decompression Compared with the Golomb-Rice Algorithm
    Andrzej Chmielowiec

    Entropy, 2021

    This article deals with compression of binary sequences with a given number of ones, which can also be considered as a list of indexes of a given length. The first part of the article shows that the entropy H of random n-element binary sequences with exactly k elements equal one satisfies the inequalities klog2(0.48·n/k)

    View PDFchevron_right
    Faster strong: string dictionary compression using sampling and fast vectorized decompression
    Robert Lasch

    The VLDB Journal, 2020

    String dictionaries constitute a large portion of the memory footprint of database applications. While strong string dictionary compression algorithms exist, these come with impractical access and compression times. Therefore, lightweight algorithms such as front coding (PFC) are favored in practice. This paper endeavors to make strong string dictionary compression practical. We focus on Re-Pair Front Coding (RPFC), a grammar-based compression algorithm, since it consistently offers better compression ratios than other algorithms in the literature. To accelerate compression times, we propose block-based RPFC (BRPFC) which consists in independently compressing small blocks of the dictionary. For further accelerated compression times especially on large string dictionaries, we also propose an alternative version of BRPFC that uses sampling to speed up compression. Moreover, to accelerate access times, we devise a vectorized access method, using $$\hbox {Intel}^{\circledR }$$ Intel ® ...

    View PDFchevron_right
    When indexing equals compression
    Luca Foschini

    ACM Transactions on Algorithms, 2006

    We report on a new experimental analysis of high-order entropy-compressed suffix arrays, which retains the theoretical performance of previous work and represents an improvement in practice.

    View PDFchevron_right
    Improving Text Indexes Using Compressed Permutations
    Jeremy Barbay

    Citeseer

    Any sorting algorithm in the comparison model defines an encoding scheme for permutations. As adaptive sorting algorithms perform o(n lg n) comparisons on restricted classes of permutations, each defines one or more compression schemes for permutations. In the case of the compression schemes inspired by Adaptive Merge Sort, a small amount of additional data allows to support in good time the access and reversed access to the compressed permutation, without decompressing it. In this paper we explore the application of two of these compressed succinct data-structures to the encoding of inverted lists and of suffix arrays, and show experimentally that they yield a practical self-index on practical data-sets, from natural language to biological data.

    View PDFchevron_right
    Compressed Text Indexes:From Theory to Practice
    rodrigo gonzalez

    Computing Research Repository, 2007

    A compressed full-text self-index represents a text in a compressed form and still answers queries efficiently. This technology represents a breakthrough over the text indexing techniques of the previous decade, whose indexes required several times the size of the text. Although it is relatively new, this technology has matured up to a point where theoretical research is giving way to practical developments. Nonetheless this requires significant programming skills, a deep engineering effort, and a strong algorithmic background to dig into the research results. To date only isolated implementations and focused comparisons of compressed indexes have been reported, and they missed a common API, which prevented their re-use or deployment within other applications.

    View PDFchevron_right
    On parsing optimality for dictionary-based text compression—the Zip case
    alessio langiu

    Journal of Discrete Algorithms, 2013

    Dictionary-based compression schemes are the most commonly used data compression schemes since they appeared in the foundational paper of Ziv and Lempel in 1977, and generally referred to as LZ77. Their work is the base of Zip, gZip, 7-Zip and many other compression software utilities. Some of these compression schemes use variants of the greedy approach to parse the text into dictionary phrases; others have left the greedy approach to improve the compression ratio. Recently, two bit-optimal parsing algorithms have been presented filling the gap between theory and best practice. We present a survey on the parsing problem for dictionary-based text compression, identifying noticeable results of both a theoretical and practical nature, which have appeared in the last three decades. We follow the historical steps of the Zip scheme showing how the original optimal parsing problem of finding a parse formed by the minimum number of phrases has been replaced by the bit-optimal parsing problem where the goal is to minimise the length in bits of the encoded text.

    View PDFchevron_right
    Note on the Greedy Parsing Optimality for Dictionary-Based Text Compression
    alessio langiu

    2012

    a r t i c l e i n f o a b s t r a c t Keywords: Data compression Greedy parsing Lempel-Ziv factorisation LZ77 algorithm Optimal parsing Text compression

    View PDFchevron_right
    Parallelism and dictionary based data compression
    S. Agostino

    Information Sciences, 2001

    Because of the size of information involved with the emerging applications in multimedia and the Human Genome Project, parallelism offers the only hope of meeting the challenges of storing such databases and searching through quickly. In this paper, we address dictionary based lossless text compression and give the state-of-the-art in the field of parallelism. Static dictionary compression and sliding window

    View PDFchevron_right
    Compressed text indexes
    Rodrigo González

    Journal of Experimental Algorithmics, 2009

    A compressed full-text self-index represents a text in a compressed form and still answers queries efficiently. This represents a significant advancement over the (full-)text indexing techniques of the previous decade, whose indexes required several times the size of the text. Although it is relatively new, this algorithmic technology has matured up to a point where theoretical research is giving way to practical developments. Nonetheless this requires significant programming skills, a deep engineering effort, and a strong algorithmic background to dig into the research results. To date only isolated implementations and focused comparisons of compressed indexes have been reported, and they missed a common API, which prevented their re-use or deployment within other applications. The goal of this article is to fill this gap. First, we present the existing implementations of compressed indexes from a practitioner's point of view. Second, we introduce the Pizza&Chili site, which of...

    View PDFchevron_right
    Universal Compressed Text Indexing 1
    Gonzalo Navarro

    2018

    The rise of repetitive datasets has lately generated a lot of interest in compressed self-indexes based on dictionary compression, a rich and heterogeneous family of techniques that exploits text repetitions in different ways. For each such compression scheme, several different indexing solutions have been proposed in the last two decades. To date, the fastest indexes for repetitive texts are based on the run-length compressed Burrows–Wheeler transform (BWT) and on the Compact Directed Acyclic Word Graph (CDAWG). The most space-efficient indexes, on the other hand, are based on the Lempel–Ziv parsing and on grammar compression. Indexes for more universal schemes such as collage systems and macro schemes have not yet been proposed. Very recently, Kempa and Prezza [STOC 2018] showed that all dictionary compressors can be interpreted as approximation algorithms for the smallest string attractor, that is, a set of text positions capturing all distinct substrings. Starting from this obse...

    View PDFchevron_right