Academia.eduAcademia.edu

Figure from:Fast Dictionary-Based Compression for Inverted Indexes

See full PDFdownloadDownload figure
Figure 4: Example in which optimal parsing requires fewer codewords than greedy parsing. The sequence “aadbaaaa” is being represented relative to the dictionary shown in Table 1(b). The cost below each node is the length of the shortest path from the origin to that point. The parse shown at the bottom of the graph has a cost of 5; whereas the greedy parse shown above requires 6 codewords. In both cases it is assumed that “d” requires a rare symbol exception (rse) codeword, followed by a patch codeword that identifies the symbol required.

Figure 4 Example in which optimal parsing requires fewer codewords than greedy parsing. The sequence “aadbaaaa” is being represented relative to the dictionary shown in Table 1(b). The cost below each node is the length of the shortest path from the origin to that point. The parse shown at the bottom of the graph has a cost of 5; whereas the greedy parse shown above requires 6 codewords. In both cases it is assumed that “d” requires a rare symbol exception (rse) codeword, followed by a patch codeword that identifies the symbol required.

Related Figures (13)

Table 1: Two examples of plurally parsable dictionaries of width € = 4 over the alphabet {a,b,c,d} where symbol “a” is highly probable, symbol “b” is moderately probable, and “-” entries indicate don’t-care values. The last column provides the length of each string and is also stored as part of the dictionary. In (b), index zero is used as the code for rare symbol exceptions.
Table 1: Two examples of plurally parsable dictionaries of width € = 4 over the alphabet {a,b,c,d} where symbol “a” is highly probable, symbol “b” is moderately probable, and “-” entries indicate don’t-care values. The last column provides the length of each string and is also stored as part of the dictionary. In (b), index zero is used as the code for rare symbol exceptions.
Table 3: Performance counts reported by the Linux perf tool, com- paring variable-length copying and constant-length copying for € = 16 when decoding the index sequences of Gov2 using a rectan- gular dictionary. Table 2: Total index size in GiB for a complete document-level index (docids and freqs combined, including block-access overhead and dictionary space) for the Gov2 collection using a block size of B = 256 items and the DSV dictionary construction approach.
Table 3: Performance counts reported by the Linux perf tool, com- paring variable-length copying and constant-length copying for € = 16 when decoding the index sequences of Gov2 using a rectan- gular dictionary. Table 2: Total index size in GiB for a complete document-level index (docids and freqs combined, including block-access overhead and dictionary space) for the Gov2 collection using a block size of B = 256 items and the DSV dictionary construction approach.
Table 4: Average number of decoded integers per codeword, m, when decoding the sequences of Gov2, using DINT-DSF with a rectangular dictionary and b = 16; and predicted and actual de- coding times measured in nanoseconds per integer, based on c, the measured decoding time per codeword.
Table 4: Average number of decoded integers per codeword, m, when decoding the sequences of Gov2, using DINT-DSF with a rectangular dictionary and b = 16; and predicted and actual de- coding times measured in nanoseconds per integer, based on c, the measured decoding time per codeword.
Figure 3: Packed layout for the dictionary shown in Table 1(b), with € = 4 and b = 3. The first number in each element in start[] is the sequence length. All trailing don’t-care symbols have been trimmed, and dictionary sequences have been removed if they are a prefix o another longer sequence. As an example, when the input codeword is 3 the four integers in dictionary[4 ...7] are copied to the output, anc then the output pointer is incremented by two. In this illustration no provision has been made for frequent symbol exceptions.
Figure 3: Packed layout for the dictionary shown in Table 1(b), with € = 4 and b = 3. The first number in each element in start[] is the sequence length. All trailing don’t-care symbols have been trimmed, and dictionary sequences have been removed if they are a prefix o another longer sequence. As an example, when the input codeword is 3 the four integers in dictionary[4 ...7] are copied to the output, anc then the output pointer is incremented by two. In this illustration no provision has been made for frequent symbol exceptions.
Table 5: Number of lists, postings and documents for the Gov2 and CCNEWS collections.
Table 5: Number of lists, postings and documents for the Gov2 and CCNEWS collections.
Table 6: Total index size (GiB) and compression rate (bits per in- teger) for docids and freqs for two test collections, using a range of compression techniques. The two DINT implementations both make use of b = 16 and ¢ = 16, single dictionaries, and greedy parsing. The rows are ordered by decreasing total index size.
Table 6: Total index size (GiB) and compression rate (bits per in- teger) for docids and freqs for two test collections, using a range of compression techniques. The two DINT implementations both make use of b = 16 and ¢ = 16, single dictionaries, and greedy parsing. The rows are ordered by decreasing total index size.
Table 7: Sequential decoding throughput in nanoseconds per in- teger, measured over the complete index. Both DINT rows make use of b = 16 and a packed dictionary. The rows are ordered by increasing speed of docids decoding on Gov?2.
Table 7: Sequential decoding throughput in nanoseconds per in- teger, measured over the complete index. Both DINT rows make use of b = 16 and a packed dictionary. The rows are ordered by increasing speed of docids decoding on Gov?2.
Figure 5: Percentage of integers, codewords, and dictionary entries associated with each target size for the docids and freqs of Gov2, using the DSF approach and parameters b = 16, € = 16.
Figure 5: Percentage of integers, codewords, and dictionary entries associated with each target size for the docids and freqs of Gov2, using the DSF approach and parameters b = 16, € = 16.
Table 8: Total index size (GiB) and compression rate (bits per inte ger) for docids and freqs using b = 16 and ¢ = 16. The first row uses DINT-DSF with greedy parsing; four enhancements are then added and in the penultimate row a total of 12 contexts are used with optimal parsing and an exhaustive search to identify the cheapest context for each block. In the last row, the dictionary indices are then assumed to be input to a set of 12 optimal entropy coders Except for the last row (gray numbers), which contains values that are calculated rather than measured, these results can be directly compared with Table 6.
Table 8: Total index size (GiB) and compression rate (bits per inte ger) for docids and freqs using b = 16 and ¢ = 16. The first row uses DINT-DSF with greedy parsing; four enhancements are then added and in the penultimate row a total of 12 contexts are used with optimal parsing and an exhaustive search to identify the cheapest context for each block. In the last row, the dictionary indices are then assumed to be input to a set of 12 optimal entropy coders Except for the last row (gray numbers), which contains values that are calculated rather than measured, these results can be directly compared with Table 6.
Table 9: Dictionary space (MiB) for different schemes and corre- sponding decoding speeds, for Gov2. In the three “x1” rows, one dictionary is used for the docids and another for the freqs. In the “x6” rows, six dictionaries are used for each stream, with (in the last row) all six of them combined into a single dictionary[] array, and six start[] arrays maintained.
Table 9: Dictionary space (MiB) for different schemes and corre- sponding decoding speeds, for Gov2. In the three “x1” rows, one dictionary is used for the docids and another for the freqs. In the “x6” rows, six dictionaries are used for each stream, with (in the last row) all six of them combined into a single dictionary[] array, and six start[] arrays maintained.
Figure 6: Final effectiveness-efficiency graph for Gov2. The vertical scale sums the per-posting docids and freqs times for each method; the horizontal scale shows total index size in GiB. The two DINT points both use a single packed dictionary per stream, b = 16, and optimal parsing. Both scales are logarithmic.
Figure 6: Final effectiveness-efficiency graph for Gov2. The vertical scale sums the per-posting docids and freqs times for each method; the horizontal scale shows total index size in GiB. The two DINT points both use a single packed dictionary per stream, b = 16, and optimal parsing. Both scales are logarithmic.