Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '98, 1998
Compressed inverted files are the most compact way of indexing large text databases, typically oc... more Compressed inverted files are the most compact way of indexing large text databases, typically occupying around 10% of the space of the collection they index. The drawback of compression is the need to decompress the index lists during query processing. Here we describe an improved implementation of compressed inverted lists that eliminates almost all redundant decoding and allows extremely fast processing of conjunctive Boolean queries and ranked queries. We also describe a pruning method to reduce the number of candidate documents considered during the evaluation of ranked queries. Experimental results with a database of 510 Mb show that the new mechanism can reduce the CPU and elapsed time for Boolean queries of 4-10 terms to one tenth and one fifth respectively of the standard technique. For ranked queries, the new mechanism reduces both CPU and elapsed time to one third and memory usage to less than one tenth of the standard algorithm, with no degradation in retrieval effectiveness. Permission to make digital/hard copy of all or part of this work for personal or classroom we is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is given that copying is by permission of ACM. Inc. To COW otherwise. to reoublish. to oost on servers or I" to redistribute to lists, requires prior specific permission and/or fee. SIGIR'98,
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '98, 1998
Compressed inverted files are the most compact way of indexing large text databases, typically oc... more Compressed inverted files are the most compact way of indexing large text databases, typically occupying around 10% of the space of the collection they index. The drawback of compression is the need to decompress the index lists during query processing. Here we describe an improved implementation of compressed inverted lists that eliminates almost all redundant decoding and allows extremely fast processing of conjunctive Boolean queries and ranked queries. We also describe a pruning method to reduce the number of candidate documents considered during the evaluation of ranked queries. Experimental results with a database of 510 Mb show that the new mechanism can reduce the CPU and elapsed time for Boolean queries of 4-10 terms to one tenth and one fifth respectively of the standard technique. For ranked queries, the new mechanism reduces both CPU and elapsed time to one third and memory usage to less than one tenth of the standard algorithm, with no degradation in retrieval effectiveness. Permission to make digital/hard copy of all or part of this work for personal or classroom we is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is given that copying is by permission of ACM. Inc. To COW otherwise. to reoublish. to oost on servers or I" to redistribute to lists, requires prior specific permission and/or fee. SIGIR'98,
Uploads
Papers by Anh Vo