Category Archives: open source

Beware the cursorMark, my son!

Posted on October 24, 2023 by Toke Eskildsen

Efficient export of stored text content from multi-shard Solr setups using cursorMark for individual shards and merging the results externally. Continue reading →

Posted in eskildsen, Hacking, open source, Performance, Solr | Tagged , , | Leave a comment

SolrWayback 4.0 release! What’s it all about? Part 2

Posted on February 26, 2021 by thomasegense

In this blog post I will go into the more technical details of SolrWayback and the new version 4.0 release. The whole frontend GUI was rewritten from scratch to be up to date with 2020 web-applications expectations along with many … Continue reading →

Posted in open source, Solr, Visualization, Web | Tagged , , , | 1 Comment

The ones that got away

Posted on November 13, 2015 by Toke Eskildsen

Two and a half ideas of improving Lucene/Solr performance that did not work out. Track the result set bits At the heart of Lucene (and consequently also Solr and ElasticSearch), there is a great amount of doc ID set handling. … Continue reading →

Posted in eskildsen, Hacking, Low-level, Lucene, open source, Performance, Solr | Leave a comment

Speeding up core search

Posted on October 5, 2015 by Toke Eskildsen

Issue a query, get back the top-X results. It does not get more basic with Solr. So great win if we can improve on that, right? Truth be told, the answer is still “maybe”, but read on for some thoughts, … Continue reading →

Posted in eskildsen, Hacking, Low-level, Lucene, open source, Performance, Solr, Uncategorized | 2 Comments

Sampling methods for heuristic faceting

Posted on July 30, 2015 by Toke Eskildsen

Initial experiments with heuristic faceting in Solr were encouraging: Using just a sample of the result set, it was possible to get correct facet results for large result sets, reducing processing time by an order of magnitude. Alas, further experimentation … Continue reading →

Posted in eskildsen, Faceting, Low-level, open source, Performance, Solr | Leave a comment

Dubious guesses, counted correctly

Posted on June 19, 2015 by Toke Eskildsen

We do have a bit of a performance challenge with heavy faceting on large result sets in our Solr based Net Archive Search. The usual query speed is < 2 seconds, but if the user requests aggregations based on large … Continue reading →

Posted in eskildsen, Faceting, Low-level, open source, Performance, Solr | 1 Comment

Net Archive Search building blocks

Posted on June 9, 2015 by Toke Eskildsen

An extremely webarchive-discovery and Statsbiblioteket centric description of some of the technical possibilities with Net Archive Search. This could be considered internal documentation, but we like to share. There are currently 2 generations of indexes at Statsbiblioteket: v1 (22TB) & … Continue reading →

Posted in eskildsen, open source, Solr | 2 Comments

Sparse facet caching

Posted on September 19, 2014 by Toke Eskildsen

As explained in Ten times faster, distributed faceting in standard Solr is two-phase: Each shard performs standard faceting and returns the top limit*1.5+10 terms. The merger calculates the top limit terms. Standard faceting is a two-step process: For each term … Continue reading →

Posted in eskildsen, Faceting, Hacking, Low-level, open source, Performance, Solr | 3 Comments

Ten times faster

Posted on August 26, 2014 by Toke Eskildsen

One week ago I complained about Solr’s two-phase distributed faceting being slow in the second phase – ten times slower than the first phase. The culprit was the fine-counting of top-X terms, with each term-count being done as an intersection … Continue reading →

Posted in eskildsen, Faceting, Hacking, Low-level, open source, Performance, Solr, Uncategorized | 5 Comments

Sparse facet counting without the downsides

Posted on April 4, 2014 by Toke Eskildsen

The SOLR-5894 issue “Speed up high-cardinality facets with sparse counters” is close to being functionally complete (facet.method=fcs and facet.sort=index still pending). This post explains the different tricks used in the implementation and their impact on performance. Baseline Most of the … Continue reading →

Posted in eskildsen, Faceting, Low-level, open source, Performance, Solr | 2 Comments