Category Archives: Hacking

Beware the cursorMark, my son!

Posted on October 24, 2023 by Toke Eskildsen

Efficient export of stored text content from multi-shard Solr setups using cursorMark for individual shards and merging the results externally. Continue reading →

Posted in eskildsen, Hacking, open source, Performance, Solr | Tagged , , | Leave a comment

Dumb-down at Indexing or Nested Data in the Solr Search Engine

Posted on August 16, 2022 by Toke Eskildsen

Sigfrid Lundberg, Ph. D., Software Developer Royal Danish Library Copenhagen Denmark twitter — github — web site Are passions, then, the Pagans of the soul? Reason alone baptized? alone ordain’d To touch things sacred? (Edward Young — 1683-1765) Introduction The … Continue reading →

Posted in sigge, Solr, usability | Leave a comment

SolrWayback 4.0 release! What’s it all about? Part 2

Posted on February 26, 2021 by thomasegense

In this blog post I will go into the more technical details of SolrWayback and the new version 4.0 release. The whole frontend GUI was rewritten from scratch to be up to date with 2020 web-applications expectations along with many … Continue reading →

Posted in open source, Solr, Visualization, Web | Tagged , , , | 1 Comment

Which type bug?

Posted on June 10, 2020 by Toke Eskildsen

A light tale of bug hunting an Out Of Memory problem with SolrCloud. The setup and the problem At the Royal Danish Library we provide full text search for the Danish Netarchive. The heavy lifting is done in a single … Continue reading →

Posted in eskildsen, Solr | Tagged , | Leave a comment

DocValues jump tables in Lucene/Solr 8

Posted on March 12, 2019 by Toke Eskildsen

Lucene/Solr 8 is about to be released. Among a lot of other things is brings LUCENE-8585, written by your truly with a heap of help from Adrien Grand. LUCENE-8585 introduces jump-tables for DocValues, is all about performance and brings speed-ups … Continue reading →

Posted in eskildsen, Hacking, Low-level, Lucene, Performance, Solr, Uncategorized | 7 Comments

Faster DocValues in Lucene/Solr 7+

Posted on October 2, 2018 by Toke Eskildsen

This is a fairly technical post explaining LUCENE-8374 and its implications on Lucene, Solr and (qualified guess) Elasticsearch search and retrieval speed. It is primarily relevant for people with indexes of 100M+ documents. Teaser We have a Solr setup for … Continue reading →

Posted in eskildsen, Hacking, Low-level, Lucene, Performance, Solr | 1 Comment

Visualising Netarchive Harvests

Posted on March 7, 2017 by nielskristianhansenskovmand

  An overview of website harvest data is important for both research and development operations in the netarchive team at Det Kgl. Bibliotek. In this post we present a recent frontend visualisation widget we have made. From the SolrWayback Machine … Continue reading →

Posted in Blogging, Solr, Web | Tagged , | Leave a comment

70TB, 16b docs, 4 machines, 1 SolrCloud

Posted on November 30, 2016 by Toke Eskildsen

At Statsbiblioteket we maintain a historical net archive for the Danish parts of the Internet. We index it all in Solr and we recently caught up with the present. Time for a status update. The focus is performance and logistics, … Continue reading →

Posted in Hacking, Low-level, Performance, Solr, Statsbiblioteket, Uncategorized | 6 Comments

The ones that got away

Posted on November 13, 2015 by Toke Eskildsen

Two and a half ideas of improving Lucene/Solr performance that did not work out. Track the result set bits At the heart of Lucene (and consequently also Solr and ElasticSearch), there is a great amount of doc ID set handling. … Continue reading →

Posted in eskildsen, Hacking, Low-level, Lucene, open source, Performance, Solr | Leave a comment

Speeding up core search

Posted on October 5, 2015 by Toke Eskildsen

Issue a query, get back the top-X results. It does not get more basic with Solr. So great win if we can improve on that, right? Truth be told, the answer is still “maybe”, but read on for some thoughts, … Continue reading →

Posted in eskildsen, Hacking, Low-level, Lucene, open source, Performance, Solr, Uncategorized | 2 Comments