Great idea! I was thinking about doing this myself too but you beat me to it.

Have you tested to make sure the monkey patching at the top doesn't affect other areas of the codebase that depend on the native database connection provided by django?

Let me know when you're ready for final review and I'll test/merge it!

I tested this on my "prod" dataset, and the index it created was huge at 1GB, 30% of the size of the data/archive folder. This seemed odd, because it's using FTS5 "contentless" indexes. I'm fairly sure that what's happening is that SQLite is indexing very long "terms", in the form of base64 strings in data: URLs, when it is given the contents of singlefile.html as the indexable content.

I thought about and experimented with several approaches to this, and I think the best approach is to (very loosely) parse singlefile.html and pass only text and some attributes to the search backend, rather than the entire HTML contents. I've opened a pull request with this approach: #1244. This resulted in a 10x reduction in index size. I think that merging this PR without somehow addressing the ballooning index size would result in issues down the road.

I think that Sonic doesn't have this problem because it limits maximum term length, but FTS5 doesn't have a configuration option for maximum (or minimum) term length. Or, maybe it's just the overall MAX_SONIC_TEXT_TOTAL_LENGTH? Even if that's the case, the approach of parsing and including only meaningful text content, not JavaScript code, data URLs, and markup, would improve the signal-to-noise ratio of indexed content, so searching for "html" wouldn't hit on every document that had its singlefile.html indexed.

Let me know when you're ready for final review and I'll test/merge it!

Alright, @pirate, I think it's ready for a hard look. Thanks!