We already use readability to extract text from singlefile.html (https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/extractors/readability.py#L57C23-L57C23), is a separate parser really needed?

Indexing prefers readability over singlefile, and ignores mercury altogether:

Even if we add mercury into the precedence list (IMO a good idea), both readability and mercury frequently fail to extract content, because their goal is to extract the only the main, human-readable text content from a page. Both tools prefer failing with empty output over generating bad-looking content.

The goal for full-text search is different. You'd rather get an accurate but ugly stream of text tokens than empty output. So--if readability or mercury produce high-quality, high-relevance content that can be indexed--great! But if they fail with empty output, it would be better to extract all the possibly meaningful text from singlefile.html for indexing rather than feed the entire singlefile output, including base64 binary data, to the search indexer.

Ok, I see the appeal, however I think if this is going to serve as a mercury/readability alternative, it should be implemented as a real extractor instead of a util.

I'm sorry for the reimplementation burden but I think if this is a true post-processing step during archiving, then it's doing the job an extractor would do, and should be formatted like a sibling of all the other extractors.

I forsee situations where a pure python text extraction option would be appealing to people, and having this outputted text directly as an extractor output like any other is nice. We can then re-use the existing code paths to ingested extractor output into sonic.

We could call it something like extractors/htmltotext.py? Open to other naming ideas.

There is also documentation on how to implement an extractor here: https://github.com/ArchiveBox/ArchiveBox#contributing-a-new-extractor

I see what you're saying... same approach but make it better? I'll add some light formatting (linebreaks, etc.) to the output to make it marginally human-readable, rather than the current "glue everything together with spaces" approach.

Incidentally, I've got a prototype ActivityPub extractor (with optional Mastodon API for authentication) in my work-in-progress branches. It downloads post-attached media more efficiently, and it will archive links included in posts. It's still a little rough, and it's been on the back burner for a while. I'll see about shaping it up to let you have a look.