feat: add per-feed article metadata thumbnail enrichment#1130
Conversation
Add a per-feed setting to enhance missing/body-derived thumbnails from article page metadata during RSS sync. When enabled, sync fetches the article page head, extracts og:image or twitter:image, and applies it only when the feed did not provide a usable non-body thumbnail. Existing feed-provided thumbnails and previously upgraded non-body thumbnails are kept. Persist the setting in the database, encrypted feed sync, and OPML import/export. Add tests for thumbnail selection policy, feed-item updates, metadata image extraction, head streaming, OPML/export serialization, and encrypted feed serialization.
Hide the reader header thumbnail only when the same non-body image URL is already rendered in the article content. This keeps feed/metadata header images visible for articles that do not duplicate them inline, while still avoiding duplicate hero images when the article body already contains the same image.
spacecowboy
left a comment
spacecowboy
left a comment
There was a problem hiding this comment.
I think it looks good. Thanks @tokenflood
One question: if the head of the article is fetched for the og:image and rest of body is not fetched, this does not mess up the okhttp cache later if the full article is fetched? I assume okhttp is smart enough to handle it and you haven't noticed anything weird in your tests I assume?
| showHeaderImage: Boolean, | ||
| image: ThumbnailImage?, | ||
| contentImageUrls: Set<String>, | ||
| ): Boolean = showHeaderImage && image?.fromBody == false && image.url !in contentImageUrls |
| * Reads just enough bytes to find `</head>`, keeping network usage minimal | ||
| * (typically 1–2 KB vs 100+ KB for a full page). | ||
| */ | ||
| internal fun streamHeadContent(inputStream: InputStream): String { |
Good question. This does a normal GET, but the body is closed as soon as I’ve also been using this locally for over a month and haven’t noticed any weird cache/full-article behavior. |
2 participants
Some feeds have missing thumbnails. This PR adds a per-feed toggle to fetch
og:image/twitter:imagefrom the article page during RSS sync and use it when an item has no better thumbnail. It also avoids duplicate header images in the article reader.OG thumbnail enrichment
New per-feed toggle in feed settings (default OFF): Enhance thumbnails from article metadata. When enabled, each synced item that only has a body-extracted image (or no image at all) gets an extra request to the article URL to look for
og:image/twitter:imagemeta tags. Only the<head>section of the page is streamed and the connection is closed as soon as the</head>boundary is hit, so no full page body is downloaded.Feed-provided thumbnails (media/enclosure) are never replaced, and enrichment only applies to items whose displayed thumbnail is missing or body-derived. If a feed provides a thumbnail that is too small to use and the item falls back to a body image, enrichment is still attempted. Once an item has been upgraded to a non-body thumbnail, it won't be fetched again on later syncs. Relative image URLs are resolved against the final redirected page URL and blank metadata values are ignored.
The toggle survives encrypted feed sync and OPML import/export. There is no separate background job; enrichment happens inline during sync.
Full-text duplicate image fix
The article reader previously always showed the header thumbnail for non-body images, which could duplicate the same image inline. It now compares the header image URL against the rendered article content and only hides it when that URL already appears inline, so articles that do not duplicate the hero image in their body still keep their header image.
DB version bumped to 39 with a new
feeds.fetch_og_imagescolumn (default 0). Tests cover thumbnail policy, feed item update merging, reader image dedupe, OG image extraction, head-content streaming, and serialization/export coverage for the new field.Fixes #1125