Query for image query result metadata separately#6370
Query for image query result metadata separately#6370WithoutPants merged 4 commits intostashapp:developfrom
Conversation
|
Much Thanks! will try on my machine once pushed. for reference my system has 4.7TB of images (7,7M) |
ce911a9 to
5eae9c2
Compare
|
I've done a bit of testing on a system with 32M images. When getting everything at once (legacy behaviour), the full query for a single image sorted by path took 29s for the aggregate query (count, filesize, megapixels), followed by 42s to get the image ids, for a total wait of 71s. When splitting it into two separate parallel queries, the query for the filesize and megapixels took 24s, then the query for the content took 5s for the count, and 42s to get the image ids. This means a total wait of 47s for the items to return, and 24s for the other info to populate. We could possibly shave some further time off that if we took out the count from the item query. Performance for larger systems is still nowhere near acceptable, but this should improve things a little. Further work needs to be done at the schema level to determine what indexes are worth adding and what we can do at the sorting level to improve performance. Related #6378 |
|
Is there a justification for calculating "total megapixels" at all? I find that metric to be completely irrelevant. Megapixels is useful when reviewing the properties of an individual image, not a whole collection. Maybe "total megapixels" is best placed on the Stats page. But I don't think it's worth calculating every time you open the Images tab, especially given how much it taxes processing time. |
Related feature request and PR: #1775 In my testing, adding megapixels added around 25% to the overhead when calculating total file size (which I think does belong). I am fairly ambivalent as to whether to keep it around - I think the utility is marginal at best. |
Makes the query for total megapixels and filesize separate from the main image query so that the former doesn't cause delays in returning the images themselves.
Shows
(...)in the metadata info section while the metadata info query is running:Haven't yet noticed a huge difference on my production system of >300k images. Going to try a larger db when I migrate it.
Resolves #6319