MSC4274: Inline media galleries via msgtypes#4274
MSC4274: Inline media galleries via msgtypes#4274Johennes wants to merge 2 commits intomatrix-org:mainfrom
Conversation
Signed-off-by: Johannes Marbach <[email protected]>
Johennes
force-pushed
the
johannes/msgtype-galleries
branch
from
d957ab5 to
4edac2e
Compare
Johennes
changed the title
turt2live
added
proposal
There was a problem hiding this comment.
Implementation requirements:
- Client
There was a problem hiding this comment.
- ruma: events: add gallery msgtype for MSC4274 ruma/ruma#2044
- Very draft matrix-rust-sdk: Add experimental support for MSC4274 (inline media galleries) matrix-rust-sdk#4838
There was a problem hiding this comment.
- Element X Android: Support gallery messages element-hq/element-x-android#6519
- Element Web: Support gallery messages element-hq/element-web#33111
| `org.matrix.custom.html` is supported. | ||
| - `formatted_body` (string): The formatted version of the body. Required if | ||
| `format` is specified. | ||
| - `itemtypes` (array): Ordered array of metadata for each item in the gallery. |
There was a problem hiding this comment.
Why is this called itemtypes and not simply items?
There was a problem hiding this comment.
Hm, I think I landed on this because itemtypes is basically the generalization of msgtype plus the associated data into an array. I agree that items would be clearer though.
This was broken out of #4838 and is a preliminary step towards implementing [MSC4274](matrix-org/matrix-spec-proposals#4274). The `media_handles` field on `SendHandle` is turned into a vector so that it can hold handles for several media when upload a gallery later. Signed-off-by: Johannes Marbach <[email protected]>
…ueuedRequestKind::UploadFileWithThumbnail to prepare for MSC4274 gallery uploads (#4897) This was broken out of #4838 and is a preliminary step towards implementing [MSC4274](matrix-org/matrix-spec-proposals#4274). `SentRequestKey::Media` and `DependentQueuedRequestKind::UploadFileWithThumbnail` are generalized to allow chaining dependent media uploads and accumulating sent media sources. - [x] Public API changes documented in changelogs (optional) --------- Signed-off-by: Johannes Marbach <[email protected]> Co-authored-by: Benjamin Bouvier <[email protected]>
This was broken out of #4838 and is a step towards implementing [MSC4274](matrix-org/matrix-spec-proposals#4274). * The entry point for sending galleries via the send queue is a new method [`RoomSendQueue::send_gallery`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-8752e86459c22cff470d7ca617240dcbdf222c3c6c98be2af2e43ddec071154cR362) which is a generalization of `RoomSendQueue::send_attachment`. * `send_gallery` takes as input parameters a [`GalleryConfig`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-d38d74cec6159cf769c32bed496199146175f05428fc23ab13bc2c629900da3eR283) (containing info about the gallery itself, such as its caption) and a vector of [`GalleryItemInfo`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-d38d74cec6159cf769c32bed496199146175f05428fc23ab13bc2c629900da3eR355)s (containing info about each image / file / etc. in the gallery). * `send_gallery` creates the gallery event content via [`Room:make_message_event`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-474f20e47fdcf60feac3f839a81f82fbb2c1fd0bb406388b0262adb60216ce3bR2281) which was renamed from `make_attachment_event` to reflect the fact that it creates general `msgtype` events now. * `send_gallery` maps the passed item infos into [`GalleryItemQueueInfo`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-d569f54c7901b5cd660aae77045c80dabbd3c251f0fa5891530469f35941327aR2008)s. This additional struct allows grouping all the metadata for a single gallery item together. * Finally `send_gallery` invokes [`QueueStorage::push_gallery`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-d569f54c7901b5cd660aae77045c80dabbd3c251f0fa5891530469f35941327aR1294) which is a generalization of `QueueStorage::push_media`. * `send_gallery` pushes upload requests for the media and thumbnails to the queue in a "daisy chain" manner. The first thumbnail (or media if no thumbnail exists) is pushed as a `QueuedRequestKind::MediaUpload`. The remaining thumbnails and media are pushed as `DependentQueuedRequestKind::UploadFileOrThumbnail`s while chaining each request to the previous one. * Finally a [`DependentQueuedRequestKind::FinishGallery`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-42a1a7ebe2446d916c9e013293bafc5eb16fd89bfe89248e3877ef031cc83ef2R265) is pushed to finalize the gallery upload (analogous to the existing `FinishUpload` for single media uploads). * The `FinishGallery` request is handled in [`QueueStorage::handle_dependent_finish_gallery_upload`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-8752e86459c22cff470d7ca617240dcbdf222c3c6c98be2af2e43ddec071154cR628) which was modeled after `handle_dependent_finish_upload`. * To be able to map the temporary event source for each gallery item to the final `AccumulatedSentMediaInfo`, a hash map is used. * Using the hash map, the gallery event is then updated to use the actual media sources via [`update_gallery_event_after_upload`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-8752e86459c22cff470d7ca617240dcbdf222c3c6c98be2af2e43ddec071154cR104) and a new method [`Room::make_gallery_item_type`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-474f20e47fdcf60feac3f839a81f82fbb2c1fd0bb406388b0262adb60216ce3bR2303) * An [integration test](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-21532ad5467a69489d7913b8da7afd4d618b7e357ce94f769e6e60e395b58055R2048) has been added to demonstrate the functionality. This is relatively large, unfortunately, but including everything needed to actually send the event made it possible to also add a test for it. It would be nice if the amount of new code could be reduced but I'm struggling a bit to find ways to integrate galleries with the existing media uploads further. - [x] Public API changes documented in changelogs (optional) --------- Signed-off-by: Johannes Marbach <[email protected]>
This was broken out of #4838 and is a step towards implementing matrix-org/matrix-spec-proposals#4274. Building upon #4977, a new method `Timeline::send_gallery` in matrix-sdk-ui for sending galleries. - [x] Public API changes documented in changelogs (optional) --------- Signed-off-by: Johannes Marbach <[email protected]>
|
Just checking. Is this feature now implemented and available? |
This is the specification of the feature. See this thread for any clients that are declaring support for the feature in it's experimental state. I don't see any client implementations yet. |
Thanks, OK. So looks like the feature set has been merged in, just still no client that is supporting it yet. |
|
|
||
| ## Proposal | ||
|
|
||
| A new `msgtype` of `m.gallery` is introduced with the following properties in |
There was a problem hiding this comment.
If we are taking in the scope not only images, but also files (m.file), why do we call this a "gallery"?
From my point of view, the best term for this feature is "attachments", because actually on the technical side we have a regular m.message with just some attached entities (images, videos, files, anything else).
And the term "gallery" is related more to the displaying style, not to the message type, because a message with 5 attached .zip files will be hard to display as a gallery ;) Even the same "5 images" can be displayed as a list of file names, not as a gallery, if we have a text-based UI.
So, naming this not as a "gallery" but using a more general term, will extend the scope of such message type, and will allow, in the future, to put into itemtypes other interesting things like references to other messages, multiple forwarded messages, etc.
What do you think about this idea?
| Finally, rather than defining galleries on the sender's side, receiving clients | ||
| could opportunistically group consecutive images in the timeline into galleries. | ||
| This might need specific rules such as grouping by sender or time windowing. | ||
| Different clients might prefer very different display rules, however. Therefore, |
There was a problem hiding this comment.
I think we should clarify here the displaying rules: where the "body" text should be displayed - above the attached items or below them, or maybe add a field that holds the display style.
Because this is crucial from the user point of view:
-
If we display the body above, it will be the main "body" of the message, and the attached images are just an addition (Slack style).
-
If we display the body below, the main content of the message will be files, and the "body" will be just a commenting text, in addition to the images (Telegram style).
Both variants of rendering are possible and available in different messengers, but they bear completely different meanings of the message.
There was a problem hiding this comment.
I, personally, am a fan of the Slack style, not Telegram style ;)
Just imagine a message:
Variant 1:
Checkout my photos from FOSDEM 2025:
[photo 1]
[photo 2]
[photo 3]
Variant 2:
[photo 1]
[photo 2]
[photo 3]
Checkout my photos from FOSDEM 2025:
The order of things in variant 1 looks more correct, right?
There was a problem hiding this comment.
Actually, these are two totally different things from the logical point of view:
- Above files - is the "message body" (main content).
- Below files - is the "message caption" (secondary thing: signature, comment, subscription, not the main content).
And if we keep the text named as "body", we should put it above. Or, rename to "caption" then.
Or, maybe even keep both ones ;) with recommendations what to render where.
|
In my opinion, this is (from a non-technical point of view) the difference between attachment(s) and gallery. For a gallery I'd prefer Variant 2 [photo 1] I think this is what non professional users expect from a gallery. As far as I remember, Telegram even allows to add captions per image, which is very useful to describe i.e. holiday pictures. For a bunch of files I'd prefer Variant 1 |
8 participants
Rendered
In line with matrix-org/matrix-spec#1700, the following disclosure applies:
I am a Systems Architect at gematik, Software Engineer at Filament and Unomed, Matrix community member and former Element employee. This proposal was written and published with my Filament hat on.