Is there an existing issue for this?
Use case
Recently, I saw Sentry released a new feature https://sentry.io/for/session-replay/. After a quick search, that feature seems to be there already with many companies and products providing it. Shortly speaking, the so-called "session replay" tries to record the user screen as a (possibly pseudo) video. Then, when a bug occurs, the user can upload that already-recorded video, and thus we can happily debug it. There are many products supporting the Web already. Many companies providing this feature says they follows GDPR, so I guess this feature does not harm to user privacy. Needless to say, this seems quite helpful for debugging ;)
Proposal
Therefore, I am proposing that, maybe we can add a session replay feature to Flutter. Or, if Flutter is unhappy to add so many first-party code, Flutter only needs to expose some low-level API, and such high-level replay features can be third-party open-source libraries. Again, I am willing to PR.
Proposal 1: At RenderObject level
Simply dump (the diff of) the RenderObject tree. However, the tree may be large. Moreover, it may not be easy to recover the UI from the tree, since we may have to store RenderObject's various fields, some of them can be very heavy or even non-serializable (such as callbacks).
Proposal 2: At Layer level
For example, a naive solution is that, make a delegate of Canvas inside PaintingContext, and record every operation like drawRect and so on.
However, since Path does not expose any Dart method to read the underlying data, drawPath may need more modifications in order to get the data. Similar things happen for Paragraph, which is needed when painting text. For example, one way is to add a Path.dump() function which dumps its underlying state.
If we can record the whole (diff) layer tree and the pictures, we can store it and replay it in the future, thus implemented the proposed feature.
Cons:
- If we have quite complex scenes, such as very heavy Paths, it may be large in disk size.
- It may be hard to shrink resolution and "image" quality if we want to reduce recorded file size, because everything is vector instead of bitmap.
Proposal 3: At Impeller / Archivist level
As mentioned in #53602 (comment), Impeller is going to have https://github.com/flutter/engine/tree/main/impeller/archivist. Therefore, one possibility is to utilize this (or some other Impeller features) to extract rendering data. I have not checked into details, whether it will be too low-level and verbose.
Discord link: https://discord.com/channels/608014603317936148/608021234516754444/1120191067195723826
Proposal 4: Transfer resized rasterized images
Hack the rasterizer. Just before it is going to present the drawn UI onto user's screen, somehow get the rasterized screen data from the GPU buffer. If possible, resize it down, such that the transfer from device to host (if any) and later postprocessing can be faster. After getting one bitmap per frame, use standard encoding techniques (e.g. ffmpeg?) to compress it.
This is better than repeatedly calling Scene.toImage, since the latter will duplicate the whole rasterization process, which can be very slow.
Proposal 5: Get bitmaps at native Android/iOS layer
It seems most existing non-open source projects utilize this approach (not checked into details though), since their Flutter integration is just a simple wrapper of Android/iOS integration.
For example, we can convert android View into a Bitmap, using standard "take a screenshot in Android" technique - PixelCopy. I guess this will not be slow, because Flutter does not have a native View hierarchy, but only one single FlutterView with rasterized bitmaps (not checked details yet).
Alternative solutions
Alternative 1: Scene.toImage
This seems to be CPU, memory and disk heavy. In order to encode the sequence of images into a compressed format (say, video format? or zip of pngs?), it will need a lot of CPU. In order to transfer the ui.Image from GPU to CPU, it needs memory transfer. In addition, the resultant video can be big and requires much disk storage.
Alternative 2: Existing products
There exists some products that supports Flutter. However, I briefly looked at them, and they seem to use native Android/iOS and record screens. I am worried this can make the resulting video quite huge, needs some CPU (especially on low-end devices), and lacks fine-grained optimizations.
As a simple example of fine-grained control, we may want to replace real images with blank placeholders when doing (pseudo) screen recording, because we know vivid images are huge to encode and transfer, while those images may add little value to our debugging process. This may not be very easy with those native Android/iOS solutions (though I have not done thorough experiments).
Is there an existing issue for this?
Use case
Recently, I saw Sentry released a new feature https://sentry.io/for/session-replay/. After a quick search, that feature seems to be there already with many companies and products providing it. Shortly speaking, the so-called "session replay" tries to record the user screen as a (possibly pseudo) video. Then, when a bug occurs, the user can upload that already-recorded video, and thus we can happily debug it. There are many products supporting the Web already. Many companies providing this feature says they follows GDPR, so I guess this feature does not harm to user privacy. Needless to say, this seems quite helpful for debugging ;)
Proposal
Therefore, I am proposing that, maybe we can add a session replay feature to Flutter. Or, if Flutter is unhappy to add so many first-party code, Flutter only needs to expose some low-level API, and such high-level replay features can be third-party open-source libraries. Again, I am willing to PR.
Proposal 1: At RenderObject level
Simply dump (the diff of) the RenderObject tree. However, the tree may be large. Moreover, it may not be easy to recover the UI from the tree, since we may have to store RenderObject's various fields, some of them can be very heavy or even non-serializable (such as callbacks).
Proposal 2: At Layer level
For example, a naive solution is that, make a delegate of
CanvasinsidePaintingContext, and record every operation likedrawRectand so on.However, since
Pathdoes not expose any Dart method to read the underlying data,drawPathmay need more modifications in order to get the data. Similar things happen forParagraph, which is needed when painting text. For example, one way is to add aPath.dump()function which dumps its underlying state.If we can record the whole (diff) layer tree and the pictures, we can store it and replay it in the future, thus implemented the proposed feature.
Cons:
Proposal 3: At Impeller / Archivist level
As mentioned in #53602 (comment), Impeller is going to have https://github.com/flutter/engine/tree/main/impeller/archivist. Therefore, one possibility is to utilize this (or some other Impeller features) to extract rendering data. I have not checked into details, whether it will be too low-level and verbose.
Discord link: https://discord.com/channels/608014603317936148/608021234516754444/1120191067195723826
Proposal 4: Transfer resized rasterized images
Hack the rasterizer. Just before it is going to present the drawn UI onto user's screen, somehow get the rasterized screen data from the GPU buffer. If possible, resize it down, such that the transfer from device to host (if any) and later postprocessing can be faster. After getting one bitmap per frame, use standard encoding techniques (e.g. ffmpeg?) to compress it.
This is better than repeatedly calling
Scene.toImage, since the latter will duplicate the whole rasterization process, which can be very slow.Proposal 5: Get bitmaps at native Android/iOS layer
It seems most existing non-open source projects utilize this approach (not checked into details though), since their Flutter integration is just a simple wrapper of Android/iOS integration.
For example, we can convert android View into a Bitmap, using standard "take a screenshot in Android" technique -
PixelCopy. I guess this will not be slow, because Flutter does not have a native View hierarchy, but only one single FlutterView with rasterized bitmaps (not checked details yet).Alternative solutions
Alternative 1:
Scene.toImageThis seems to be CPU, memory and disk heavy. In order to encode the sequence of images into a compressed format (say, video format? or zip of pngs?), it will need a lot of CPU. In order to transfer the ui.Image from GPU to CPU, it needs memory transfer. In addition, the resultant video can be big and requires much disk storage.
Alternative 2: Existing products
There exists some products that supports Flutter. However, I briefly looked at them, and they seem to use native Android/iOS and record screens. I am worried this can make the resulting video quite huge, needs some CPU (especially on low-end devices), and lacks fine-grained optimizations.
As a simple example of fine-grained control, we may want to replace real images with blank placeholders when doing (pseudo) screen recording, because we know vivid images are huge to encode and transfer, while those images may add little value to our debugging process. This may not be very easy with those native Android/iOS solutions (though I have not done thorough experiments).