Repository navigation
Screenshot support in impeller for AI tooling #170357
Description
Use case
The upcoming Dart MCP server wants to be able to expose screenshotting capability to LLMs through an MCP tool. There are many potential use cases:
- Allow the LLM to validate its changes visually (if you ask to change the color of a button, after hot reloading did the color actually change as expected?).
- Prompt the LLM to improve the visual style of your app, this allows it to see what it currently looks like in order to suggest changes.
- Allows you to ask the LLM to edit something by describing it visually, it can screenshot the current page to understand which thing you are talking about.
Proposal
We should have a VM service method that can reliably take screenshots on any device, probably the _flutter.screenshot extension method. This would allow us to take screenshots for a specific app on any device.
Current state
Calling the _flutter.screenshot VM service method (the non-skia specific extension), returns _flutter.screenshot: (-32000) Could not capture image screenshot.
Similarly, running flutter screenshot --type device --vm-service-url <url> currently crashes with:
Oops; flutter has exited unexpectedly: "_flutter.screenshotSkp: (-32000) Cannot capture SKP screenshot with Impeller enabled.".
This is using a different extension method that seems skia specific, if we do get impeller support we might want to change this tool to invoke the more general version.
Ideally, one of these would be implemented on impeller (the former one doesn't say specifically that it is an impeller issue but I think it likely is?).
Why not device screenshots?
The upcoming Dart MCP server links up to your apps by VM service URI, and doesn't know the device type, so --type device isn't a great option (we would have to guess the device type or list all devices and require the user to select the device type).
Today, we only get screenshot support through the vm service method by default on desktop devices though, so that means the AI can't validate its changes visually any other device.
Other options
Get device name over VM service
We could potentially do a device screenshot if we could get the device name over the vm service, so possibly flutter tool could register an extension method to get the device id? This might be a simpler solution if it is challenging to implement screenshots on impeller.
Take device screenshots but identified by VM service URI
Alternatively, the flutter screenshot command could be able to take device screenshots, but for a specific app identified by its VM service URI, instead of a device ID.
Activity
jonahwilliams commentedon Jun 10, 2025
SKP is an internal skia picture format (draw Rect with color, saveLayer, restore) , we wired it up for debugging shader compilation jank issues.
I would highly recommend using a Flutter Tool mediated vm service method that takes a screenshot through the existing mechanisms and then reads the bytes off the file system.
jakemac53 commentedon Jun 10, 2025
I updated the proposal - we actually use the
_flutter.screenshotextension method today which should be more general. It is my understanding that eventually impeller will need this support anyways in order to support golden testing.I am not generally opposed to this - it is the path I initially went down - the problem is that we have just a VM service connection, and don't have a way to call
flutter screenshotand reliably get a screenshot for the device that corresponds to that connection specifically. We don't know which device to choose, and in the case of multiple apps connected to the same device (assuming this is possible), I don't think we have any way to select the correct one.See my "other options" section which does include two possible alternatives that would allow us to go through the
flutter screenshotcommand.jakemac53 commentedon Jun 10, 2025
Actually, it looks like there is a common DTD event that does give us the device ID (see EditorDebugSession here). However, I think we are planning to move away from this API, cc @kenzieschmoll I can't recall if the newer VM service registry will contain this similar info?
jonahwilliams commentedon Jun 10, 2025
Its possible to make the regular screenshot work. There are two caveats: 1) we can never give you the pixels from a platform view (those would show up empty) and 2) its unclear how it works with multiview.
If neither of those are blockers then implementing this API is possible
jakemac53 commentedon Jun 10, 2025
Those are good points - I assume a device screenshot would capture everything on screen including platform views.
Not sure what it means for multi-window in general - or what a device screenshot means for desktop (whole desktop?) or browser devices (entire browser window?).
jonahwilliams commentedon Jun 10, 2025
I think for desktop we just capture the main window. For browser I forgot how it works :)
jakemac53 commentedon Jun 10, 2025
I suppose that maybe the MCP tool should actually expose both options - device screenshots and flutter screenshots? Similar to how the flutter tool works.
jakemac53 commentedon Jun 11, 2025
Although, I realize now that device screenshots might be a bit sketchy in terms of what they might capture - especially in terms of the desktop device, but also other devices, especially if we aren't sure its going to capture the correct screen/tab/whatever.
jakemac53 commentedon Jun 27, 2025
Especially in the face of issues like this one - the idea of an unsuspecting user leaking a screenshot of their entire desktop or device is fairly scary. It is a lot less scary imo if it is just the flutter app pixels - although certainly still not ideal if that were to happen.
disable the screenshots tool by default (#210)
sethladd commentedon Oct 31, 2025
did we ever find a workaround to feed a screenshot from the running app into the coding agent?
jakemac53 commentedon Nov 3, 2025
We are currently using
flutter_driverto do this which seems to work on all platforms, but requires you to set up a flutter_driver entry point for your appsethladd commentedon Nov 3, 2025
thanks for confirming. Having to setup a flutter_driver entry point is less than ideal, but it does work.
I found some other MCP servers that do enable screenshots of mobile devices, I can try those.
jakemac53 commentedon Nov 4, 2025
This is fine if you just care about the devices that those support etc - but I think for the dart/flutter MCP server itself the primary goal is supporting all devices, so it works for all users regardless of their target device. For this purpose, flutter_driver seems to be the best right now.
But, resolving this issue would unlock the ability to use the screenshot extension method without going through flutter driver, no matter the renderer that is used (skia/impeller).
AriaShishegaran commentedon Nov 21, 2025
I want to mention that in order to bring actual usefulness and superiority into the LLM-in-the-Loop thing, It needs to, imo, have a couple of other factors in which could be considered other tasks or enhancement but I believe it is worth mentioning.
It needs to have direct access to navigating the app while having access to profiler and boundary drawings over the layout of the app, so that if for example and element is having a poor padding or nested inside other containers that are also having their own paddings added on top it can visually understand what's causing the problem.
Navigation + Screenshots + Screenshots with Boundaries overlay + Profiler access with rich information over what's exactly happening as it is navigating provides such rich set of information that debugging becomes easy, basically turning flutter into an extremely Agent friendly framework where LLMs can actually use the information as it navigates through the app, just like Chrome Dev MCP, and read logs, profiler info, bottlenecks, performance degraders, network calls, payloads, request/responses, headers and more and address visual/logical bugs perfectly.
I'm not fully sure how deep or available is this information already via the available tools, but what I'm sure is that auto-navigation plus screenshots is definitely missing and can add a ton of tangible value on the short and long-term especially that it increase the developer-surface for less savvy devs to dabble into Flutter since this is the one framework that does all of this out of the box.
Navigation leads to better automated testing, screenshots provide context, overlays provide deeper visual context regarding what's going visually, and profiler provides hardware-level performance improvements that can help boost a better development scaffolding right off the bat to avoid massive refactors for less savvy users.