Skip to content

Conversation

@timmaffett
Copy link
Contributor

@timmaffett timmaffett commented Sep 17, 2025

Fixes #172899

This PR adds a format_version number to the root table of the 3 flat buffer formats that impellerc creates; runtime stages, shader bundles and shader archives.
It also adds a 'format_version' key to the root table of the shader bundle json format for completeness (Though it could be argued that the json format does not require this, I feel it may still be useful there).

This allows for format compatibility checks with future versions of the table as @chinmaygarde expressed concerns about in #168294 and @jonahwilliams mentions in #166944 .
The current code only checks for compatibility with the current version, but this mechanism allows for backwards compatibility in the future and not handling flat buffers with formats with new version number.
The big win here is that format_version numbers allow us to abort parsing anything that we may not understand that could possible cause problems, corruptions or crashes. The changes are simple and straightforward and I feel they lay a reasonable foundation for version handling for any future changes.

The version numbers themselves are defined within the runtime_stage_types.fbs, shader_bundle.fbs and shader_archive.fbs where I feel they should be as they can directly be incremented if any changes to the flat buffer format are made. Mechanisms for handling of backwards compatibility in the future should be straightforward to make in the corresponding c++ code allowing for the continued support of older (lower) version numbers if so desired.

This addresses #172899 and the other concerns mentioned in the issues/pr's mentioned above.

I plan on including my tests for exclusion of non-matching versions - Existing test verify that the version checking confirms matching versions. I wanted to get this out there for discussion before spending more time on this in case this approach is rejected outright.

I have a companion PR which adds FragmentProgram.fromBytes() to the engine (c++ and canvaskit/wasm). I wanted to separate the impellerc output versioning into it's own (this) PR.

Pre-launch Checklist

  • I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
  • I read the [Tree Hygiene] wiki page, which explains my responsibilities.
  • I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
  • I signed the [CLA].
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • [FORTHCOMING] I added new tests to check the change I am making, or this PR is [test-exempt].
  • I followed the [breaking change policy] and added [Data Driven Fixes] where supported.
  • All existing and new tests are passing.

@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests or get an explicit test exemption before merging.

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. If you believe this PR qualifies for a test exemption, contact "@test-exemption-reviewer" in the #hackers channel in Discord (don't just cc them here, they won't see it!). The test exemption team is a small volunteer group, so all reviewers should feel empowered to ask for tests, without delegating that responsibility entirely to the test exemption group.

@github-actions github-actions bot added engine flutter/engine related. See also e: labels. e: impeller Impeller rendering backend issues and features requests labels Sep 17, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a format_version to Impeller's FlatBuffer formats for runtime stages, shader bundles, and shader archives. This is a valuable addition for ensuring format compatibility and enabling safer deserialization in the future. The changes are consistently applied across the schemas, the serialization code, and the deserialization code, including version checks. My feedback focuses on improving the readability of the newly added version-checking logic.

timmaffett and others added 4 commits September 16, 2025 21:26
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@gaaclarke gaaclarke changed the title Adds Impellerc flatbuffer format versioning. Fixes #172899 Adds Impellerc flatbuffer format versioning. Sep 26, 2025
Copy link
Member

@gaaclarke gaaclarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good to me. I'd like to see tests that cover parsing as well as serializing the version number. I also wonder if relying on VALIDATION_LOG is the right thing. Wouldn't we want to surface these errors to users in the form on dart exceptions?

@gaaclarke
Copy link
Member

I also wonder if relying on VALIDATION_LOG is the right thing. Wouldn't we want to surface these errors to users in the form on dart exceptions?

I've filed an issue for this so we don't have to fix it here ( #176593 ). This PR is just following the example set from the other errors.

@gaaclarke
Copy link
Member

@timmaffett I checked with the rest of the team and they are happy with this approach. It looks like you addressed all the other feedback except the one that asks for a test that parses version, preferably in the negative case. I didn't hear back from you after you addressed the other feedback; is that something you are planning to look into or could use some more guidance on? Let me know where your head is at.

@timmaffett
Copy link
Contributor Author

Yes, I am planning on adding the negative test - I just got busy with work. I was planning on adding a negative parsing case as all of the existing test confirm the positive (successful parsing).
I researched 'surfacing' the parsing errors out to dart in addition to relying on the VALIDATION_LOG log. I some non-exhaustive searching I could not find any examples of this in the code that I looked through (of surfacing an error from C++ to dart as an exception) - do you have any suggestions here ? I see that you filed #176593 and these can all be addressed at the same time.
I will try to get to this ASAP.

ps. I was also hoping to get some feedback on #175479 if possible ;)

@flutter-dashboard
Copy link

This pull request executed golden file tests, but it has not been updated in a while (20+ days). Test results from Gold expire after as many days, so this pull request will need to be updated with a fresh commit in order to get results from Gold.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@timmaffett
Copy link
Contributor Author

timmaffett commented Dec 2, 2025

@gaaclarke Thank you for your work on this!

I have been so busy with work that I had not been able to give this the attention that was needed.

@gaaclarke
Copy link
Member

@timmaffett, you know how we do =)

Copy link
Member

@gaaclarke gaaclarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@gaaclarke
Copy link
Member

I ended up implementing a bit of #176593 since once you start using absl::StatusOr it gets easier to start bubbling up errors.

auto blob_library = ShaderArchive{std::move(library)};
if (!blob_library.IsValid()) {
auto blob_library = ShaderArchive::Create(std::move(library));
if (!blob_library.ok()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably be reporting the status here, but we already have a PR open to go back and surface these to dart.

@gaaclarke gaaclarke added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 2, 2025
@auto-submit
Copy link
Contributor

auto-submit bot commented Dec 2, 2025

autosubmit label was removed for flutter/flutter/175470, because - The status or check suite Mac mac_unopt has failed. Please fix the issues identified (or deflake) before re-applying this label.

@auto-submit auto-submit bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Dec 2, 2025
@gaaclarke gaaclarke added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 2, 2025
@gaaclarke gaaclarke mentioned this pull request Dec 2, 2025
9 tasks
@auto-submit auto-submit bot added this pull request to the merge queue Dec 2, 2025
Merged via the queue into flutter:master with commit 2bd9ecb Dec 2, 2025
184 checks passed
@flutter-dashboard flutter-dashboard bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Dec 2, 2025
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 3, 2025
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 3, 2025
auto-submit bot pushed a commit to flutter/packages that referenced this pull request Dec 3, 2025
flutter/flutter@5545bb3...e274574

2025-12-03 [email protected] Roll Skia from 20829e37dfb8 to db4c79d41513 (1 revision) (flutter/flutter#179401)
2025-12-03 [email protected] Roll Skia from adc7ea94cada to 20829e37dfb8 (6 revisions) (flutter/flutter#179385)
2025-12-03 [email protected] Refactor GetShaderClipDepth for clarity (flutter/flutter#179110)
2025-12-03 [email protected] Roll Skia from 3b339a83959b to adc7ea94cada (1 revision) (flutter/flutter#179376)
2025-12-03 [email protected] Roll Dart SDK from eb743a1d4ade to 0bb365d7ac74 (7 revisions) (flutter/flutter#179372)
2025-12-03 [email protected] feat: Add `mainAxisExtent` parameter to `GridView` constructors (flutter/flutter#176927)
2025-12-03 [email protected] Roll Skia from eb01fff20df8 to 3b339a83959b (4 revisions) (flutter/flutter#179371)
2025-12-02 [email protected] Fix crash when text editing value changes between scrolls (flutter/flutter#179163)
2025-12-02 [email protected] Roll Skia from 6bd3b06b1e08 to eb01fff20df8 (3 revisions) (flutter/flutter#179362)
2025-12-02 [email protected] Adds Impellerc flatbuffer format versioning. (flutter/flutter#175470)
2025-12-02 [email protected] Adds format argument to Picture.toImageSync (flutter/flutter#178691)
2025-12-02 [email protected] Delete disabled workflow and add missing permissions key to workflow (flutter/flutter#178911)
2025-12-02 [email protected] [web] Fix some gn warnings (flutter/flutter#178313)
2025-12-02 [email protected] Roll Skia from 45337c4e919d to 6bd3b06b1e08 (4 revisions) (flutter/flutter#179353)
2025-12-02 [email protected] [ios] Reland Dynamic Content Resizing (flutter/flutter#179153)
2025-12-02 [email protected] [web] Fix onTextScaleFactorChanged not getting called. (flutter/flutter#178862)
2025-12-02 [email protected] Roll Packages from c8be05d to 148dcd2 (9 revisions) (flutter/flutter#179343)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
github-merge-queue bot pushed a commit that referenced this pull request Dec 5, 2025
fixes #176593
That issue was really fixed in
#175470 but this is extra cleanup
for error handling with shaders, including better error messages when
loading them in tests.

changes:
- Removes RuntimeStage::IsValid to statically assert that we only ever
have valid runtime stages
- Finally introduces ASSERT_OK which should have been added long ago

This doesn't propagate up shaderlibrary errors since that's flutter_gpu.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
reidbaker pushed a commit to AbdeMohlbi/flutter that referenced this pull request Dec 10, 2025
Fixes flutter#172899

This PR adds a format_version number to the root table of the 3 flat
buffer formats that impellerc creates; runtime stages, shader bundles
and shader archives.
It also adds a 'format_version' key to the root table of the shader
bundle json format for completeness (Though it could be argued that the
json format does not require this, I feel it may still be useful there).

This allows for format compatibility checks with future versions of the
table as @chinmaygarde expressed concerns about in flutter#168294 and
@jonahwilliams mentions in flutter#166944 .
The current code only checks for compatibility with the current version,
but this mechanism allows for backwards compatibility in the future and
not handling flat buffers with formats with new version number.
The big win here is that format_version numbers allow us to abort
parsing anything that we may not understand that could possible cause
problems, corruptions or crashes. The changes are simple and
straightforward and I feel they lay a reasonable foundation for version
handling for any future changes.

The version numbers themselves are defined within the
`runtime_stage_types.fbs`, `shader_bundle.fbs` and `shader_archive.fbs`
where I feel they should be as they can directly be incremented if any
changes to the flat buffer format are made. Mechanisms for handling of
backwards compatibility in the future should be straightforward to make
in the corresponding c++ code allowing for the continued support of
older (lower) version numbers if so desired.

This addresses flutter#172899 and the other concerns mentioned in the
issues/pr's mentioned above.

I plan on including my tests for exclusion of non-matching versions -
Existing test verify that the version checking confirms matching
versions. I wanted to get this out there for discussion before spending
more time on this in case this approach is rejected outright.
   
I have a companion PR which adds FragmentProgram.fromBytes() to the
engine (c++ and canvaskit/wasm). I wanted to separate the impellerc
output versioning into it's own (this) PR.

  
## Pre-launch Checklist

- [X] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [X] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [X] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [X] I signed the [CLA].
- [X] I listed at least one issue that this PR fixes in the description
above.
- [X] I updated/added relevant documentation (doc comments with `///`).
- [FORTHCOMING] I added new tests to check the change I am making, or
this PR is [test-exempt].
- [X] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [X] All existing and new tests are passing.

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gaaclarke <[email protected]>
Co-authored-by: Aaron Clarke <[email protected]>
reidbaker pushed a commit to AbdeMohlbi/flutter that referenced this pull request Dec 10, 2025
fixes flutter#176593
That issue was really fixed in
flutter#175470 but this is extra cleanup
for error handling with shaders, including better error messages when
loading them in tests.

changes:
- Removes RuntimeStage::IsValid to statically assert that we only ever
have valid runtime stages
- Finally introduces ASSERT_OK which should have been added long ago

This doesn't propagate up shaderlibrary errors since that's flutter_gpu.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

e: impeller Impeller rendering backend issues and features requests engine flutter/engine related. See also e: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add versioning and validation metadata to Impeller shader blobs to ensure forward compatibility

3 participants