-
Notifications
You must be signed in to change notification settings - Fork 29.7k
Adds Impellerc flatbuffer format versioning. #175470
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds Impellerc flatbuffer format versioning. #175470
Conversation
|
It looks like this pull request may not have tests. Please make sure to add tests or get an explicit test exemption before merging. If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix? Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. If you believe this PR qualifies for a test exemption, contact "@test-exemption-reviewer" in the #hackers channel in Discord (don't just cc them here, they won't see it!). The test exemption team is a small volunteer group, so all reviewers should feel empowered to ask for tests, without delegating that responsibility entirely to the test exemption group. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces a format_version to Impeller's FlatBuffer formats for runtime stages, shader bundles, and shader archives. This is a valuable addition for ensuring format compatibility and enabling safer deserialization in the future. The changes are consistently applied across the schemas, the serialization code, and the deserialization code, including version checks. My feedback focuses on improving the readability of the newly added version-checking logic.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
gaaclarke
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good to me. I'd like to see tests that cover parsing as well as serializing the version number. I also wonder if relying on VALIDATION_LOG is the right thing. Wouldn't we want to surface these errors to users in the form on dart exceptions?
I've filed an issue for this so we don't have to fix it here ( #176593 ). This PR is just following the example set from the other errors. |
|
@timmaffett I checked with the rest of the team and they are happy with this approach. It looks like you addressed all the other feedback except the one that asks for a test that parses version, preferably in the negative case. I didn't hear back from you after you addressed the other feedback; is that something you are planning to look into or could use some more guidance on? Let me know where your head is at. |
|
Yes, I am planning on adding the negative test - I just got busy with work. I was planning on adding a negative parsing case as all of the existing test confirm the positive (successful parsing). ps. I was also hoping to get some feedback on #175479 if possible ;) |
|
This pull request executed golden file tests, but it has not been updated in a while (20+ days). Test results from Gold expire after as many days, so this pull request will need to be updated with a fresh commit in order to get results from Gold. For more guidance, visit Writing a golden file test for Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. |
|
@gaaclarke Thank you for your work on this! I have been so busy with work that I had not been able to give this the attention that was needed. |
|
@timmaffett, you know how we do =) |
gaaclarke
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm!
|
I ended up implementing a bit of #176593 since once you start using absl::StatusOr it gets easier to start bubbling up errors. |
| auto blob_library = ShaderArchive{std::move(library)}; | ||
| if (!blob_library.IsValid()) { | ||
| auto blob_library = ShaderArchive::Create(std::move(library)); | ||
| if (!blob_library.ok()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably be reporting the status here, but we already have a PR open to go back and surface these to dart.
|
autosubmit label was removed for flutter/flutter/175470, because - The status or check suite Mac mac_unopt has failed. Please fix the issues identified (or deflake) before re-applying this label. |
flutter/flutter@5545bb3...e274574 2025-12-03 [email protected] Roll Skia from 20829e37dfb8 to db4c79d41513 (1 revision) (flutter/flutter#179401) 2025-12-03 [email protected] Roll Skia from adc7ea94cada to 20829e37dfb8 (6 revisions) (flutter/flutter#179385) 2025-12-03 [email protected] Refactor GetShaderClipDepth for clarity (flutter/flutter#179110) 2025-12-03 [email protected] Roll Skia from 3b339a83959b to adc7ea94cada (1 revision) (flutter/flutter#179376) 2025-12-03 [email protected] Roll Dart SDK from eb743a1d4ade to 0bb365d7ac74 (7 revisions) (flutter/flutter#179372) 2025-12-03 [email protected] feat: Add `mainAxisExtent` parameter to `GridView` constructors (flutter/flutter#176927) 2025-12-03 [email protected] Roll Skia from eb01fff20df8 to 3b339a83959b (4 revisions) (flutter/flutter#179371) 2025-12-02 [email protected] Fix crash when text editing value changes between scrolls (flutter/flutter#179163) 2025-12-02 [email protected] Roll Skia from 6bd3b06b1e08 to eb01fff20df8 (3 revisions) (flutter/flutter#179362) 2025-12-02 [email protected] Adds Impellerc flatbuffer format versioning. (flutter/flutter#175470) 2025-12-02 [email protected] Adds format argument to Picture.toImageSync (flutter/flutter#178691) 2025-12-02 [email protected] Delete disabled workflow and add missing permissions key to workflow (flutter/flutter#178911) 2025-12-02 [email protected] [web] Fix some gn warnings (flutter/flutter#178313) 2025-12-02 [email protected] Roll Skia from 45337c4e919d to 6bd3b06b1e08 (4 revisions) (flutter/flutter#179353) 2025-12-02 [email protected] [ios] Reland Dynamic Content Resizing (flutter/flutter#179153) 2025-12-02 [email protected] [web] Fix onTextScaleFactorChanged not getting called. (flutter/flutter#178862) 2025-12-02 [email protected] Roll Packages from c8be05d to 148dcd2 (9 revisions) (flutter/flutter#179343) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
fixes #176593 That issue was really fixed in #175470 but this is extra cleanup for error handling with shaders, including better error messages when loading them in tests. changes: - Removes RuntimeStage::IsValid to statically assert that we only ever have valid runtime stages - Finally introduces ASSERT_OK which should have been added long ago This doesn't propagate up shaderlibrary errors since that's flutter_gpu. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Fixes flutter#172899 This PR adds a format_version number to the root table of the 3 flat buffer formats that impellerc creates; runtime stages, shader bundles and shader archives. It also adds a 'format_version' key to the root table of the shader bundle json format for completeness (Though it could be argued that the json format does not require this, I feel it may still be useful there). This allows for format compatibility checks with future versions of the table as @chinmaygarde expressed concerns about in flutter#168294 and @jonahwilliams mentions in flutter#166944 . The current code only checks for compatibility with the current version, but this mechanism allows for backwards compatibility in the future and not handling flat buffers with formats with new version number. The big win here is that format_version numbers allow us to abort parsing anything that we may not understand that could possible cause problems, corruptions or crashes. The changes are simple and straightforward and I feel they lay a reasonable foundation for version handling for any future changes. The version numbers themselves are defined within the `runtime_stage_types.fbs`, `shader_bundle.fbs` and `shader_archive.fbs` where I feel they should be as they can directly be incremented if any changes to the flat buffer format are made. Mechanisms for handling of backwards compatibility in the future should be straightforward to make in the corresponding c++ code allowing for the continued support of older (lower) version numbers if so desired. This addresses flutter#172899 and the other concerns mentioned in the issues/pr's mentioned above. I plan on including my tests for exclusion of non-matching versions - Existing test verify that the version checking confirms matching versions. I wanted to get this out there for discussion before spending more time on this in case this approach is rejected outright. I have a companion PR which adds FragmentProgram.fromBytes() to the engine (c++ and canvaskit/wasm). I wanted to separate the impellerc output versioning into it's own (this) PR. ## Pre-launch Checklist - [X] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [X] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [X] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [X] I signed the [CLA]. - [X] I listed at least one issue that this PR fixes in the description above. - [X] I updated/added relevant documentation (doc comments with `///`). - [FORTHCOMING] I added new tests to check the change I am making, or this PR is [test-exempt]. - [X] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [X] All existing and new tests are passing. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: gaaclarke <[email protected]> Co-authored-by: Aaron Clarke <[email protected]>
fixes flutter#176593 That issue was really fixed in flutter#175470 but this is extra cleanup for error handling with shaders, including better error messages when loading them in tests. changes: - Removes RuntimeStage::IsValid to statically assert that we only ever have valid runtime stages - Finally introduces ASSERT_OK which should have been added long ago This doesn't propagate up shaderlibrary errors since that's flutter_gpu. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Fixes #172899
This PR adds a format_version number to the root table of the 3 flat buffer formats that impellerc creates; runtime stages, shader bundles and shader archives.
It also adds a 'format_version' key to the root table of the shader bundle json format for completeness (Though it could be argued that the json format does not require this, I feel it may still be useful there).
This allows for format compatibility checks with future versions of the table as @chinmaygarde expressed concerns about in #168294 and @jonahwilliams mentions in #166944 .
The current code only checks for compatibility with the current version, but this mechanism allows for backwards compatibility in the future and not handling flat buffers with formats with new version number.
The big win here is that format_version numbers allow us to abort parsing anything that we may not understand that could possible cause problems, corruptions or crashes. The changes are simple and straightforward and I feel they lay a reasonable foundation for version handling for any future changes.
The version numbers themselves are defined within the
runtime_stage_types.fbs,shader_bundle.fbsandshader_archive.fbswhere I feel they should be as they can directly be incremented if any changes to the flat buffer format are made. Mechanisms for handling of backwards compatibility in the future should be straightforward to make in the corresponding c++ code allowing for the continued support of older (lower) version numbers if so desired.This addresses #172899 and the other concerns mentioned in the issues/pr's mentioned above.
I plan on including my tests for exclusion of non-matching versions - Existing test verify that the version checking confirms matching versions. I wanted to get this out there for discussion before spending more time on this in case this approach is rejected outright.
I have a companion PR which adds FragmentProgram.fromBytes() to the engine (c++ and canvaskit/wasm). I wanted to separate the impellerc output versioning into it's own (this) PR.
Pre-launch Checklist
///).