You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Flutter engine git push is broken on Windows #108122
The pre-commit git hook does not find depot_tool's python3.bat file, causing git push to fail on Windows unless python3 is installed elsewhere. You may not run into this issue if you've installed the "real" python3 SDK on your machine.
Error:
> git push -u origin HEAD
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
error: failed to push some refs to 'github.com:loic-sharma/flutter-engine.git'
Solutions
In the short-term, we will revert the PR that caused this regression. In the long-term, potential solutions include:
Create a separate pre-push git hook entry point for Windows, or...
Rewrite the pre-push git hook entry point to use pure bash instead of python3, or...
Add a step to Flutter engine's setup guide to add /path/to/depot_tools/python-bin to your path
I'm leaning towards option 1 as that's the prior art.
I didn't have to disable any app execution aliases. Even if you do disable it, you'll get a different error:
> git push -u origin HEAD
/usr/bin/env: ‘python3’: No such file or directory
error: failed to push some refs to 'github.com:loic-sharma/flutter-engine.git'
First of all, sorry about the issues this change caused. I see that the team chose to revert this to address the issue quickly. As far as I can see, the issue wasn't with the code, but with my communication skills. My understanding was that you have to have python pre-installed on your system in order to do any engine development. I misinterpreted this message on Discord. Again, my bad.
Turns out python is not required (or there is an assumption that depot_tools' python should be discoverable somehow?). It's not an issue on macOS and most Linux distros as Python is pre-installed on these systems. Windows however is not so lucky. Problem is, the depot_tools is not necessarily on a standard path nor can be queried easily as far as I can see. Please correct me if I'm wrong.
It would be great if we could re-land this patch, so I'd like some guidance on how to approach this @chinmaygarde@loic-sharma. Should I re-implement the pre-push file on all platforms as a bash script? It seems bash is bundled with git on Windows and available on macOS and most if not all Linux distros. Or should python be a prerequisite? Should I try to discover depot_tools' python3.bat (I don't recommend going this way).
I'm happy to go either way so we can re-enable Windows as a full-fledged development platform. Thanks!
Hello, no worries at all! We very much appreciate your contributions! These were subtle issues so it's completely understandable. In the future, we should make sure that reviewers test the git hooks on their own environments to try to catch these kinds of problems sooner. This was a good learning experience! 😄
or there is an assumption that depot_tools' python should be discoverable somehow?
This is correct. If you check the setup guide, it tells you to put depot_tools in your path. Also the depot_tools guide mentions that depot_tools must be on the front of your PATH, ahead of any Python installations.
Interesting -- I had no idea that shebangs were supported on Windows, but looks like they are, though it's unclear to me whether this requires some pre-configuration on the part of the user. If so, we should clearly document that in the wiki.
Hello, no worries at all! We very much appreciate your contributions! These were subtle issues so it's completely understandable. In the future, we should make sure that reviewers test the git hooks on their own environments to try to catch these kinds of problems sooner. This was a good learning experience! 😄
Oh I agree, the more people can test this, the better. However I'm new to the Flutter Engine community and don't know how to facilitate such a drive. Any help is greatly appreciated.
This is correct. If you check the setup guide, it tells you to put depot_tools in your path.
According to that setup guide python is a separate prerequisite, and at the depot_tools dependency line it does not mention python3.bat at all, just gclient.
Also the depot_tools guide mentions that depot_tools must be on the front of your PATH, ahead of any Python installations.
Honestly, as a developer I'm not comfortable with this, so I wanna make absolutely sure whether the Flutter team wants to go down this route. This means that python, git, gn, ninja etc. will be forced to the version depot_tools bundles if you just type in those commands at a console. Which is going to trip you up down the road on completely unrelated projects. It's not in the Setting up the Engine development wiki either. Based on these I don't think we can count on people adhering to this requirement. Just by installing depot_tools with the Scoop package manager nets you a single gclient on your PATH, no other tools.
It uses python3.bat for the shebang: #!/usr/bin/env python3.bat
If depot_tools should be ahead of any other python installation, then why specify explicitly python3.bat? #!/usr/bin/env python3 will automatically resolve to depot_tools' python3.bat. And if it can be #!/usr/bin/env python3 then it's the exact same as in the non-Windows pre-push script, so why even have a separate Windows pre-push? I recommend using the non-Windows version pre-push on Windows as well (just like in my original patch) and replacing the warning in the Windows version to run gclient sync -D to enable pre-push hooks on Windows instead of deleting it (as I foolishly did).
It uses the correct path to discover the prepush Dart script
Add comments to all pre-push scripts that they should be kept in sync
I believe this may be the cleanest solution. What do you think? /cc @cbracken@yaakovschectman for feedback
If we can avoid having a separate platform exception, I think it should be avoided. Duplicated code is always harder to maintain.
Honestly, as a developer I'm not comfortable with this, so I wanna make absolutely sure whether the Flutter team wants to go down this route. This means that python, git, gn, ninja etc. will be forced to the version depot_tools bundles if you just type in those commands at a console.
One of the most important features for a large software project is consistent, reproducible builds. To do that, we go to lengths to ensure that no matter what machine you're on, we're building with the same toolchain and environment. For example, if you have clang installed, we don't use your version of clang, we download and use our own clang toolchain. Similarly, we don't want to deal with the dozens of potential versions of python developers may have installed and the inconsistencies/bugs etc. that may arise from such a setup, we want to use a single, consistent version, which is why @a-wallen recently landed a patch to use vpython3 (part of depot_tools) for all exec_script() calls during the build.
Bazel (which is the open-source version of Google's internal build tool, blaze) has a nice page discussing the advantages of hermetic builds and why Google spends so much effort on making builds hermetic.
Regarding the concern with putting depot_tools first (or at least early) in your PATH, I'm with you on this; requiring it to be early indicates that we're not being as hermetic as we should be, and that we're pulling python off the PATH somewhere rather than using a specific version. Rather than ask people to put depot_tools at the top of their path, we should fix those cases.
With regards to Python specifically, I'd suggest that rather than going to the effort of us creating and maintaining our own CIPD package for python3 on every platform like we do with clang, we should do what Chromium does and compromise by using vpython3 off the PATH in cases like gn.bat. It's unlikely that most developers have any other binary named vpython3 on their system that will conflict.
Either way, we should figure out why we're currently recommending putting depot_tools early on the path and file a bug so we can change the recommendation to simply putting depot_tools (anywhere) on the path.
Based on these I don't think we can count on people adhering to this requirement.
This is probably the best argument for not requiring it. For what it's worth, I don't have depot_tools at the front of my path, though I also don't have anything likely to conflict with it on my Windows machine; on my Mac and Linux machines I do though, and it hasn't been a problem.
Reacted by Loïc Sharma, Alex Wallen and Márk Tolmács
Thanks for the detailed answer. If confusion around tooling versions is not a big issue for Flutter, then this is a good compromise from an individual dev's perspective. Short term some people will now realize they didn't follow the Setup Guide closely (the section related to PATH and depot_tools specifically), but that's unavoidable.
Otherwise (or if having hermetic local builds is very important), a standardized environment via a bootstrap script can be a good alternative. A canary env variable can be set by this script so tooling can detect if not running in the right context.
I'll move the affected python scripts in my PR to vpython3, so this topic can be resolved in the context of this particular issue.
Either way, we should figure out why we're currently recommending putting depot_tools early on the path and file a bug so we can change the recommendation to simply putting depot_tools (anywhere) on the path.
It does actually look like there's a CIPD package for python3 -- I can see it used here on our bots in the following build config, for example -- look for cipd_bin_packages/cpython3:
That might also be an option, but I haven't given it a look (or much thought) and not sure what exactly Chromium does here or why I see a mix of vpython3 and python3 in their scripts.
Otherwise (or if having hermetic local builds is very important), a standardized environment via a bootstrap script can be a good alternative.
Yeah, this is pretty common in a lot of projects and definitely an option. This is effectively how our bots operate.
If we can make the build more resilient to conflicting python/etc. on the path and remove the depot_tools path positioning wording from the wiki, I think we're probably in "good enough" territory. Fingers crossed.
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.
Activity
chinmaygarde commentedon Jul 21, 2022
Isn't disabling the app execution alias a pre-requisite for engine development? I couldn't do gclient sync without it.
loic-sharma commentedon Jul 21, 2022
I didn't have to disable any app execution aliases. Even if you do disable it, you'll get a different error:
mtolmacs commentedon Jul 25, 2022
First of all, sorry about the issues this change caused. I see that the team chose to revert this to address the issue quickly. As far as I can see, the issue wasn't with the code, but with my communication skills. My understanding was that you have to have python pre-installed on your system in order to do any engine development. I misinterpreted this message on Discord. Again, my bad.
Turns out python is not required (or there is an assumption that depot_tools' python should be discoverable somehow?). It's not an issue on macOS and most Linux distros as Python is pre-installed on these systems. Windows however is not so lucky. Problem is, the depot_tools is not necessarily on a standard path nor can be queried easily as far as I can see. Please correct me if I'm wrong.
It would be great if we could re-land this patch, so I'd like some guidance on how to approach this @chinmaygarde @loic-sharma. Should I re-implement the pre-push file on all platforms as a bash script? It seems bash is bundled with git on Windows and available on macOS and most if not all Linux distros. Or should python be a prerequisite? Should I try to discover depot_tools' python3.bat (I don't recommend going this way).
I'm happy to go either way so we can re-enable Windows as a full-fledged development platform. Thanks!
loic-sharma commentedon Jul 25, 2022
Hello, no worries at all! We very much appreciate your contributions! These were subtle issues so it's completely understandable. In the future, we should make sure that reviewers test the git hooks on their own environments to try to catch these kinds of problems sooner. This was a good learning experience! 😄
This is correct. If you check the setup guide, it tells you to put
depot_toolsin your path. Also thedepot_toolsguide mentions thatdepot_toolsmust be on the front of your PATH, ahead of any Python installations.I think the best solution would be to update the prepush Python script for Windows to be a copy of the non-Windows prepush python script, except:
python3.batfor the shebang:#!/usr/bin/env python3.batI believe this may be the cleanest solution. What do you think? /cc @cbracken @yaakovschectman for feedback
cbracken commentedon Jul 25, 2022
Interesting -- I had no idea that shebangs were supported on Windows, but looks like they are, though it's unclear to me whether this requires some pre-configuration on the part of the user. If so, we should clearly document that in the wiki.
loic-sharma commentedon Jul 25, 2022
AFAIK shebangs aren't supported on powershell or cmd, this only works because Git uses bash emulation for its hooks on Windows.
mtolmacs commentedon Jul 26, 2022
Oh I agree, the more people can test this, the better. However I'm new to the Flutter Engine community and don't know how to facilitate such a drive. Any help is greatly appreciated.
According to that setup guide python is a separate prerequisite, and at the depot_tools dependency line it does not mention
python3.batat all, justgclient.Honestly, as a developer I'm not comfortable with this, so I wanna make absolutely sure whether the Flutter team wants to go down this route. This means that
python,git,gn,ninjaetc. will be forced to the version depot_tools bundles if you just type in those commands at a console. Which is going to trip you up down the road on completely unrelated projects. It's not in the Setting up the Engine development wiki either. Based on these I don't think we can count on people adhering to this requirement. Just by installing depot_tools with the Scoop package manager nets you a singlegclienton your PATH, no other tools.If depot_tools should be ahead of any other python installation, then why specify explicitly python3.bat?
#!/usr/bin/env python3will automatically resolve to depot_tools' python3.bat. And if it can be#!/usr/bin/env python3then it's the exact same as in the non-Windows pre-push script, so why even have a separate Windows pre-push? I recommend using the non-Windows version pre-push on Windows as well (just like in my original patch) and replacing the warning in the Windows version to rungclient sync -Dto enable pre-push hooks on Windows instead of deleting it (as I foolishly did).If we can avoid having a separate platform exception, I think it should be avoided. Duplicated code is always harder to maintain.
cbracken commentedon Jul 27, 2022
One of the most important features for a large software project is consistent, reproducible builds. To do that, we go to lengths to ensure that no matter what machine you're on, we're building with the same toolchain and environment. For example, if you have clang installed, we don't use your version of clang, we download and use our own clang toolchain. Similarly, we don't want to deal with the dozens of potential versions of python developers may have installed and the inconsistencies/bugs etc. that may arise from such a setup, we want to use a single, consistent version, which is why @a-wallen recently landed a patch to use
vpython3(part of depot_tools) for allexec_script()calls during the build.Bazel (which is the open-source version of Google's internal build tool, blaze) has a nice page discussing the advantages of hermetic builds and why Google spends so much effort on making builds hermetic.
Regarding the concern with putting
depot_toolsfirst (or at least early) in yourPATH, I'm with you on this; requiring it to be early indicates that we're not being as hermetic as we should be, and that we're pulling python off thePATHsomewhere rather than using a specific version. Rather than ask people to putdepot_toolsat the top of their path, we should fix those cases.With regards to Python specifically, I'd suggest that rather than going to the effort of us creating and maintaining our own CIPD package for python3 on every platform like we do with clang, we should do what Chromium does and compromise by using
vpython3off thePATHin cases like gn.bat. It's unlikely that most developers have any other binary namedvpython3on their system that will conflict.Either way, we should figure out why we're currently recommending putting depot_tools early on the path and file a bug so we can change the recommendation to simply putting depot_tools (anywhere) on the path.
This is probably the best argument for not requiring it. For what it's worth, I don't have depot_tools at the front of my path, though I also don't have anything likely to conflict with it on my Windows machine; on my Mac and Linux machines I do though, and it hasn't been a problem.
mtolmacs commentedon Jul 28, 2022
Thanks for the detailed answer. If confusion around tooling versions is not a big issue for Flutter, then this is a good compromise from an individual dev's perspective. Short term some people will now realize they didn't follow the Setup Guide closely (the section related to PATH and depot_tools specifically), but that's unavoidable.
Otherwise (or if having hermetic local builds is very important), a standardized environment via a bootstrap script can be a good alternative. A canary env variable can be set by this script so tooling can detect if not running in the right context.
I'll move the affected python scripts in my PR to vpython3, so this topic can be resolved in the context of this particular issue.
mtolmacs commentedon Jul 28, 2022
Oh and even gclient does not put the depot_tools dir in front of the PATH, so this might be a stale documentation issue?
cbracken commentedon Jul 28, 2022
Sounds good -- @a-wallen just sent out a couple PRs that help move this forward:
It does actually look like there's a CIPD package for python3 -- I can see it used here on our bots in the following build config, for example -- look for
cipd_bin_packages/cpython3:That might also be an option, but I haven't given it a look (or much thought) and not sure what exactly Chromium does here or why I see a mix of vpython3 and python3 in their scripts.
Yeah, this is pretty common in a lot of projects and definitely an option. This is effectively how our bots operate.
If we can make the build more resilient to conflicting python/etc. on the path and remove the depot_tools path positioning wording from the wiki, I think we're probably in "good enough" territory. Fingers crossed.
loic-sharma commentedon Aug 11, 2022
Closing this as
git pushworks again on Windows.github-actions commentedon Aug 25, 2022
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.