Skip to content

vpython setup flaky failure on Engine Windows builds #88719

Closed

Description

@zanderso

A couple of instances of this in the past few days:

https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20Host%20Engine/11519/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20Unopt/1248/overview

[E2021-08-20T18:19:34.841758-07:00 5092 0 venv.go:987] Command (cwd=C:\b\s\w\ir\x\t\vpython_bootstrap475928925\packages\virtualenv-16.7.10): [C:\b\s\w\ir\cipd_bin_packages\cpython3\bin\python3.exe -B -E -s virtualenv.py --no-download C:\b\s\w\ir\cache\vpython\51ae26]
Process output:
Using base prefix 'C:\\b\\s\\w\\ir\\cipd_bin_packages\\cpython3\\bin'
Traceback (most recent call last):
  File "virtualenv.py", line 2642, in <module>
    main()
  File "virtualenv.py", line 859, in main
    create_environment(
  File "virtualenv.py", line 1161, in create_environment
    install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
  File "virtualenv.py", line 1450, in install_python
    copy_required_modules(home_dir, symlink)
  File "virtualenv.py", line 1342, in copy_required_modules
    copyfile(filename, dst_filename, symlink)
  File "virtualenv.py", line 425, in copyfile
    copy_file_or_folder(src, dest, symlink)
  File "virtualenv.py", line 400, in copy_file_or_folder
    shutil.copytree(src, dest, symlink)
  File "C:\b\s\w\ir\cipd_bin_packages\cpython3\bin\lib\shutil.py", line 557, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "C:\b\s\w\ir\cipd_bin_packages\cpython3\bin\lib\shutil.py", line 513, in _copytree
    raise Error(errors)
shutil.Error: [('C:\\b\\s\\w\\ir\\cipd_bin_packages\\cpython3\\bin\\lib\\encodings\\__pycache__\\utf_8.cpython-38.pyc.2398892502592', 'C:\\b\\s\\w\\ir\\cache\\vpython\\51ae26\\lib\\encodings\\__pycache__\\utf_8.cpython-38.pyc.2398892502592', "[Errno 2] No such file or directory: 'C:\\\\b\\\\s\\\\w\\\\ir\\\\cipd_bin_packages\\\\cpython3\\\\bin\\\\lib\\\\encodings\\\\__pycache__\\\\utf_8.cpython-38.pyc.2398892502592'")]

Activity

  1. added
    team-infraOwned by Infrastructure team
    P1High-priority issues at the top of the work list
    on Aug 23, 2021
  2. added and removed
    P1High-priority issues at the top of the work list
    on Aug 23, 2021
  3. keyonghan commented on Aug 23, 2021

    @keyonghan
    Contributor

    Issue started happening on 08/20 twice, and one more three hours ago. These three failures happened on different bots. Don't see any suspicious recipe changes either.
    Will keep debugging/monitoring.

  4. godofredoc commented on Aug 24, 2021

    @godofredoc
    Contributor

    There was a change to use vpython instead of python in the buildroot: flutter-team-archive/buildroot#501 which was rolled here: flutter-team-archive/engine#28226

  5. zanderso commented on Aug 24, 2021

    @zanderso
    MemberAuthor
  6. iskakaushik commented on Aug 25, 2021

    @iskakaushik
    Contributor

    The original change was needed so fuchsia gn rules can parse yaml files, it was recommended that we use vpython to fetch dependencies and also provide reproducible builds across all environments. I am not sure why its flaking on windows.

  7. godofredoc commented on Aug 25, 2021

    @godofredoc
    Contributor

    Seems like a vpython cache corruption problem:

    Using base prefix 'C:\\b\\s\\w\\ir\\cipd_bin_packages\\cpython3\\bin'
    Traceback (most recent call last):
      File "virtualenv.py", line 2642, in <module>
        main()
      File "virtualenv.py", line 859, in main
        create_environment(
      File "virtualenv.py", line 1161, in create_environment
        install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
      File "virtualenv.py", line 1450, in install_python
        copy_required_modules(home_dir, symlink)
      File "virtualenv.py", line 1342, in copy_required_modules
        copyfile(filename, dst_filename, symlink)
      File "virtualenv.py", line 425, in copyfile
        copy_file_or_folder(src, dest, symlink)
      File "virtualenv.py", line 400, in copy_file_or_folder
        shutil.copytree(src, dest, symlink)
      File "C:\b\s\w\ir\cipd_bin_packages\cpython3\bin\lib\shutil.py", line 557, in copytree
        return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
      File "C:\b\s\w\ir\cipd_bin_packages\cpython3\bin\lib\shutil.py", line 513, in _copytree
        raise Error(errors)
    shutil.Error: [('C:\\b\\s\\w\\ir\\cipd_bin_packages\\cpython3\\bin\\lib\\encodings\\__pycache__\\utf_8.cpython-38.pyc.2398892502592', 'C:\\b\\s\\w\\ir\\cache\\vpython\\51ae26\\lib\\encodings\\__pycache__\\utf_8.cpython-38.pyc.2398892502592', "[Errno 2] No such file or directory: 'C:\\\\b\\\\s\\\\w\\\\ir\\\\cipd_bin_packages\\\\cpython3\\\\bin\\\\lib\\\\encodings\\\\__pycache__\\\\utf_8.cpython-38.pyc.2398892502592'")]```
    
  8. added a commit that references this issue on Aug 25, 2021
  9. cbracken commented on Aug 26, 2021

    @cbracken
    Member

    Migration of the Windows gn wrapper to vpython3 is complete; if we haven't seen any more of these after a week, we'll close.

  10. cbracken commented on Aug 26, 2021

    @cbracken
    Member

    Unfortunately we did see further flakes after landing the gn.bat change:

    Instead I've landed the roll of the buildroot in flutter-team-archive/engine#28302, which necessitated landing flutter-team-archive/engine#28313, which introduced a dependency on the yaml module in the fuchsia build, and which isn't included in the CIPD package for python3.

  11. cbracken commented on Aug 26, 2021

    @cbracken
    Member

    I'll wait for ~20 commits, and if we don't see a recurrence of this flake, I'll close this.

    Further, I'll file two bugs:

    • Land improvements to vpython3 to avoid the cache-related issues we're seeing (File in Chromium's Infra>ThirdPartySoftware component).
    • Re-roll our python3 runtime back to vpython3 (File a Flutter issue for this)
  12. godofredoc commented on Sep 2, 2021

    @godofredoc
    Contributor

    @cbracken do you have by any chance the link to the chromium bug?

  13. zanderso commented on Nov 19, 2021

    @zanderso
    MemberAuthor

    @cbracken Can we close this?

  14. cbracken commented on Jan 4, 2022

    @cbracken
    Member

    I believe so. Thanks for the reminder.

  15. github-actions commented on Jan 18, 2022

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  16. locked as resolved and limited conversation to collaborators on Jan 18, 2022
  17. added
    P0Critical issues such as a build break or regression
    and removed on Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionteam-infraOwned by Infrastructure team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions