Upgrade yarn to latest #9989

kmcgrady · 2024-12-10T00:40:50Z

Describe your changes

Upgrade yarn to yarn berry. In order to do this we had to do the following:

Support yarn licenses in our checker
Install packages in both directories and run scripts from the specific directory
Fixes to enable yarn in GitHub Actions and whatnot.

Testing Plan

Tests should pass

Contribution License Agreement

By submitting this pull request you agree that all contributions to this project are made under the Apache 2.0 license.

kmcgrady · 2024-12-11T04:52:11Z

.github/actions/make_init/action.yml

      uses: actions/setup-node@v4
      with:
        node-version-file: ".nvmrc"
-        cache: "yarn"


Assuming approval works, I will re-enable this cache and clear the caches for actions so that this can replace it and merge. It will be a temporary blocker for people, but one that is easy to fix.

raethlein · 2024-12-11T15:15:13Z

.github/workflows/component-template-e2e-tests.yml

-          streamlit_wheel=$(readlink -e "${streamlit_wheel}")
-          echo "output_file=${streamlit_wheel}" >> $GITHUB_OUTPUT
-
+      # Build the component lib first so we can use the old version of yarn


Out of curiosity, do you know what's the delta for using the new yarn version also in the component-lib module? If it is not big, I think it'd be worth it to do the update simultaneously instead of fragmenting the versions

I'll play a little more today, but I had issues with resolution of dependencies and type checking. Will do a little more exploration today to see if it's possible (Really, we should put this in the monorepo so it can benefit from further optimization from our systems).

I got it working, so I upgraded it. Same comment about the cache. I need to break the caches and merge.

raethlein · 2024-12-11T15:17:08Z

.pre-commit-config.yaml

        # NOTE: This hook currently does not work on Windows due to "yarn" not being an executable and win32api.CreateProcess
        # turning `subprocess.run(["yarn", "prettier", "--write"])` into a call to `yarn.exe prettier --write` which does not exist
-        entry: ./scripts/run_in_subdirectory.py frontend yarn prettier ../.github --write
+        entry: ./scripts/run_in_subdirectory.py frontend/app yarn prettier ../../.github --write


Why is this change here needed?

Yarn is strict on not using scripts in folders that don't have them in dependencies. The frontend directory does not have prettier, but both lib and app do. Since we are applying these on the .github folder (in neither, I just chose one to leverage its prettier).

Future work would be to break out the eslint/prettier logic into another package for reuse.

raethlein · 2024-12-11T15:19:48Z

Makefile

 # Build frontend into static files faster by setting BUILD_AS_FAST_AS_POSSIBLE=true flag, which disables eslint and typechecking.
 frontend-fast:
-	cd frontend/ ; yarn run buildFast
+	cd frontend/ ; yarn workspace @streamlit/app buildFast


Could you please elaborate on why this is now only running for the app workspace?

It actually does it already. I just simplified it to clean up the scripts in the main frontend package.json https://github.com/streamlit/streamlit/blob/develop/frontend/package.json#L13

raethlein · 2024-12-11T15:23:25Z

scripts/update_version.py

    update_files(NODE_ROOT, semver_version)
    update_files(NODE_APP, semver_version)
    update_files(NODE_LIB, semver_version)
-    update_files(NODE_APP_ST_LIB, semver_version)


Why is this not needed anymore in the update_version.py script? Also, it looks like we can remove the NODE_APP_ST_LIB variable

Yarn workspaces reference each other, so this will now references the library in the workspace https://github.com/streamlit/streamlit/blob/refactor/upgrade-yarn/frontend/app/package.json#L30
which does not change.

scripts/audit_frontend_licenses.py

lukasmasuch · 2025-01-03T00:00:08Z

NOTICES

There seems to be a lot of changes in the NOTICES file (~5k additions, 18k deletions). Do you know what causes this big difference?

Ah the biggest difference was that we didn't include transitive dependencies. But I got it working! The numbers should be more reasonable.

Now it seems that the NOTICES file mostly has deletions (8k lines). Have you checked a couple deleted samples if these are expected?

I'll take a look. Likely it's due to the fact that Vite (and associated libraries) has been moved dev dependencies and that removes items from the NOTICES. But I'll double check.

lukasmasuch · 2025-01-03T00:07:05Z

frontend/package.json

-    "build": "yarn workspace @streamlit/lib build && yarn workspace @streamlit/app build",
-    "buildFast": "yarn workspace @streamlit/app buildFast",
-    "buildLib": "yarn workspace @streamlit/lib build",
-    "buildLibProd": "yarn workspace @streamlit/lib build:prod",
-    "buildApp": "yarn workspace @streamlit/app build",
-    "typecheck": "yarn workspace @streamlit/lib typecheck && yarn workspace @streamlit/app typecheck",
-    "lint": "yarn lintApp && yarn lintLib",
-    "lintApp": "yarn workspace @streamlit/app lint",
-    "lintLib": "yarn workspace @streamlit/lib lint",
-    "format": "prettier --write --config .prettierrc './{app,lib}/src/**/*.{js,ts,jsx,tsx}'",
-    "formatCheck": "prettier --check --config .prettierrc './{app,lib}/src/**/*.{js,ts,jsx,tsx}'",
-    "lint:interactive": "yarn eslint-interactive ./app/src ./lib/src",
-    "test": "yarn workspace @streamlit/lib test && yarn workspace @streamlit/app test",
-    "testcoverage": "yarn workspace @streamlit/lib test --coverage && yarn workspace @streamlit/app test --coverage",
-    "testLib": "yarn workspace @streamlit/lib testWatch",
-    "testApp": "yarn workspace @streamlit/app testWatch"


Are these scripts mostly removed as a cleanup, or is there also a technical issue related to this?

Yes. I thought it would be better to just use the syntax for people and thought this would be a reset. Especially cause you can run tests on all workspaces etc.

yeah, I think I haven't been using these commands in a longer time. Previously, I sometimes used testLib and testApp form the frontend folder but since migrating to vite I have been running all tests just in the VS Code test explorer.

frontend/.yarnrc.yml

scripts/audit_frontend_licenses.py

This reverts commit a348776c5aa8c8b7f9cc55b2f8376ae36cf83841.

lukasmasuch

LGTM 🚀

sfc-gh-bnisco · 2025-01-10T16:51:30Z

.github/workflows/component-template-e2e-tests.yml

      - name: Build Package
        timeout-minutes: 120
-        run: make package
+        run: BUILD_AS_FAST_AS_POSSIBLE=1 make package


question: Is this still an expected change? I believe we removed BUILD_AS_FAST_AS_POSSIBLE

Ah, this may have been a merge issue. I'll double-check and remove.

sfc-gh-bnisco · 2025-01-10T16:53:27Z

.github/workflows/js-tests.yml

+      - name: Enable and Prepare Latest Yarn
+        run: |
+          corepack enable
+          corepack prepare yarn@4 --activate


question: Do we need to specify the exact version of yarn 4 in this command? We have the expected yarn version listed in frontend/package.json, I wonder if they should match here?

Good point! I lean towards making the package manager in package.json more generic (yarn@4) to help identify issues as they come up with new releases/fixes. Does that work with you, or do you feel like having more consistent version/reliable builds more value here?

I think we should stick to a specific version since it ensures reproducibility across builds and environments. Helps remove one potential root cause of "works on my machine." 😄

Sounds good. I'll update it accordingly.

sfc-gh-bnisco

LGTM! 🚀

## Describe your changes Upgrade yarn to yarn berry. In order to do this we had to do the following: 1. Support yarn licenses in our checker 2. Install packages in both directories and run scripts from the specific directory 3. Fixes to enable yarn in GitHub Actions and whatnot. ## Testing Plan - Tests should pass --- **Contribution License Agreement** By submitting this pull request you agree that all contributions to this project are made under the Apache 2.0 license.

kmcgrady added security-assessment-completed Security assessment has been completed for PR impact:internal PR changes only affect internal code change:refactor PR contains code refactoring without behavior change labels Dec 10, 2024

kmcgrady force-pushed the refactor/upgrade-yarn branch 6 times, most recently from ae96648 to d29670a Compare December 11, 2024 04:46

kmcgrady marked this pull request as ready for review December 11, 2024 04:48

kmcgrady requested review from a team and mayagbarnes as code owners December 11, 2024 04:48

kmcgrady changed the title ~~[WIP] Upgrade yarn to latest~~ Upgrade yarn to latest Dec 11, 2024

kmcgrady commented Dec 11, 2024

View reviewed changes

raethlein reviewed Dec 11, 2024

View reviewed changes

lukasmasuch reviewed Jan 2, 2025

View reviewed changes

scripts/audit_frontend_licenses.py Show resolved Hide resolved

lukasmasuch reviewed Jan 3, 2025

View reviewed changes

frontend/.yarnrc.yml Show resolved Hide resolved

lukasmasuch reviewed Jan 3, 2025

View reviewed changes

scripts/audit_frontend_licenses.py Show resolved Hide resolved

kmcgrady force-pushed the refactor/upgrade-yarn branch 5 times, most recently from 21f7118 to a348776 Compare January 9, 2025 17:52

kmcgrady added 5 commits January 9, 2025 11:55

Upgrade yarn to latest

4eb0345

Update yarn lock file

562a29f

Update frozen-lockfile to immutable

b772baa

Move vite to dev dependencies

3bf0e24

Add dummy test to trigger changed files

b48d36c

kmcgrady added 3 commits January 9, 2025 11:55

Sync files post build with profiler

ca0bcc9

Revert "Add dummy test to trigger changed files"

3087e5a

This reverts commit a348776c5aa8c8b7f9cc55b2f8376ae36cf83841.

Update notices

7751b27

kmcgrady force-pushed the refactor/upgrade-yarn branch from 7289a9a to 7751b27 Compare January 9, 2025 19:55

lukasmasuch approved these changes Jan 10, 2025

View reviewed changes

sfc-gh-bnisco reviewed Jan 10, 2025

View reviewed changes

Address PR comments

3d013ae

sfc-gh-bnisco approved these changes Jan 10, 2025

View reviewed changes

kmcgrady added 6 commits January 10, 2025 13:35

Re-introduce cache

c01106c

Support patch-package

d02b2bb

Empty Commit

171d574

Enable corepack first

508f907

Enable corepack for component-template

94a87d0

Enable corepack for component-template

7bcdf71

kmcgrady merged commit a19d2d4 into develop Jan 11, 2025
33 checks passed

kmcgrady deleted the refactor/upgrade-yarn branch January 11, 2025 02:10

Upgrade yarn to latest #9989

Upgrade yarn to latest #9989

Uh oh!

Conversation

kmcgrady commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes

Testing Plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kmcgrady Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukasmasuch Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lukasmasuch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfc-gh-bnisco left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kmcgrady commented Dec 10, 2024 •

edited

Loading

kmcgrady Jan 8, 2025 •

edited

Loading

lukasmasuch Jan 10, 2025 •

edited

Loading