Implement caching of NPM in CI / local dev #43167
Description
Currently when NPM packages are installed on CI they are installed from the scratch by "compile www assets" pre-commit.
This causes fairly frequent, intermittent issues when there is a networking issue or NPM server issue - for example:
Compile www assets (manual)..............................................Failed
- hook id: compile-www-assets
- exit code: 1
yarn install v1.22.21
(node:490) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
[1/4] Resolving packages...
[2/4] Fetching packages...
[] 0/1573[] 5/1573[] 11/1573[] 18/1573[] 26/1573[] 34/1573[] 41/1573[] 45/1573[] 53/1573[] 61/1573[] 68/1573[] 75/1573[] 84/1573[] 90/1573[] 97/1573[] 104/1573[] 111/1573[] 116/1573[] 122/1573[] 128/1573[] 135/1573[] 140/1573[] 147/1573[] 153/1573[] 160/1573[] 167/1573[] 176/1573[] 185/1573[] 192/1573[] 197/1573[] 198/1573[] 202/1573[] 206/1573[] 211/1573[] 216/1573[] 223/1573[] 229/1573[] 232/1573[] 234/1573[] 238/1573[] 241/1573[] 243/1573[] 246/1573[] 248/1573[] 250/1573[] 255/1573[] 260/1573[] 265/1573[] 269/1573[] 276/1573[] 284/1573[] 286/1573[] 287/1573[] 290/1573[] 292/1573[] 295/1573[] 298/1573[] 305/1573[] 311/1573[] 316/1573[] 325/1573[] 331/1573[] 336/1573[] 342/1573[] 346/1573[] 349/1573[] 353/1573[] 359/1573[] 364/1573[] 368/1573[] 372/1573[] 378/1573[] 385/1573[] 391/1573[] 396/1573[] 402/1573[] 409/1573[] 414/1573[] 420/1573[] 427/1573[] 432/1573[] 437/1573[] 441/1573[] 443/1573[] 444/1573[] 445/1573[] 448/1573[] 451/1573[] 454/1573[] 457/1573[] 462/1573[] 467/1573[] 470/1573[] 473/1573[] 477/1573[] 481/1573[] 485/1573[] 490/1573[] 496/1573[] 503/1573[] 512/1573[] 519/1573[] 524/1573[] 526/1573[] 529/1573[] 531/1573[] 535/1573[] 539/1573[] 545/1573[] 548/1573[] 550/1573[] 553/1573[] 560/1573[] 567/1573[] 574/1573[] 580/1573[] 591/1573[] 599/1573[] 607/1573[] 616/1573[] 620/1573[] 624/1573[] 627/1573[] 630/1573[] 634/1573[] 641/1573[] 650/1573[] 658/1573[] 668/1573[] 676/1573[] 684/1573[] 692/1573error Error: https://registry.yarnpkg.com/@babel/plugin-syntax-logical-assignment-operators/-/plugin-syntax-logical-assignment-operators-7.10.4.tgz: Request failed "500 Internal Server Error"
at ResponseError.ExtendableBuiltin (/opt/airflow/files/home/.cache/pre-commit/repocv9ljx22/node_env-22.2.0/lib/node_modules/yarn/lib/cli.js:696:66)
at new ResponseError (/opt/airflow/files/home/.cache/pre-commit/repocv9ljx22/node_env-22.2.0/lib/node_modules/yarn/lib/cli.js:802:124)
at Request.<anonymous> (/opt/airflow/files/home/.cache/pre-commit/repocv9ljx22/node_env-22.2.0/lib/node_modules/yarn/lib/cli.js:66218:16)
at Request.emit (node:events:520:28)
at module.exports.Request.onRequestResponse (/opt/airflow/files/home/.cache/pre-commit/repocv9ljx22/node_env-22.2.0/lib/node_modules/yarn/lib/cli.js:141751:10)
at ClientRequest.emit (node:events:520:28)
at HTTPParser.parserOnIncomingClient (node:_http_client:700:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)
at TLSSocket.socketOnData (node:_http_client:542:22)
at TLSSocket.emit (node:events:520:28)
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Traceback (most recent call last):
File "/opt/airflow/./scripts/ci/pre_commit/compile_www_assets.py", line 71, in <module>
subprocess.check_call(["yarn", "install", "--frozen-lockfile"], cwd=os.fspath(www_directory))
File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['yarn', 'install', '--frozen-lockfile']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/airflow/files/home/.local/share/hatch/env/virtual/apache-airflow-build/lib/python3.9/site-packages/hatchling/__main__.py", line 6, in <module>
sys.exit(hatchling())
File "/opt/airflow/files/home/.local/share/hatch/env/virtual/apache-airflow-build/lib/python3.9/site-packages/hatchling/cli/__init__.py", line 26, in hatchling
command(**kwargs)
File "/opt/airflow/files/home/.local/share/hatch/env/virtual/apache-airflow-build/lib/python3.9/site-packages/hatchling/cli/build/__init__.py", line 82, in build_impl
for artifact in builder.build(
File "/opt/airflow/files/home/.local/share/hatch/env/virtual/apache-airflow-build/lib/python3.9/site-packages/hatchling/builders/plugin/interface.py", line 155, in build
artifact = version_api[version](directory, **build_data)
File "/opt/airflow/hatch_build.py", line 613, in build_standard
run(cmd, cwd=work_dir.as_posix(), check=True, shell=True)
File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pre-commit run --hook-stage manual compile-www-assets --all-files']' returned non-zero exit status 1.
Error building Airflow packages
Error preparing Airflow package
We should - similarly to Python packages, implement caching strategy that should speed up the installation of npm packages on a clean CI runner, as well as local development instance - where previously installed packages could be stored in the cache and reused. This should increase both - speed of CI jobs an stability of them (and speed of local installation of npm packages).