Skip to content

Running job that hits cache and has cached outputs slowed down after upgrading to 16.9.0 #19627

Description

@summis

Current Behavior

Hi!

In our monorepo we have build jobs that produce output files. When running build jobs with command nx run-many --target build, and every job hits local cache, the performance profile obtained following instructions from here looks as follows:

image

In total, build takes ~900 ms. Based on my testing we get similar numbers starting from nx version 16.9.0 and reproduces also in 17.0.0-beta.6.

Expected Behavior

Previously, the job was significantly faster under similar conditions, where every build job got local cache hit. Here is the same example when using nx version 16.8.1.

image

In total, build took ~20 ms. Therefore, the time it takes to complete job when getting a cache hit has increased by an order of magnitude.

GitHub Repo

No response

Steps to Reproduce

  1. Benchmark the time it takes to complete a job that hits cache and has cached output when nx version 16.8.1 is installed
  2. Repeat 1. with nx version 16.9.0 and observe the difference

Nx Report

>  NX   Report complete - copy this into the issue template

   Node   : 18.15.0
   OS     : linux-x64
   npm    : 9.5.0
   
   nx             : 16.9.0
   @nx/workspace  : 16.5.5
   @nx/devkit     : 16.8.1
   @nrwl/tao      : 16.6.0
   typescript     : 5.1.6
   ---------------------------------------
   Local workspace plugins:
         @XXX/local-nx-plugins
   ---------------------------------------
   The following packages should match the installed version of nx
     - @nx/[email protected]
     - @nrwl/[email protected]
     - @nx/[email protected]
     - @nrwl/[email protected]
     - @nrwl/[email protected]
   
   To fix this, run `nx migrate [email protected]`

Failure Logs

No response

Package Manager Version

No response

Operating System

  • macOS
  • Linux
  • Windows
  • Other (Please specify)

Additional Information

We have observed this slowing down only in jobs that produce output files that are stored to cache.

My colleague - a MAC user, I use linux - reported even bigger slowing down in similar setting. For him running the build with every individual build job hitting local cache may take up to 10 seconds.

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions