Use a dev drive for testing on Windows by zanieb · Pull Request #15664 · astral-sh/ruff

zanieb · 2025-01-21T23:42:03Z

Investigating the benefits of this here.

Requires Windows 2025, switching over in #15661

github-actions · 2025-01-21T23:54:11Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

zanieb · 2025-01-22T00:10:04Z

Hm okay this gives a big speed-up but a test is failing

──── STDERR:             red_knot::file_watching directory_deleted
thread 'directory_deleted' panicked at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869\library\core\src\ops\function.rs:250:5:
Didn't observe expected change:
  - Deleted { path: "E:\\ruff-tmp\\.tmppcH4p4\\project\\sub\\a.py", kind: Any }
  - Deleted { path: "E:\\ruff-tmp\\.tmppcH4p4\\project\\sub\\__init__.py", kind: Any }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

zanieb · 2025-01-22T03:20:49Z

I think the small runner on the D:\ drive is too slow at ~12 minutes.

@MichaReiser would you be interested in investigating if this is a real bug with Dev Drives? I'm not sure if there are known issues with the file watcher?

I'm happy to procure more logs if you tell me how. Or I can test a minimal snippet, if you can extract the core folder-watching behavior or point me to it?

zanieb · 2025-01-22T16:03:28Z

.github/workflows/setup-dev-drive.ps1

@@ -0,0 +1,62 @@
+# Configures a drive for testing in CI.


Scary, but just copied over from uv.

zanieb · 2025-01-22T16:59:23Z

Glorious, here are the logs

──── STDERR:             red_knot::file_watching directory_deleted
2025-01-22T16:57:43.337130Z DEBUG red_knot_project::metadata: Searching for a project in 'E:\ruff-tmp\.tmpLgedMu\project'
2025-01-22T16:57:43.337503Z DEBUG red_knot_project::metadata: The ancestor directories contain no `pyproject.toml`. Falling back to a virtual project.
2025-01-22T16:57:43.338307Z  INFO red_knot_python_semantic::program: Python version: Python 3.9, platform: all
2025-01-22T16:57:43.338388Z DEBUG red_knot_python_semantic::module_resolver::resolver: Adding first-party search path 'E:\ruff-tmp\.tmpLgedMu\project'
2025-01-22T16:57:43.338468Z DEBUG red_knot_python_semantic::module_resolver::resolver: Using vendored stdlib
2025-01-22T16:57:43.345608Z TRACE red_knot_project::db: Salsa event: Event { thread_id: ThreadId(2), kind: WillExecute { database_key: dynamic_resolution_paths(Id(800)) } }
2025-01-22T16:57:43.345729Z DEBUG red_knot_python_semantic::module_resolver::resolver: Resolving dynamic module resolution paths
2025-01-22T16:57:43.345984Z DEBUG red_knot_project::watch::watcher: Watching path: `E:\ruff-tmp\.tmpLgedMu\project`
2025-01-22T16:57:43.346325Z  INFO red_knot_project::watch::project_watcher: Set up file watchers for ["E:\ruff-tmp\.tmpLgedMu\project"]
2025-01-22T16:57:43.448406Z TRACE ruff_db::files: Adding file 'E:\ruff-tmp\.tmpLgedMu\project\bar.py'
2025-01-22T16:57:43.449023Z TRACE red_knot_project::db: Salsa event: Event { thread_id: ThreadId(2), kind: WillExecute { database_key: resolve_module_query(Id(1000)) } }
2025-01-22T16:57:43.449297Z TRACE resolve_module: ruff_db::files: Adding file 'E:\ruff-tmp\.tmpLgedMu\project\sub\__init__.py' name=sub.a
2025-01-22T16:57:43.449472Z TRACE resolve_module: ruff_db::files: Adding file 'E:\ruff-tmp\.tmpLgedMu\project\sub\a\__init__.pyi' name=sub.a
2025-01-22T16:57:43.449590Z TRACE resolve_module: ruff_db::files: Adding file 'E:\ruff-tmp\.tmpLgedMu\project\sub\a\__init__.py' name=sub.a
2025-01-22T16:57:43.449696Z TRACE resolve_module: ruff_db::files: Adding file 'E:\ruff-tmp\.tmpLgedMu\project\sub\a.pyi' name=sub.a
2025-01-22T16:57:43.449789Z TRACE resolve_module: ruff_db::files: Adding file 'E:\ruff-tmp\.tmpLgedMu\project\sub\a.py' name=sub.a
2025-01-22T16:57:43.449905Z TRACE resolve_module: red_knot_python_semantic::module_resolver::resolver: Resolved module `sub.a` to `E:\ruff-tmp\.tmpLgedMu\project\sub\a.py` name=sub.a
2025-01-22T16:57:43.455836Z  INFO Project::index_files: red_knot_project: Found 3 files in project `project` package=project
thread 'directory_deleted' panicked at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869\library\core\src\ops\function.rs:250:5:
Didn't observe expected change:
  - Deleted { path: "E:\\ruff-tmp\\.tmpLgedMu\\project\\sub\\a.py", kind: Any }
  - Deleted { path: "E:\\ruff-tmp\\.tmpLgedMu\\project\\sub\\__init__.py", kind: Any }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

zanieb · 2025-01-22T17:28:06Z

(they don't seem helpful — not sure how to get more...)

MichaReiser · 2025-01-24T17:25:48Z

(they don't seem helpful — not sure how to get more...)

How dare you! I improved them a lot and I think there's some useful content in them.

The specific test tests if the file watcher correctly handles the case where an entire directory gets deleted, specifically the sub directory. That's why the test expects a change event for the sub directory. What's interesting is that no such event gets emitted on the devdrive. It only emits the events for the files in that directory

Didn't observe expected change:
  - Deleted { path: "E:\\ruff-tmp\\.tmpLgedMu\\project\\sub\\a.py", kind: Any }
  - Deleted { path: "E:\\ruff-tmp\\.tmpLgedMu\\project\\sub\\__init__.py", kind: Any }

Now, we could fix the test to expect the events for one of the files instead of the directories, and I suspect it would then pass just fine on all platforms.

However, this reveals a more fundamental problem with changing to dev drives: Windows's file-watching API returns different events depending on whether the path is a dev drive or not. Migrating to a dev drive would, therefore, remove the coverage for regular Windows drives, which I consider fairly important.

That means I'm not sure if we should change our runner to use dev drivers. Is there a way to only use a dev drive for some directories, e.g., not the directory created by tempdir?

MichaReiser · 2025-01-24T17:38:49Z

.github/workflows/ci.yaml

+          RED_KNOT_LOG: debug
+          RUST_LOG: debug


This should not be needed when you use testing::setup_tracing() It always writes the logs and they should be visible if the test fails

MichaReiser · 2025-01-24T17:39:40Z

.github/workflows/setup-dev-drive.ps1

+} else {
+	# The size (20 GB) is chosen empirically to be large enough for our
+	# workflows; larger drives can take longer to set up.
+	$Volume = New-VHD -Path C:/ruff_dev_drive.vhdx -SizeBytes 20GB |


We can probably get away with a smaller drive for Ruff.

I actually tried 5 GB then 10 GB and both ran out of space during compilation.

MichaReiser · 2025-01-24T17:40:18Z

.github/workflows/setup-dev-drive.ps1

+
+# Note we use `Get-PSDrive` is not sufficient because the drive letter is assigned.
+if (Test-Path "D:\") {
+    Write-Output "Using existing drive at D:"


A beautiful mix of space and tab indentation. hehe

zanieb · 2025-01-24T17:48:33Z

Funny, I read

Didn't observe expected change:
  - Deleted { path: "E:\\ruff-tmp\\.tmpLgedMu\\project\\sub\\a.py", kind: Any }
  - Deleted { path: "E:\\ruff-tmp\\.tmpLgedMu\\project\\sub\\__init__.py", kind: Any }

as: we expected these changes but did not observe them.

Now I see — the message is much more helpful read that way.

zanieb · 2025-01-24T17:50:18Z

However, this reveals a more fundamental problem with changing to dev drives: Windows's file-watching API returns different events depending on whether the path is a dev drive or not. Migrating to a dev drive would, therefore, remove the coverage for regular Windows drives, which I consider fairly important.

Yeah I agree coverage for a regular drive seems important. It sounds like it'd actually be important to get both coverage for a Dev Drive and the regular file system. I can own that.

That means I'm not sure if we should change our runner to use dev drivers. Is there a way to only use a dev drive for some directories, e.g., not the directory created by tempdir?

Yeah, but then we'll probably miss out on the benefits. I can explore.

MichaReiser · 2025-01-24T17:50:29Z

That's fair. There's definetely still room for improvement. Ideally, the message would also say which event we expected. But it's much better than in the past where it just told you: Too bad, didn't work 😆

zanieb · 2025-01-24T17:50:36Z

Thanks for taking a look Micha. I see this as relatively low priority, but I can explore it in my spare time,

MichaReiser · 2025-01-24T17:51:35Z

Thank you. Excluding the temp directory might be fine for Ruff. We don't write as many files as you do in uv. It's only a handful tests that make use of temp (although I don't know if cargo or other tools make heavy use of temp)

I really misunderstood this in #15664 (comment)

github-actions · 2025-06-18T12:19:09Z

`mypy_primer` results

Changes were detected when running on open source projects

mypy_primer (https://github.com/hauntsaninja/mypy_primer)
-     memo fields = ~41MB
+     memo fields = ~45MB

aioredis (https://github.com/aio-libs/aioredis)
-     memo fields = ~54MB
+     memo fields = ~60MB

attrs (https://github.com/python-attrs/attrs)
- TOTAL MEMORY USAGE: ~72MB
+ TOTAL MEMORY USAGE: ~80MB

Expression (https://github.com/cognitedata/Expression)
-     memo fields = ~49MB
+     memo fields = ~54MB

operator (https://github.com/canonical/operator)
- TOTAL MEMORY USAGE: ~97MB
+ TOTAL MEMORY USAGE: ~106MB
-     memo fields = ~80MB
+     memo fields = ~88MB

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- TOTAL MEMORY USAGE: ~80MB
+ TOTAL MEMORY USAGE: ~88MB

mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
-     memo fields = ~171MB
+     memo fields = ~156MB

paasta (https://github.com/yelp/paasta)
-     memo fields = ~171MB
+     memo fields = ~156MB

From 6m 15s -> 3m 54s (total runtime) See also astral-sh/uv#14122 We don't use a Dev Drive here so this is trivial (ref #15664)

This reverts commit 82dc27f.

codspeed-hq · 2025-07-03T12:11:24Z

CodSpeed Performance Report

Merging #15664 will not alter performance

_{Comparing zb/dev-drive (40b4aa2) with main (d0f0577)}

Summary

✅ 8 untouched

MichaReiser · 2025-12-10T17:38:44Z

I'll close this as it looks pretty stale but please re-open it if you think we should keep this around.

zanieb added the internal An internal refactor or improvement label Jan 21, 2025

zanieb force-pushed the zb/dev-drive branch from 7f839e5 to e31209a Compare January 21, 2025 23:44

zanieb mentioned this pull request Jan 21, 2025

Use new-style Windows runner tag and 2025 preview #15661

Merged

zanieb force-pushed the zb/dev-drive branch 3 times, most recently from 47662ec to 322c5ab Compare January 22, 2025 00:04

zanieb force-pushed the zb/dev-drive branch from 14751d6 to 30447fb Compare January 22, 2025 00:22

zanieb mentioned this pull request Jan 22, 2025

Cache the Rust toolchain in CI #15660

Merged

zanieb added ci Related to internal CI tooling and removed internal An internal refactor or improvement labels Jan 22, 2025

zanieb commented Jan 22, 2025

View reviewed changes

zanieb force-pushed the zb/dev-drive branch 2 times, most recently from 052303b to 4977a38 Compare January 22, 2025 16:36

MichaReiser reviewed Jan 24, 2025

View reviewed changes

zanieb mentioned this pull request Jan 24, 2025

Improve the file watching failure error message #15728

Merged

zanieb added a commit that referenced this pull request Jan 24, 2025

Improve the file watching failure error message (#15728)

fcd0f34

I really misunderstood this in #15664 (comment)

zanieb force-pushed the zb/dev-drive branch from 2473404 to 925f433 Compare June 18, 2025 12:15

zanieb mentioned this pull request Jun 18, 2025

Use Depot Windows runners for cargo test #18754

Merged

zanieb added a commit that referenced this pull request Jun 18, 2025

Use Depot Windows runners for cargo test (#18754)

cb512ba

From 6m 15s -> 3m 54s (total runtime) See also astral-sh/uv#14122 We don't use a Dev Drive here so this is trivial (ref #15664)

zanieb force-pushed the zb/dev-drive branch 3 times, most recently from 61f58df to 6965001 Compare June 18, 2025 16:13

MichaReiser closed this Jun 20, 2025

MichaReiser reopened this Jun 20, 2025

zanieb added 9 commits July 3, 2025 07:01

Use a dev drive for testing on Windows

0112f7f

Add debug logs?

540cbd9

Only run the relevant test

b233888

Enable more logs

306f6f1

Remove fuzz corpus hack

a014d55

Remove log variables

38bb96a

Do not set TMP

34c9836

Update for Depot

7f4aa4b

Revert "Only run the relevant test"

ea4bf00

This reverts commit 82dc27f.

zanieb force-pushed the zb/dev-drive branch from 0706e68 to ea4bf00 Compare July 3, 2025 12:01

Zizmor

40b4aa2

MichaReiser closed this Dec 10, 2025

Comments

Conversation

zanieb commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

zanieb commented Jan 22, 2025

Uh oh!

zanieb commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zanieb Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

zanieb commented Jan 22, 2025

Uh oh!

zanieb commented Jan 22, 2025

Uh oh!

MichaReiser commented Jan 24, 2025

Uh oh!

MichaReiser Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

MichaReiser Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

zanieb Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

MichaReiser Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

zanieb commented Jan 24, 2025

Uh oh!

zanieb commented Jan 24, 2025

Uh oh!

MichaReiser commented Jan 24, 2025

Uh oh!

zanieb commented Jan 24, 2025

Uh oh!

MichaReiser commented Jan 24, 2025

Uh oh!

github-actions bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

codspeed-hq bot commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #15664 will not alter performance

Summary

Uh oh!

MichaReiser commented Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zanieb commented Jan 21, 2025 •

edited

Loading

github-actions bot commented Jan 21, 2025 •

edited

Loading

`ruff-ecosystem` results

zanieb commented Jan 22, 2025 •

edited

Loading

github-actions bot commented Jun 18, 2025 •

edited

Loading

`mypy_primer` results

codspeed-hq bot commented Jul 3, 2025 •

edited

Loading