Cache tokens on expressions and restore after pickle roundtrip #11791
Conversation
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 9 files ± 0 9 suites ±0 3h 25m 58s ⏱️ - 1m 54s For more details on these failures, see this check. Results for commit f4cd5ac. ± Comparison against base commit 5f61e42. ♻️ This comment has been updated with latest results. |
dask/_expr.py
Outdated
| @functools.cached_property | ||
| def _name(self) -> str: | ||
| return self._funcname + "-" + _tokenize_deterministic(*self.operands) | ||
| if not self._token: |
There was a problem hiding this comment.
The name is overwritten in a few places, is this an issue?
There was a problem hiding this comment.
there are just 38 places this is overridden one way or another... no big deal!
|
I think I now broke out all tokenization from the actual naming. There are a few subclasses that overwrite the tokenization although I believe that should not be necessary. |
|
One of the test failures is fixed by #11795 and the other one is a random weird thing of aioboto |
|
thanks |
2 participants
xref #11736
This caches the tokens on the expression and restores calculated tokens after a pickle roundtrip. This helps us keep tokens consistent between client and scheduler.
This is safe to do since Expr are by definition immutable (since otherwise our singleton approach would fall apart anyhow)