Add extra check in _normalize_lambda for lambda function #689
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Will close dask/dask#10760
TL/DR; dask-expr needs an extra check for a lambda function in
_normalize_lambdasince free functions are also instances oftypes.LambdaType(it's actually an alias totypes.FunctionType...because, why not?). And failing on Python 3.12 because that's the only one that also includes adask-exprdependency.git bisect'd to #10676 (the last tests in that PR unsurprisingly also show the failing 3.12 test)
Running the failing test by itself (
test_use_cloudpickle_to_tokenize_functions_in__main__) will work fine, but running the module (python -m pytest -k test_base) will bring the issue out. Then commenting out the two tests added in that PR will also allow everything to pass.Specifically, running by itself returns the expected bytes but running the module the
normalize_tokenwill instead return a string, like'<function inc at 0x7f5e97dad9e0>', giving the error we saw in the failing test.This was because dask-expr's
_utils._normalize_lambdaregistered ontypes.LambdaType. Adding an extra check for the'lambda'function name as well as<locals>to avoid cloudpipe/cloudpickle#385 resolves the issue. Just so happens the tests added in the referenced PR managed to import dask-expr and thus added_normalize_lambdato the mix.