Running some profiling on the TCPH benchmarks, I noticed a significant portion of the main thread to be blocked by the calculation of _meta during the deserialization of Expr objects

This screenshot indicates that the expr instance that is being (de-)serialized is the Merge expression.

Looking through the traces a bit more, I can also find references to a parquet layer which almost looks like it would fetch some data (but I hope this is false). It is definitely performing some pyarrow table conversion.

speedscope file (a Worker)
tls-10_0_32_152-45157.json.zip

Just skimming the code, I saw stuff like this

https://github.com/dask-contrib/dask-expr/blob/52346744cf530ca86108553fb0427f552d91854e/dask_expr/_concat.py#L183

where a method of an expression is used to define a dask task. This inevitably requires us to serialize the instance itself which is something we should avoid. Ideally, the graph only includes global functions such that we can pickle by reference and not by value.

I don't know if there are other places where this pattern comes up.

related #284