Repository navigation
Dask 2024.5.1 removed .attrs #11146
Description
The latest release removed .attrs. This breaks backward compatibility and also Pandas still uses it, so I kindly ask if it can be reinstated.
To reproduce, the last assert should pass.
import dask.dataframe as dd
df = dd.from_pandas(pd.DataFrame())
assert hasattr(df.compute(), 'attrs') == True
assert hasattr(df, 'attrs') == TrueEnvironment:
- Dask version: 2024.5.1
- Python version: 3.12.2
- Operating System: macOS Sonoma 14.5
- Install method (conda, pip, source): conda
Reacted by Mark Keller and Lukas Heumos
Activity
phofl commentedon May 25, 2024
Thanks for the report.
Could you add a bit more context about what you are trying to achieve with attr? attr doesn't really work in pandas either and support is spotty at most.
LucaMarconato commentedon May 27, 2024
I also experienced a problem with
pandasand.attrs: copiedDataFrameobjects where not containing a copy of.attrs. (I initially thought it was ageopandasproblem geopandas/geopandas#2920), but then it turned out to be frompandaspandas-dev/pandas#54134. Fortunately the bug above is now fixed. Which other problems did you find?The use case for
.attrswe have in the library we developed https://github.com/scverse/spatialdata/, is to store metadata associated toGeoDataFrameandDataFrameobjects (both lazy and non-lazy). The metadata mostly contain json-like information that describes how various spatial objects are aligned together.LucaMarconato commentedon Jun 2, 2024
Do you have any update on this? 😊 Some users reported problems with installation due to having pinned an older version of Dask in our config so it's an important issue for us. Thank you for your time.
phofl commentedon Jun 6, 2024
This is non-trivial to add and the semantics aren't completely clear either. This can't live on the collection level, it has to be on the expression level since we are constantly recreating the underlying expression, which makes it non-trivial. That said, the following is unclear to me:
dask-expr will reorder the query and push the projection in front of the fillna. So what attr should take precedence here?
Contributions are very welcome. I won't have much time to think about this though
LucaMarconato commentedon Jun 9, 2024
Thanks for the answer. From my understanding (and for the purpose of our use case) the
df.attrsslot should not be treated as lazy but always be executed immediately. This was the behavior implemented before Dask 2024.5.1. In particular in your example the computational graph would never contain nodes related to modifying.atttrs.I think making a PR for this should be quick, if you agree with this semantic I could give it a try.
phofl commentedon Jun 9, 2024
That would be fine but this still doesn't cover what should happen after df.optimize() is called, which is the tricky part here
LucaMarconato commentedon Jun 9, 2024
I will try making some experiments and get back to you.
fjetter commentedon Jun 10, 2024
can you elaborate? My initial gut reaction is that this should only live on the collection level and not on expr.
--
pandas-dev/pandas#52166 reads like there are a lot of questions around this feature and I wouldn't be surprised if some of this is subject to change.
phofl commentedon Jun 10, 2024
We constantly re-create the expressions underneath the collection, creating completely new collections in the optimiser. Ensuring that this propagates seems non-trivial if the information is only available on the collection.
hendrikmakait commentedon Jun 10, 2024
Are we ever interested in maintaining the correct value of
.attrsfor any intermediate expression or only in its value on the root collection? In the latter case, attaching it to the collection and fixing it during the optimization step feels manageable.fjetter commentedon Jun 10, 2024
To summarize a bit of an offline conversation since I got confused about some of the earlier comments
attrsis currently a poorly defined API in pandas where semantics are not always clearly defined (some examples are defined in DEPR: attrs pandas-dev/pandas#52166 for instance around copy-on-write)Note: this is not related to 2024.5.1 but was introduced in 2024.3.0 when we enabled query optimization by default
LucaMarconato commentedon Nov 13, 2024
I made some experiments but never reported here; doing it now.
I tried this but, TLDR, I didn't manage to make it work and ended up temporarily disabling the new backend (here is a more elaborate comment of what I tried: scverse/spatialdata#570 (comment)).
Now with the latest release 2024.11.01, the old backend is disabled and and we would lose access to
.attrs, which plays a key role in thespatialdatalibrary.Do you have any update on this? Could you please try a fix? Please notice that as shown in the discussion here pandas-dev/pandas#52166, there are many developers that are relying on
.attrsinpandas, and therefore not having this available indask.dataframesmay prevent its use for them as well.I think that a "quick" fix could be the one implied by this comment #11146 (comment), i.e. keeping only the value of the root object. I also outlined what could be a longer term fix in my comment here: scverse/spatialdata#570 (comment). Happy to hear your thoughts on this.
phofl commentedon Nov 13, 2024
I currently don't have enough bandwidth to take this on. This would need someone from the community taking a look at it and providing a first PR for it.