Repository navigation
Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas
df1 = pandas.DataFrame()
boolean_series = pandas.Series(dtype=bool)
df1[boolean_series]
# Empty DataFrame
# Columns: []
# Index: []
df2 = pandas.DataFrame(index=[0, 1])
df2[boolean_series]
# IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).Issue Description
Trying to use an empty boolean Series to select on an empty DataFrame that has an index results in an error.
Expected Behavior
I would expect to return an empty DataFrame. The expectation might make more sense with an example.
import pandas
df1 = pandas.DataFrame(['a', 'b'], index = [0, 1])
df1[df1.duplicated()]
# Empty DataFrame
# Columns: [0]
# Index: []
df2 = pandas.DataFrame(index = [0, 1])
df2[df2.duplicated()]
# IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).
Both of these DataFrames have no duplicate values, but only one results in an error. It would be nice not to require a test for this special case and just get an empty DataFrame as the result since an empty DataFrame does not contain any duplicates.
I looked into this a little bit because I thought maybe the .duplicated method just needed to have the empty Series also return the index, but it is not possible, as far as I can tell, to create a Series with an index but no values like you can with a DataFrame. If you try, the values are set to some default. In the case for bool it is True. I think the selection code would have to check for an empty Series before trying to use the index and return an empty DataFrame. If I am investigating this correctly, it looks like in pandas/core/frame.py in the ._getitem_bool_array method you could add a case to the if chain at the top. Something like:
if isinstance(key, Series) and key.empty:
return self._take_with_is_copy(Index([]), axis=0)
Installed Versions
Details
INSTALLED VERSIONS
commit : 0691c5c
python : 3.10.5
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en
LOCALE : English_United States.1252
pandas : 2.2.3
numpy : 1.24.4
pytz : 2022.1
dateutil : 2.8.2
pip : 25.0.1
Cython : 3.0.11
sphinx : 5.1.1
IPython : 8.21.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : 1.1
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : 4.9.1
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.4
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 19.0.1
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 2.0.9
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
xlsxwriter : 3.2.0
zstandard : None
tzdata : 2024.1
qtpy : 2.4.1
pyqt5 : None
Activity
rhshadrach commentedon Mar 28, 2025
Thanks for the report. First a case where we do not deal with empty objects:
I do not think there is an appetite for changing this behavior. I agree it would be great if
could always work, but I do not see a way to change
duplicatednor__getitem__to make this so. E.g. if we were to follow your proposal:this would introduce an edge case that goes against the general rule (unalignable Series will raise, except when they are empty). As such, I'm opposed to this way forward.
ptth222 commentedon Mar 28, 2025
You are just picking your edge case though. You either introduce the one I suggested or leave in the one that's there. The one that's there is much more onerous in my opinion. Is anyone really relying on an empty Series mask to result in an error? How many people would rather rely on the DataFrame's own built in methods to always work with itself? Do you want to keep the current edge case so some arbitrary general rule doesn't have an exception, resulting in a less functional class? Or do you want the edge case that improves functionality, but breaks a general rule in a scenario where it basically doesn't matter (IMO)? Note that these are questions to actually consider, not me just trying to win an argument or something.
Some alternatives:
You could change Series so it could have an index without values, or DataFrame so it can't. It's weird that these don't have the same behavior in that respect. That seems a bit more involved though.
Another solution is to have every method that returns a boolean Series check if it is going to return an empty Series and instead return one with all False values the same length as the DataFrame. This would also be a pain tracking all of these down. Having boolean Series default value be False instead of True and always passing the DataFrame index into the Series constructor would also work, but it's likely people rely on the default True somewhere.
Ultimately this is a structural issue of Series not being able to have an index without values while DataFrames can, and choosing to return boolean Series for methods.
The way I see it the options are:
Note that there is a similar error with unaligned sizes:
To me this suggests having methods always return a Series of the same size as the DataFrame might be a better overall fix. At least when the Series dtype is bool. What's more logical, returning False for every row when there are no columns in a DataFrame, or returning an empty Series? I don't really know, but for the scenario that I started this Issue for the all False Series doesn't result in an error.
Edit: To do this in .duplicated it looks like you could change:
To:
I'm not sure how many methods do something similar, but it would be a good idea to change those too if this is acceptable.
Edit 2: Note that both the "all" and "any" methods return values in this same situation.
There is more historical precedent for these methods returning True or False for empty sets, but I think the reasoning could also be applied to other boolean methods. Making it so that a Series can be empty with an index I think is the best fix, but having reasonable default values for boolean methods is also okay.
Edit 3: So due to how any() and all() work, boolean methods cannot return default boolean values for empty DataFrames. Since all() must return True for an empty set, methods like isna have to return an empty set when used on an empty DataFrame. To return anything else will result in an incorrect result when chained with any() or all(). This basically nullifies everything I said in regards to that possible solution. I think the only real choices are allowing an empty Series to have an index or modifying the getitem to ignore the size of the input if the DataFrame is empty and just return empty.
BUG: DataFrame.duplicated() loses index when DataFrame has no columns (…