Skip to content

errors with type narrowing #516

Closed
Closed

Description

@jasonboh

Hi, I am having problems understanding what I am seeing here in the below example.

Why can't mypy figure out that after the else statement, we definitely have a DataFrame?

from typing import Union

import pandas as pd

x: Union[pd.Series, pd.DataFrame, pd.Index] = pd.Series([1, 2, 3])
if isinstance(x, (pd.Series, pd.Index)):
    reveal_type(x)  # Revealed type is "Union[pandas.core.series.TimestampSeries, pandas.core.indexes.numeric.NumericIndex]"
    dtype = x.dtype
    reveal_type(dtype)  # Revealed type is "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype]"
else:  # a DataFrame
    reveal_type(x)  # Revealed type is "Union[pandas.core.series.Series[Any], pandas.core.frame.DataFrame, pandas.core.indexes.base.Index]"
    dtypes = x.dtypes  # yields error: Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr] 
    reveal_type(dtypes)  # "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype, pandas.core.series.Series[Any], Any]"
    dtypes_set = set(x.dtypes) # yield 2 errors: 
    # first error Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
    # second error: Argument 1 to "set" has incompatible type "Union[dtype[generic], ExtensionDtype, Series[Any], Any]"; expected "Iterable[Any]"  [arg-type]

Activity

  1. gandhis1 commented on Jan 21, 2023

    @gandhis1
    Contributor

    This would appear to be a Mypy bug. It's probably worth trying pyright as well - I suspect they've got it right. Either way, not a pandas-stubs issue.

  2. Dr-Irv commented on Jan 21, 2023

    @Dr-Irv
    Collaborator

    I modified the code a bit, and this is a mypy bug. One issue is that you bind x to a Series, so the else part is never entered. So I wrapped it in a function

    import pandas as pd
    
    def foo(x: Union[pd.Series, pd.DataFrame, pd.Index]): # = pd.Series([1, 2, 3])
        if isinstance(x, (pd.Series, pd.Index)):
            reveal_type(x)  # Revealed type is "Union[pandas.core.series.TimestampSeries, pandas.core.indexes.numeric.NumericIndex]"
            dtype = x.dtype
            reveal_type(dtype)  # Revealed type is "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype]"
        else:  # a DataFrame
            reveal_type(x)  # Revealed type is "Union[pandas.core.series.Series[Any], pandas.core.frame.DataFrame, pandas.core.indexes.base.Index]"
            dtypes = x.dtypes  # yields error: Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
            reveal_type(dtypes)  # "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype, pandas.core.series.Series[Any], Any]"
            dtypes_set = set(x.dtypes) # yield 2 errors:
            # first error Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
            # second error: Argument 1 to "set" has incompatible type "Union[dtype[generic], ExtensionDtype, Series[Any], Any]"; expected "Iterable[Any]"  [arg-type]

    pyright correctly reports:

      issue516a.py:7:21 - information: Type of "x" is "Series[Unknown] | Index"
      issue516a.py:9:21 - information: Type of "dtype" is "dtype[generic] | ExtensionDtype"
      issue516a.py:11:21 - information: Type of "x" is "DataFrame"
      issue516a.py:13:21 - information: Type of "dtypes" is "Series[Unknown]"
    

    mypy incorrectly reports:

    issue516a.py:7: note: Revealed type is "Union[pandas.core.series.TimestampSeries, pandas.core.indexes.numeric.NumericIndex]"
    issue516a.py:9: note: Revealed type is "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype]"
    issue516a.py:11: note: Revealed type is "Union[pandas.core.series.Series[Any], pandas.core.frame.DataFrame, pandas.core.indexes.base.Index]"
    issue516a.py:12: error: Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
    issue516a.py:13: note: Revealed type is "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype, pandas.core.series.Series[Any], Any]"
    issue516a.py:14: error: Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
    issue516a.py:14: error: Argument 1 to "set" has incompatible type "Union[dtype[generic], ExtensionDtype, Series[Any], Any]"; expected "Iterable[Any]"  [arg-type]
    
  3. Dr-Irv commented on Jan 22, 2023

    @Dr-Irv
    Collaborator

    Created a mypy issue: python/mypy#14502

  4. jasonboh commented on Jan 31, 2023

    @jasonboh
    Author

    thanks everyone! I have started using pyright in VS Code.

  5. Dr-Irv commented on Jan 31, 2023

    @Dr-Irv
    Collaborator

    thanks everyone! I have started using pyright in VS Code.

    You can also use pyright from the command line as well.

  6. Dr-Irv commented on Feb 17, 2023

    @Dr-Irv
    Collaborator

    reopening so we can track if mypy ever fixes the bug

  7. reopened this on Feb 17, 2023
  8. Dr-Irv commented on Jan 3, 2025

    @Dr-Irv
    Collaborator

    With mypy 1.14.1, this now works correctly. Using the original code:

    play\issue516.py:7: note: Revealed type is "Union[pandas.core.series.Series[builtins.float], pandas.core.indexes.base.Index[builtins.int]]"
    play\issue516.py:9: note: Revealed type is "Union[numpy.dtype[numpy.generic[Any]], pandas.core.dtypes.base.ExtensionDtype]"
    play\issue516.py:11: note: Revealed type is "pandas.core.frame.DataFrame"
    play\issue516.py:13: note: Revealed type is "pandas.core.series.Series[Any]"
    

    Closing as a result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    mypy bugRequires mypy to fix a bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions