You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dd69475
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: doc/source/whatsnew/v3.1.0.rst
+50-20Lines changed: 50 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -147,36 +147,54 @@ is parallelized (:issue:`64347`).
147
147
148
148
Other enhancements
149
149
^^^^^^^^^^^^^^^^^^
150
-
- :class:`Period` now supports f-string formatting via ``__format__``, e.g. ``f"{period:%Y-%m}"`` (:issue:`48536`)
151
-
- :class:`Series` and :class:`DataFrame` with ``timedelta64`` dtype now aligns fractional seconds in string representation for easier reading (:issue:`57188`)
152
-
- :meth:`.DataFrameGroupBy.agg` now allows for the provided ``func`` to return a NumPy array (:issue:`63957`)
153
-
- :meth:`.DataFrameGroupBy.transform` now accepts list-like and dict arguments similar to :meth:`~.DataFrameGroupBy.agg`, and supports ``NamedFunc`` (alias of :class:`NamedAgg`) (:issue:`58318`)
150
+
151
+
New methods/arguments:
152
+
153
+
- Added ``union_categories`` parameter to :func:`concat` to preserve categorical dtype by unioning categories when concatenating categoricals with different categories (:issue:`14177`)
154
+
- Added :meth:`Index.replace` method to support value replacement functionality similar to :meth:`Series.replace` (:issue:`19495`)
155
+
156
+
I/O:
157
+
154
158
- :meth:`Series.to_json` now supports serializing custom ExtensionArrays (by correctly using the ``_values_for_json`` method of an ExtensionArray) (:issue:`65047`)
155
-
- :meth:`Timestamp.round`, :meth:`Timestamp.floor`, and :meth:`Timestamp.ceil` now officially accept :class:`Timedelta` arguments (:issue:`63687`)
156
-
- Added ``NamedFunc``, an alias to :class:`NamedAgg` for a more semantically accurate name when used with non-aggregation functions; either can accept arbitrary functions (:issue:`65164`)
157
-
- :meth:`.ExtensionArray.map` now calls ``ExtensionArray._cast_pointwise_result`` to retain the dtype backend, e.g. Arrow-backed arrays now preserve their Arrow dtype through ``map`` (:issue:`57189`, :issue:`62164`)
158
159
- :func:`read_csv` now supports ``dtype="complex64"`` and ``dtype="complex128"`` with the C engine, enabling round-tripping of complex-number columns written by :meth:`DataFrame.to_csv` (:issue:`9379`)
159
160
- :func:`read_excel` and :class:`ExcelFile` now infer ``engine`` from a workbook object created by a supported engine's library, such as an ``openpyxl.Workbook``, instead of raising (:issue:`46352`)
161
+
- Improved the precision of float parsing in :func:`read_csv` (:issue:`64395`)
162
+
163
+
Groupby/resample/rolling:
164
+
165
+
- :meth:`.DataFrameGroupBy.agg` now allows for the provided ``func`` to return a NumPy array (:issue:`63957`)
166
+
- :meth:`.DataFrameGroupBy.transform` now accepts list-like and dict arguments similar to :meth:`~.DataFrameGroupBy.agg`, and supports ``NamedFunc`` (alias of :class:`NamedAgg`) (:issue:`58318`)
167
+
- Added ``NamedFunc``, an alias to :class:`NamedAgg` for a more semantically accurate name when used with non-aggregation functions; either can accept arbitrary functions (:issue:`65164`)
168
+
169
+
Datetimelike:
170
+
171
+
- :class:`Period` now supports f-string formatting via ``__format__``, e.g. ``f"{period:%Y-%m}"`` (:issue:`48536`)
172
+
- :class:`Series` and :class:`DataFrame` with ``timedelta64`` dtype now aligns fractional seconds in string representation for easier reading (:issue:`57188`)
173
+
- :meth:`Timestamp.round`, :meth:`Timestamp.floor`, and :meth:`Timestamp.ceil` now officially accept :class:`Timedelta` arguments (:issue:`63687`)
160
174
- :func:`to_datetime` and ``strptime`` parsing now support the ``%N`` directive for matching exactly 9 digits representing nanoseconds, providing symmetry with ``strftime`` formatting (:issue:`65863`)
161
-
- :meth:`DataFrame.info` now respects the ``display.unicode.east_asian_width`` option when aligning column names (:issue:`46082`)
162
-
- :meth:`DataFrame.select_dtypes` can now select ``datetime64`` and ``timedelta64`` columns by a specific resolution (e.g. ``"datetime64[us]"``, ``np.dtype("timedelta64[ms]")``), which previously raised; a unit-specific spec matches only columns with exactly that resolution, while a unitless ``"datetime64"``/``"timedelta64"`` still matches every resolution (:issue:`40234`)
163
-
- :meth:`Series.round` now works on third-party numeric :class:`.ExtensionArray` types via a default :meth:`.ExtensionArray.round` (:issue:`49387`)
164
175
- :meth:`Timestamp.strftime` and the array-level :meth:`DatetimeIndex.strftime` / :meth:`Series.dt.strftime` now support a ``%N`` directive that formats nanoseconds as a 9-digit zero-padded number; the standard ``%f`` directive is unchanged and continues to format microseconds only (:issue:`29461`)
176
+
- Time strings with a space before ``AM``/``PM`` (e.g. ``"3:25:00 PM"``) are now recognized; :meth:`DataFrame.between_time` previously raised on them, and :meth:`DataFrame.at_time` warned that they would raise in a future version. Casting such strings to a pyarrow time dtype (``time32``/``time64``) no longer silently gives a missing value (:issue:`18793`)
177
+
- Vertical bar plots with a regular-frequency :class:`DatetimeIndex` or :class:`PeriodIndex` now format the x-axis with the same dynamic date tick labeling as line plots, instead of labeling every bar with the full string representation of its timestamp. The tick labels of such plots are no longer rotated by default, matching line plots; pass ``rot`` to override. Bar plots over a frequency the date ticks cannot be aligned to keep the previous labeling (:issue:`1918`)
178
+
179
+
ExtensionArray:
180
+
181
+
- :meth:`.ExtensionArray.map` now calls ``ExtensionArray._cast_pointwise_result`` to retain the dtype backend, e.g. Arrow-backed arrays now preserve their Arrow dtype through ``map`` (:issue:`57189`, :issue:`62164`)
182
+
- :meth:`Series.round` now works on third-party numeric :class:`.ExtensionArray` types via a default :meth:`.ExtensionArray.round` (:issue:`49387`)
- Added :meth:`.ExtensionArray.sort` for in-place sorting of :class:`.ExtensionArray` (:issue:`64977`)
167
185
- Added ``ExtensionDtype._get_plot_converter`` allowing :class:`.ExtensionDtype` to register their own :class:`matplotlib.units.ConversionInterface` to support plotting for non-numeric :class:`.ExtensionArray` (:issue:`64300`)
168
-
- Added :meth:`Index.replace` method to support value replacement functionality similar to :meth:`Series.replace` (:issue:`19495`)
169
-
- Added ``SparseArray.from_indices`` to construct a :class:`.SparseArray` from sparse values and their indices without materializing a dense array (:issue:`43628`)
170
-
- Added ``union_categories`` parameter to :func:`concat` to preserve categorical dtype by unioning categories when concatenating categoricals with different categories (:issue:`14177`)
171
186
- Added reduction methods as public API on pandas-implemented extension arrays where applicable (:issue:`63512`)
187
+
188
+
Other:
189
+
190
+
- :meth:`DataFrame.select_dtypes` can now select ``datetime64`` and ``timedelta64`` columns by a specific resolution (e.g. ``"datetime64[us]"``, ``np.dtype("timedelta64[ms]")``), which previously raised; a unit-specific spec matches only columns with exactly that resolution, while a unitless ``"datetime64"``/``"timedelta64"`` still matches every resolution (:issue:`40234`)
191
+
- :meth:`DataFrame.info` now respects the ``display.unicode.east_asian_width`` option when aligning column names (:issue:`46082`)
172
192
- Display formatting for float sequences in DataFrame cells now respects the ``display.precision`` option (:issue:`60503`).
173
-
- Improved the error message raised when constructing an :class:`IntervalIndex` or :class:`~arrays.IntervalArray` with a left side greater than the right side, which now shows the offending intervals and their positions (:issue:`66807`)
174
-
- Improved the precision of float parsing in :func:`read_csv` (:issue:`64395`)
175
193
- Improved the string ``repr`` of :class:`.SparseArray` (:issue:`64547`)
194
+
- Added ``SparseArray.from_indices`` to construct a :class:`.SparseArray` from sparse values and their indices without materializing a dense array (:issue:`43628`)
195
+
- Improved the error message raised when constructing an :class:`IntervalIndex` or :class:`~arrays.IntervalArray` with a left side greater than the right side, which now shows the offending intervals and their positions (:issue:`66807`)
176
196
- Improved type inference of comparison and arithmetic operators on :class:`Series` and :class:`DataFrame` for static type checkers (e.g. ``ser == "a"`` is now inferred as :class:`Series` instead of ``Any``) (:issue:`40762`)
177
197
- Setting values with :meth:`DataFrame.at` or :meth:`Series.at` using a non-scalar indexer (e.g. a boolean mask, list, or array) now raises a clearer ``InvalidIndexError`` directing users to ``.loc`` (:issue:`51866`)
178
-
- Time strings with a space before ``AM``/``PM`` (e.g. ``"3:25:00 PM"``) are now recognized; :meth:`DataFrame.between_time` previously raised on them, and :meth:`DataFrame.at_time` warned that they would raise in a future version. Casting such strings to a pyarrow time dtype (``time32``/``time64``) no longer silently gives a missing value (:issue:`18793`)
179
-
- Vertical bar plots with a regular-frequency :class:`DatetimeIndex` or :class:`PeriodIndex` now format the x-axis with the same dynamic date tick labeling as line plots, instead of labeling every bar with the full string representation of its timestamp. The tick labels of such plots are no longer rotated by default, matching line plots; pass ``rot`` to override. Bar plots over a frequency the date ticks cannot be aligned to keep the previous labeling (:issue:`1918`)
The introduction of the default :class:`StringDtype`, using PyArrow under the hood when
289
+
installed, in pandas 3.0 led to certain behaviour changes for string methods using regex
290
+
operations (because PyArrow using a different regex flavor) (:issue:`63683`).
291
+
Certain of those cases have been fixed in 3.1 to ensure the string methods'
292
+
behaviour is consistent again with Python's ``re`` module (and with pandas 2.x):
293
+
294
+
- Bug where ``\w``, ``\d``, ``\s`` and ``\b`` (and their negations) in a regular expression only matched ASCII characters. Now they also match unicode characters again (:issue:`70770`)
295
+
- Bug using a regex pattern with ``$`` to indicate the end of the string: when using PyArrow-backed string dtypes, for certain cases this no longer matched before or after an ending newline; now it is consistent again with Python's ``re`` behaviour in all cases (:issue:`63683`)
296
+
- Bug using a regex pattern with unicode escape sequences (``\u``/``\U``/``\N{}``) with PyArrow-backed string dtypes failing, now working again as in pandas 2.x (:issue:`63901`)
- Bug in :meth:`Series.str.normalize`, :meth:`Series.str.zfill` and their :class:`Index` counterparts with PyArrow-backed string dtypes raising ``ArrowInvalid`` or returning a ``null`` dtype when the object was empty, the values were all missing, or the missing values were grouped into a chunk of their own (:issue:`66822`)
973
1006
- Bug in addition and multiplication on length-zero ``string``-dtype :class:`Series` and :class:`Index` returning ``object`` dtype instead of the string dtype, and raising ``TypeError`` when the other operand was length-zero with a different string storage or ``object`` dtype (:issue:`40624`)
974
1007
- Bug in :meth:`Series.str.match` raising for regex flags other than ``re.IGNORECASE`` (e.g. ``re.MULTILINE``, ``re.DOTALL``, ``re.ASCII``), whether passed via ``flags`` or carried by a compiled ``re.Pattern`` (:issue:`63108`, :issue:`66348`)
975
-
- Bug in :meth:`Series.str.contains`, :meth:`~Series.str.match`, :meth:`~Series.str.fullmatch`, :meth:`~Series.str.replace` and :meth:`~Series.str.count` for PyArrow-backed :class:`StringDtype` where ``\w``, ``\d``, ``\s`` and ``\b`` (and their negations) in a regular expression only matched ASCII characters (:issue:`70770`)
976
1008
- Bug in :meth:`Series.str.contains`, :meth:`~Series.str.match`, :meth:`~Series.str.fullmatch`, :meth:`~Series.str.replace` and :meth:`~Series.str.count` for PyArrow-backed :class:`StringDtype` where a lookaround or backreference inside a repeat such as ``(?:a(?=b))+`` raised an error (:issue:`70770`)
977
1009
- Bug in :meth:`Series.str.match` where passing ``flags=0`` was not equivalent to omitting ``flags``, raising for patterns using PyArrow-only regex syntax such as ``\p{L}`` or an inline flag such as ``(?i)`` (:issue:`63108`)
978
1010
- Bug in :meth:`Series.str.match`, :meth:`~Series.str.fullmatch`, :meth:`~Series.str.contains`, :meth:`~Series.str.count`, :meth:`~Series.str.extract`, :meth:`~Series.str.replace` and their :class:`Index` counterparts with :class:`ArrowDtype` raising instead of honoring ``flags`` such as ``re.ASCII``, or the flags carried by a compiled ``re.Pattern`` (:issue:`66348`)
@@ -981,8 +1013,6 @@ Strings
981
1013
- Bug in :meth:`Series.str.split` with :class:`ArrowDtype` ``string`` not inferring regex for multi-character patterns when ``regex=None``, causing the pattern to be treated as a literal instead of a regular expression (:issue:`58321`)
982
1014
- Bug in :meth:`~Series.unique` and :func:`factorize` on object dtype returning incorrect results when the values were not UTF-8 encodable, e.g. lone surrogates (:issue:`34550`)
983
1015
- Bug in :meth:`~Series.unique`, :meth:`~Series.nunique`, :func:`factorize` and ``groupby`` on object dtype, and on ``str``/``string`` dtype backed by python storage, collapsing distinct strings that are identical up to an embedded NUL byte, e.g. ``""`` and ``"\x00"``, into a single value (:issue:`34551`)
984
-
- Bug in string methods using a regex pattern with ``$`` to indicate the end of the string: when using PyArrow-backed string dtypes, for certain cases this no longer matched before or after an ending newline; now it is consistent again with Python's ``re`` behaviour in all cases (:issue:`63683`)
985
-
- Bug in string methods using a regex pattern with unicode escape sequences (``\u``/``\U``/``\N{}``) with PyArrow-backed string dtypes failing, now working again as in pandas 2.x (:issue:`63901`)
0 commit comments