Skip to content

Commit dd69475

Browse files
Backport PR #71468 on branch 3.1.x (DOC: 3.1 whatsnew: add some structure to Other enhancements + add string regex bug section) (#71469)
Co-authored-by: Joris Van den Bossche <[email protected]>
1 parent 4ccc19a commit dd69475

1 file changed

Lines changed: 50 additions & 20 deletions

File tree

‎doc/source/whatsnew/v3.1.0.rst‎

Lines changed: 50 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -147,36 +147,54 @@ is parallelized (:issue:`64347`).
147147

148148
Other enhancements
149149
^^^^^^^^^^^^^^^^^^
150-
- :class:`Period` now supports f-string formatting via ``__format__``, e.g. ``f"{period:%Y-%m}"`` (:issue:`48536`)
151-
- :class:`Series` and :class:`DataFrame` with ``timedelta64`` dtype now aligns fractional seconds in string representation for easier reading (:issue:`57188`)
152-
- :meth:`.DataFrameGroupBy.agg` now allows for the provided ``func`` to return a NumPy array (:issue:`63957`)
153-
- :meth:`.DataFrameGroupBy.transform` now accepts list-like and dict arguments similar to :meth:`~.DataFrameGroupBy.agg`, and supports ``NamedFunc`` (alias of :class:`NamedAgg`) (:issue:`58318`)
150+
151+
New methods/arguments:
152+
153+
- Added ``union_categories`` parameter to :func:`concat` to preserve categorical dtype by unioning categories when concatenating categoricals with different categories (:issue:`14177`)
154+
- Added :meth:`Index.replace` method to support value replacement functionality similar to :meth:`Series.replace` (:issue:`19495`)
155+
156+
I/O:
157+
154158
- :meth:`Series.to_json` now supports serializing custom ExtensionArrays (by correctly using the ``_values_for_json`` method of an ExtensionArray) (:issue:`65047`)
155-
- :meth:`Timestamp.round`, :meth:`Timestamp.floor`, and :meth:`Timestamp.ceil` now officially accept :class:`Timedelta` arguments (:issue:`63687`)
156-
- Added ``NamedFunc``, an alias to :class:`NamedAgg` for a more semantically accurate name when used with non-aggregation functions; either can accept arbitrary functions (:issue:`65164`)
157-
- :meth:`.ExtensionArray.map` now calls ``ExtensionArray._cast_pointwise_result`` to retain the dtype backend, e.g. Arrow-backed arrays now preserve their Arrow dtype through ``map`` (:issue:`57189`, :issue:`62164`)
158159
- :func:`read_csv` now supports ``dtype="complex64"`` and ``dtype="complex128"`` with the C engine, enabling round-tripping of complex-number columns written by :meth:`DataFrame.to_csv` (:issue:`9379`)
159160
- :func:`read_excel` and :class:`ExcelFile` now infer ``engine`` from a workbook object created by a supported engine's library, such as an ``openpyxl.Workbook``, instead of raising (:issue:`46352`)
161+
- Improved the precision of float parsing in :func:`read_csv` (:issue:`64395`)
162+
163+
Groupby/resample/rolling:
164+
165+
- :meth:`.DataFrameGroupBy.agg` now allows for the provided ``func`` to return a NumPy array (:issue:`63957`)
166+
- :meth:`.DataFrameGroupBy.transform` now accepts list-like and dict arguments similar to :meth:`~.DataFrameGroupBy.agg`, and supports ``NamedFunc`` (alias of :class:`NamedAgg`) (:issue:`58318`)
167+
- Added ``NamedFunc``, an alias to :class:`NamedAgg` for a more semantically accurate name when used with non-aggregation functions; either can accept arbitrary functions (:issue:`65164`)
168+
169+
Datetimelike:
170+
171+
- :class:`Period` now supports f-string formatting via ``__format__``, e.g. ``f"{period:%Y-%m}"`` (:issue:`48536`)
172+
- :class:`Series` and :class:`DataFrame` with ``timedelta64`` dtype now aligns fractional seconds in string representation for easier reading (:issue:`57188`)
173+
- :meth:`Timestamp.round`, :meth:`Timestamp.floor`, and :meth:`Timestamp.ceil` now officially accept :class:`Timedelta` arguments (:issue:`63687`)
160174
- :func:`to_datetime` and ``strptime`` parsing now support the ``%N`` directive for matching exactly 9 digits representing nanoseconds, providing symmetry with ``strftime`` formatting (:issue:`65863`)
161-
- :meth:`DataFrame.info` now respects the ``display.unicode.east_asian_width`` option when aligning column names (:issue:`46082`)
162-
- :meth:`DataFrame.select_dtypes` can now select ``datetime64`` and ``timedelta64`` columns by a specific resolution (e.g. ``"datetime64[us]"``, ``np.dtype("timedelta64[ms]")``), which previously raised; a unit-specific spec matches only columns with exactly that resolution, while a unitless ``"datetime64"``/``"timedelta64"`` still matches every resolution (:issue:`40234`)
163-
- :meth:`Series.round` now works on third-party numeric :class:`.ExtensionArray` types via a default :meth:`.ExtensionArray.round` (:issue:`49387`)
164175
- :meth:`Timestamp.strftime` and the array-level :meth:`DatetimeIndex.strftime` / :meth:`Series.dt.strftime` now support a ``%N`` directive that formats nanoseconds as a 9-digit zero-padded number; the standard ``%f`` directive is unchanged and continues to format microseconds only (:issue:`29461`)
176+
- Time strings with a space before ``AM``/``PM`` (e.g. ``"3:25:00 PM"``) are now recognized; :meth:`DataFrame.between_time` previously raised on them, and :meth:`DataFrame.at_time` warned that they would raise in a future version. Casting such strings to a pyarrow time dtype (``time32``/``time64``) no longer silently gives a missing value (:issue:`18793`)
177+
- Vertical bar plots with a regular-frequency :class:`DatetimeIndex` or :class:`PeriodIndex` now format the x-axis with the same dynamic date tick labeling as line plots, instead of labeling every bar with the full string representation of its timestamp. The tick labels of such plots are no longer rotated by default, matching line plots; pass ``rot`` to override. Bar plots over a frequency the date ticks cannot be aligned to keep the previous labeling (:issue:`1918`)
178+
179+
ExtensionArray:
180+
181+
- :meth:`.ExtensionArray.map` now calls ``ExtensionArray._cast_pointwise_result`` to retain the dtype backend, e.g. Arrow-backed arrays now preserve their Arrow dtype through ``map`` (:issue:`57189`, :issue:`62164`)
182+
- :meth:`Series.round` now works on third-party numeric :class:`.ExtensionArray` types via a default :meth:`.ExtensionArray.round` (:issue:`49387`)
165183
- Added :meth:`.ExtensionArray.count` (:issue:`64450`)
166184
- Added :meth:`.ExtensionArray.sort` for in-place sorting of :class:`.ExtensionArray` (:issue:`64977`)
167185
- Added ``ExtensionDtype._get_plot_converter`` allowing :class:`.ExtensionDtype` to register their own :class:`matplotlib.units.ConversionInterface` to support plotting for non-numeric :class:`.ExtensionArray` (:issue:`64300`)
168-
- Added :meth:`Index.replace` method to support value replacement functionality similar to :meth:`Series.replace` (:issue:`19495`)
169-
- Added ``SparseArray.from_indices`` to construct a :class:`.SparseArray` from sparse values and their indices without materializing a dense array (:issue:`43628`)
170-
- Added ``union_categories`` parameter to :func:`concat` to preserve categorical dtype by unioning categories when concatenating categoricals with different categories (:issue:`14177`)
171186
- Added reduction methods as public API on pandas-implemented extension arrays where applicable (:issue:`63512`)
187+
188+
Other:
189+
190+
- :meth:`DataFrame.select_dtypes` can now select ``datetime64`` and ``timedelta64`` columns by a specific resolution (e.g. ``"datetime64[us]"``, ``np.dtype("timedelta64[ms]")``), which previously raised; a unit-specific spec matches only columns with exactly that resolution, while a unitless ``"datetime64"``/``"timedelta64"`` still matches every resolution (:issue:`40234`)
191+
- :meth:`DataFrame.info` now respects the ``display.unicode.east_asian_width`` option when aligning column names (:issue:`46082`)
172192
- Display formatting for float sequences in DataFrame cells now respects the ``display.precision`` option (:issue:`60503`).
173-
- Improved the error message raised when constructing an :class:`IntervalIndex` or :class:`~arrays.IntervalArray` with a left side greater than the right side, which now shows the offending intervals and their positions (:issue:`66807`)
174-
- Improved the precision of float parsing in :func:`read_csv` (:issue:`64395`)
175193
- Improved the string ``repr`` of :class:`.SparseArray` (:issue:`64547`)
194+
- Added ``SparseArray.from_indices`` to construct a :class:`.SparseArray` from sparse values and their indices without materializing a dense array (:issue:`43628`)
195+
- Improved the error message raised when constructing an :class:`IntervalIndex` or :class:`~arrays.IntervalArray` with a left side greater than the right side, which now shows the offending intervals and their positions (:issue:`66807`)
176196
- Improved type inference of comparison and arithmetic operators on :class:`Series` and :class:`DataFrame` for static type checkers (e.g. ``ser == "a"`` is now inferred as :class:`Series` instead of ``Any``) (:issue:`40762`)
177197
- Setting values with :meth:`DataFrame.at` or :meth:`Series.at` using a non-scalar indexer (e.g. a boolean mask, list, or array) now raises a clearer ``InvalidIndexError`` directing users to ``.loc`` (:issue:`51866`)
178-
- Time strings with a space before ``AM``/``PM`` (e.g. ``"3:25:00 PM"``) are now recognized; :meth:`DataFrame.between_time` previously raised on them, and :meth:`DataFrame.at_time` warned that they would raise in a future version. Casting such strings to a pyarrow time dtype (``time32``/``time64``) no longer silently gives a missing value (:issue:`18793`)
179-
- Vertical bar plots with a regular-frequency :class:`DatetimeIndex` or :class:`PeriodIndex` now format the x-axis with the same dynamic date tick labeling as line plots, instead of labeling every bar with the full string representation of its timestamp. The tick labels of such plots are no longer rotated by default, matching line plots; pass ``rot`` to override. Bar plots over a frequency the date ticks cannot be aligned to keep the previous labeling (:issue:`1918`)
180198

181199
.. ---------------------------------------------------------------------------
182200
.. _whatsnew_310.notable_bug_fixes:
@@ -262,6 +280,21 @@ A positional ``tzinfo`` (the eighth argument, as in :class:`datetime.datetime`)
262280
is now attached to the result instead of being dropped, so :func:`copy.replace`
263281
also keeps the timezone (:issue:`31930`, :issue:`45307`).
264282

283+
.. _whatsnew_310.notable_bug_fixes.string_regex:
284+
285+
String regex methods aligning again with Python's ``re`` module
286+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
287+
288+
The introduction of the default :class:`StringDtype`, using PyArrow under the hood when
289+
installed, in pandas 3.0 led to certain behaviour changes for string methods using regex
290+
operations (because PyArrow using a different regex flavor) (:issue:`63683`).
291+
Certain of those cases have been fixed in 3.1 to ensure the string methods'
292+
behaviour is consistent again with Python's ``re`` module (and with pandas 2.x):
293+
294+
- Bug where ``\w``, ``\d``, ``\s`` and ``\b`` (and their negations) in a regular expression only matched ASCII characters. Now they also match unicode characters again (:issue:`70770`)
295+
- Bug using a regex pattern with ``$`` to indicate the end of the string: when using PyArrow-backed string dtypes, for certain cases this no longer matched before or after an ending newline; now it is consistent again with Python's ``re`` behaviour in all cases (:issue:`63683`)
296+
- Bug using a regex pattern with unicode escape sequences (``\u``/``\U``/``\N{}``) with PyArrow-backed string dtypes failing, now working again as in pandas 2.x (:issue:`63901`)
297+
265298
.. ---------------------------------------------------------------------------
266299
.. _whatsnew_310.api_breaking:
267300

@@ -972,7 +1005,6 @@ Strings
9721005
- Bug in :meth:`Series.str.normalize`, :meth:`Series.str.zfill` and their :class:`Index` counterparts with PyArrow-backed string dtypes raising ``ArrowInvalid`` or returning a ``null`` dtype when the object was empty, the values were all missing, or the missing values were grouped into a chunk of their own (:issue:`66822`)
9731006
- Bug in addition and multiplication on length-zero ``string``-dtype :class:`Series` and :class:`Index` returning ``object`` dtype instead of the string dtype, and raising ``TypeError`` when the other operand was length-zero with a different string storage or ``object`` dtype (:issue:`40624`)
9741007
- Bug in :meth:`Series.str.match` raising for regex flags other than ``re.IGNORECASE`` (e.g. ``re.MULTILINE``, ``re.DOTALL``, ``re.ASCII``), whether passed via ``flags`` or carried by a compiled ``re.Pattern`` (:issue:`63108`, :issue:`66348`)
975-
- Bug in :meth:`Series.str.contains`, :meth:`~Series.str.match`, :meth:`~Series.str.fullmatch`, :meth:`~Series.str.replace` and :meth:`~Series.str.count` for PyArrow-backed :class:`StringDtype` where ``\w``, ``\d``, ``\s`` and ``\b`` (and their negations) in a regular expression only matched ASCII characters (:issue:`70770`)
9761008
- Bug in :meth:`Series.str.contains`, :meth:`~Series.str.match`, :meth:`~Series.str.fullmatch`, :meth:`~Series.str.replace` and :meth:`~Series.str.count` for PyArrow-backed :class:`StringDtype` where a lookaround or backreference inside a repeat such as ``(?:a(?=b))+`` raised an error (:issue:`70770`)
9771009
- Bug in :meth:`Series.str.match` where passing ``flags=0`` was not equivalent to omitting ``flags``, raising for patterns using PyArrow-only regex syntax such as ``\p{L}`` or an inline flag such as ``(?i)`` (:issue:`63108`)
9781010
- Bug in :meth:`Series.str.match`, :meth:`~Series.str.fullmatch`, :meth:`~Series.str.contains`, :meth:`~Series.str.count`, :meth:`~Series.str.extract`, :meth:`~Series.str.replace` and their :class:`Index` counterparts with :class:`ArrowDtype` raising instead of honoring ``flags`` such as ``re.ASCII``, or the flags carried by a compiled ``re.Pattern`` (:issue:`66348`)
@@ -981,8 +1013,6 @@ Strings
9811013
- Bug in :meth:`Series.str.split` with :class:`ArrowDtype` ``string`` not inferring regex for multi-character patterns when ``regex=None``, causing the pattern to be treated as a literal instead of a regular expression (:issue:`58321`)
9821014
- Bug in :meth:`~Series.unique` and :func:`factorize` on object dtype returning incorrect results when the values were not UTF-8 encodable, e.g. lone surrogates (:issue:`34550`)
9831015
- Bug in :meth:`~Series.unique`, :meth:`~Series.nunique`, :func:`factorize` and ``groupby`` on object dtype, and on ``str``/``string`` dtype backed by python storage, collapsing distinct strings that are identical up to an embedded NUL byte, e.g. ``""`` and ``"\x00"``, into a single value (:issue:`34551`)
984-
- Bug in string methods using a regex pattern with ``$`` to indicate the end of the string: when using PyArrow-backed string dtypes, for certain cases this no longer matched before or after an ending newline; now it is consistent again with Python's ``re`` behaviour in all cases (:issue:`63683`)
985-
- Bug in string methods using a regex pattern with unicode escape sequences (``\u``/``\U``/``\N{}``) with PyArrow-backed string dtypes failing, now working again as in pandas 2.x (:issue:`63901`)
9861016

9871017
Interval
9881018
^^^^^^^^

0 commit comments

Comments
 (0)