Skip to content

Commit 3188ced

Browse files
[backport 3.0.x] BUG: read_csv(sep=None) raised TypeError instead of falling back to the python engine (GH#66639) (#66790) (#68970)
Co-authored-by: jbrockmendel <[email protected]>
1 parent f097905 commit 3188ced

5 files changed

Lines changed: 53 additions & 30 deletions

File tree

‎doc/source/user_guide/io.rst‎

Lines changed: 9 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -65,12 +65,12 @@ filepath_or_buffer : various
6565
locations), or any object with a ``read()`` method (such as an open file or
6666
:class:`~python:io.StringIO`).
6767
sep : str, defaults to ``','`` for :func:`read_csv`, ``\t`` for :func:`read_table`
68-
Delimiter to use. If sep is ``None``, the C engine cannot automatically detect
69-
the separator, but the Python parsing engine can, meaning the latter will be
70-
used and automatically detect the separator by Python's builtin sniffer tool,
71-
:class:`python:csv.Sniffer`. In addition, separators longer than 1 character and
72-
different from ``'\s+'`` will be interpreted as regular expressions and
73-
will also force the use of the Python parsing engine. Note that regex
68+
Delimiter to use. ``sep=None`` detects the separator from the first valid row
69+
of the file with Python's builtin sniffer tool, :class:`python:csv.Sniffer`; it
70+
is supported only by the Python parsing engine, which will be used
71+
automatically. In addition, separators longer than 1 character
72+
and different from ``'\s+'`` will be interpreted as regular expressions and
73+
will force the use of the Python parsing engine. Note that regex
7474
delimiters are prone to ignoring quoted data. Regex example: ``'\\r\\t'``.
7575
delimiter : str, default ``None``
7676
Alternative argument name for sep.
@@ -1418,7 +1418,9 @@ Automatically "sniffing" the delimiter
14181418

14191419
``read_csv`` is capable of inferring delimited (not necessarily
14201420
comma-separated) files, as pandas uses the :class:`python:csv.Sniffer`
1421-
class of the csv module. For this, you have to specify ``sep=None``.
1421+
class of the csv module. For this, you have to specify ``sep=None``. Passing
1422+
``engine='python'`` as well avoids the ``ParserWarning`` raised by the
1423+
fallback.
14221424

14231425
.. ipython:: python
14241426
@@ -1495,12 +1497,6 @@ Currently, options unsupported by the C and pyarrow engines include:
14951497

14961498
* ``sep`` other than a single character (e.g. regex separators)
14971499
* ``skipfooter``
1498-
1499-
Specifying any of the above options will produce a ``ParserWarning`` unless the
1500-
python engine is selected explicitly using ``engine='python'``.
1501-
1502-
Options that are unsupported by the pyarrow engine which are not covered by the list above include:
1503-
15041500
* ``float_precision``
15051501
* ``chunksize``
15061502
* ``comment``

‎doc/source/whatsnew/v3.0.6.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ Fixed regressions
3131
select the columns into a 1-column DataFrame with an extension dtype, which
3232
surfaced as a regression for a DataFrame with string data (:issue:`66527`,
3333
:issue:`66255`)
34+
- Fixed a regression in :func:`read_csv` where ``sep=None`` raised ``TypeError: object of type 'NoneType' has no len()`` instead of sniffing the separator with the ``python`` engine; passing ``sep=None`` together with ``engine="c"`` or ``engine="pyarrow"``, which cannot sniff the separator, again raises an informative ``ValueError`` instead of raising ``TypeError`` or silently parsing each line as a single column (:issue:`66639`)
3435
- Fixed a regression in :meth:`Series.interpolate` with ``method="linear"`` and a pyarrow-backed dtype leaving consecutive and trailing missing values unfilled, and truncating interpolated values for integer dtypes; the result now matches the equivalent masked (e.g. ``Int64``) dtype (:issue:`65345`)
3536

3637
.. ---------------------------------------------------------------------------

‎pandas/io/parsers/readers.py‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -425,11 +425,10 @@ def read_csv(
425425
By file-like object, we refer to objects with a ``read()`` method, such as
426426
a file handle (e.g. via builtin ``open`` function) or ``StringIO``.
427427
sep : str, default ','
428-
Character or regex pattern to treat as the delimiter. If ``sep=None``, the
429-
C engine cannot automatically detect
430-
the separator, but the Python parsing engine can, meaning the latter will
431-
be used and automatically detect the separator from only the first valid
432-
row of the file by Python's builtin sniffer tool, ``csv.Sniffer``.
428+
Character or regex pattern to treat as the delimiter. ``sep=None`` detects
429+
the separator from the first valid row of the file with Python's builtin
430+
sniffer tool, ``csv.Sniffer``; it is supported only by the Python parsing
431+
engine, which will be used automatically.
433432
In addition, separators longer than 1 character and different from
434433
``'\\s+'`` will be interpreted as regular expressions and will also force
435434
the use of the Python parsing engine. Note that regex delimiters are prone
@@ -992,11 +991,10 @@ def read_table(
992991
By file-like object, we refer to objects with a ``read()`` method, such as
993992
a file handle (e.g. via builtin ``open`` function) or ``StringIO``.
994993
sep : str, default '\\t' (tab-stop)
995-
Character or regex pattern to treat as the delimiter. If ``sep=None``, the
996-
C engine cannot automatically detect
997-
the separator, but the Python parsing engine can, meaning the latter will
998-
be used and automatically detect the separator from only the first valid
999-
row of the file by Python's builtin sniffer tool, ``csv.Sniffer``.
994+
Character or regex pattern to treat as the delimiter. ``sep=None`` detects
995+
the separator from the first valid row of the file with Python's builtin
996+
sniffer tool, ``csv.Sniffer``; it is supported only by the Python parsing
997+
engine, which will be used automatically.
1000998
In addition, separators longer than 1 character and different from
1001999
``'\\s+'`` will be interpreted as regular expressions and will also force
10021000
the use of the Python parsing engine. Note that regex delimiters are prone
@@ -1731,7 +1729,12 @@ def _clean_options(
17311729

17321730
sep = options["delimiter"]
17331731

1734-
if sep is not None and len(sep) > 1:
1732+
if sep is None:
1733+
# sniffing the separator with csv.Sniffer is python-engine only
1734+
if engine in ("c", "pyarrow"):
1735+
fallback_reason = f"the '{engine}' engine does not support sep=None"
1736+
engine = "python"
1737+
elif len(sep) > 1:
17351738
if engine == "c" and sep == r"\s+":
17361739
# delim_whitespace passed on to pandas._libs.parsers.TextReader
17371740
result["delim_whitespace"] = True
@@ -1744,7 +1747,7 @@ def _clean_options(
17441747
r"different from '\s+' are interpreted as regex)"
17451748
)
17461749
engine = "python"
1747-
elif sep is not None:
1750+
else:
17481751
encodeable = True
17491752
encoding = sys.getfilesystemencoding() or "utf-8"
17501753
try:

‎pandas/tests/io/parser/common/test_read_errors.py‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -252,11 +252,8 @@ def test_open_file(all_parsers, temp_file):
252252

253253
msg = "Could not determine delimiter"
254254
err = csv.Error
255-
if parser.engine == "c":
256-
msg = "object of type 'NoneType' has no len"
257-
err = TypeError
258-
elif parser.engine == "pyarrow":
259-
msg = "'utf-8' codec can't decode byte 0xe4"
255+
if parser.engine in ("c", "pyarrow"):
256+
msg = f"the '{parser.engine}' engine does not support sep=None"
260257
err = ValueError
261258

262259
file = Path(temp_file)

‎pandas/tests/io/parser/test_unsupported.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515

1616
from pandas.errors import ParserError
1717

18+
from pandas import DataFrame
1819
import pandas._testing as tm
1920

2021
from pandas.io.parsers import read_csv
@@ -197,6 +198,31 @@ def test_invalid_file_inputs(request, all_parsers):
197198
parser.read_csv([])
198199

199200

201+
def test_sep_none_falls_back_to_python_engine():
202+
# GH#66639 sniffing the separator is python-engine only, so the default
203+
# engine falls back to it rather than handing sep=None to the C parser
204+
data = "a;b\n1;2\n"
205+
expected = DataFrame({"a": [1], "b": [2]})
206+
207+
with tm.assert_produces_warning(parsers.ParserWarning, match="sep=None"):
208+
result = read_csv(StringIO(data), sep=None)
209+
tm.assert_frame_equal(result, expected)
210+
211+
with tm.assert_produces_warning(None):
212+
result = read_csv(StringIO(data), sep=None, engine="python")
213+
tm.assert_frame_equal(result, expected)
214+
215+
216+
@pytest.mark.parametrize("engine", ["c", "pyarrow"])
217+
def test_sep_none_explicit_engine_raises(engine):
218+
# GH#66639 an explicitly requested engine that cannot sniff the separator
219+
# reports that rather than falling back
220+
msg = f"the '{engine}' engine does not support sep=None"
221+
222+
with pytest.raises(ValueError, match=msg):
223+
read_csv(StringIO("a;b\n1;2\n"), sep=None, engine=engine)
224+
225+
200226
def test_invalid_dtype_backend(all_parsers):
201227
parser = all_parsers
202228
msg = (

0 commit comments

Comments
 (0)