Skip to content

BUG: pd.read_html() is broken with beautifulsoup4 4.14.0 #62492

Closed

Description

@Dr-Irv

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
DF = pd.DataFrame({"a": [1, 2, 3], "b": [0.0, 0.0, 0.0]})
DF.to_html("foo.html")
pd.read_html("foo.html", flavor=["bs4"])

Issue Description

The above code works fine with beautifulsoup4 version 4.13.5.
It breaks with beautiifulsoup4 version 4.14.0 that was released on 9/27/25

So we either need to pin beautifulsoup4 or fix the bug.

Expected Behavior

No bug with pandas 2.3.2

Installed Versions

Details

INSTALLED VERSIONS

commit : 4665c10
python : 3.11.13
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.26100
machine : AMD64
processor : Intel64 Family 6 Model 183 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.3.2
numpy : 2.3.3
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 25.1
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.14.0
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : 1.1
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : 6.0.2
matplotlib : 3.10.6
numba : None
numexpr : 2.13.0
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 21.0.0
pyreadstat : 1.3.1
pytest : 8.4.2
python-calamine : None
pyxlsb : 1.0.10
s3fs : None
scipy : 1.16.2
sqlalchemy : 2.0.43
tables : 3.10.2
tabulate : 0.9.0
xarray : 2025.9.0
xlrd : 2.0.2
xlsxwriter : 3.2.9
zstandard : 0.24.0
tzdata : 2025.2
qtpy : None
pyqt5 : None

Activity

  1. added
    Needs TriageIssue that has not been reviewed by a pandas team member
    on Sep 29, 2025
  2. neutrinoceros commented on Sep 29, 2025

    @neutrinoceros
    Contributor

    This is affecting astropy's CI (xref: astropy/astropy#18646)
    and I noticed that gh-58086 already contained a test that could have caught this change. @Dr-Irv's reproducer is simpler so I suggest using it instead, but it's frustrating that this functionality is seemingly still not being tested.

  3. neutrinoceros commented on Sep 29, 2025

    @neutrinoceros
    Contributor

    (FWIW I'm working on a bisection, and hopefully a fix will ensue. Worst case scenario I'll report this to bs4 devs)

  4. neutrinoceros commented on Sep 29, 2025

    @neutrinoceros
    Contributor

    seems like a clear regression to me, so I'll report there

  5. neutrinoceros commented on Sep 29, 2025

    @neutrinoceros
    Contributor

    upstream report: https://bugs.launchpad.net/beautifulsoup/+bug/2125906

    as I mentioned there, I'll let bs4's maintainers decide wether they want to address this on their side or not. If they don't, I note that a fix is also possible on pandas' side, albeit a bit ugly.

  6. leonardr commented on Sep 29, 2025

    @leonardr

    The example code works (or at least doesn't raise an exception) using the recently released Beautiful Soup 4.14.2.

    >>> import pandas as pd
    >>> DF = pd.DataFrame({"a": [1, 2, 3], "b": [0.0, 0.0, 0.0]})
    >>> DF.to_html("foo.html")
    >>> pd.read_html("foo.html", flavor=["bs4"])
    [   Unnamed: 0  a    b
    0           0  1  0.0
    1           1  2  0.0
    2           2  3  0.0]
    
  7. neutrinoceros commented on Sep 29, 2025

    @neutrinoceros
    Contributor

    Indeed it does. I suggest this issue could be closed by the addition of a test. Thanks @leonardr !

  8. Dr-Irv commented on Sep 29, 2025

    @Dr-Irv
    ContributorAuthor

    Indeed it does. I suggest this issue could be closed by the addition of a test. Thanks @leonardr !

    I think some YAML files need to be updated to make sure that beautifulsoup4 4.14.0 and 4.14.1 don't get installed, or just change it to have beautifulsoup4 >= 4.14.2

  9. mroeschke commented on Sep 29, 2025

    @mroeschke
    Member

    4.14.0 and 4.14.1 don't get installed

    We normally don't pin dependencies to avoid a bug in an optional dependency. In a unit testing context, the recommendation is just to xfail or skip the test based on the failing dependency version.

  10. rhshadrach commented on Sep 30, 2025

    @rhshadrach
    Member

    The linked bug report to Beautiful Soup indicates the issue has been resolved upstream. Closing. Thanks @neutrinoceros and @leonardr

  11. added
    IO HTMLread_html, to_html, Styler.apply, Styler.applymap
    Upstream issueIssue related to pandas dependency
    and removed
    Needs TriageIssue that has not been reviewed by a pandas team member
    on Sep 30, 2025
  12. rhshadrach commented on Sep 30, 2025

    @rhshadrach
    Member

    Ah, I see that a test can be added, reopening.

  13. added
    Needs TestsUnit test(s) needed to prevent regressions
    and removed
    Upstream issueIssue related to pandas dependency
    on Sep 30, 2025
  14. neutrinoceros commented on Sep 30, 2025

    @neutrinoceros
    Contributor

    nevermind I did check that the existing test suite already captures the issue we saw with bs4 4.14.0. It can still happen that downstream testers find problems first but that's not something we can fix with more tests. Feel free to close !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugIO HTMLread_html, to_html, Styler.apply, Styler.applymapNeeds TestsUnit test(s) needed to prevent regressions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions