Repository navigation
Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
DF = pd.DataFrame({"a": [1, 2, 3], "b": [0.0, 0.0, 0.0]})
DF.to_html("foo.html")
pd.read_html("foo.html", flavor=["bs4"])Issue Description
The above code works fine with beautifulsoup4 version 4.13.5.
It breaks with beautiifulsoup4 version 4.14.0 that was released on 9/27/25
So we either need to pin beautifulsoup4 or fix the bug.
Expected Behavior
No bug with pandas 2.3.2
Installed Versions
Details
INSTALLED VERSIONS
commit : 4665c10
python : 3.11.13
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.26100
machine : AMD64
processor : Intel64 Family 6 Model 183 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252
pandas : 2.3.2
numpy : 2.3.3
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 25.1
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.14.0
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : 1.1
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : 6.0.2
matplotlib : 3.10.6
numba : None
numexpr : 2.13.0
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 21.0.0
pyreadstat : 1.3.1
pytest : 8.4.2
python-calamine : None
pyxlsb : 1.0.10
s3fs : None
scipy : 1.16.2
sqlalchemy : 2.0.43
tables : 3.10.2
tabulate : 0.9.0
xarray : 2025.9.0
xlrd : 2.0.2
xlsxwriter : 3.2.9
zstandard : 0.24.0
tzdata : 2025.2
qtpy : None
pyqt5 : None
Activity
neutrinoceros commentedon Sep 29, 2025
This is affecting astropy's CI (xref: astropy/astropy#18646)
and I noticed that gh-58086 already contained a test that could have caught this change. @Dr-Irv's reproducer is simpler so I suggest using it instead, but it's frustrating that this functionality is seemingly still not being tested.
neutrinoceros commentedon Sep 29, 2025
(FWIW I'm working on a bisection, and hopefully a fix will ensue. Worst case scenario I'll report this to bs4 devs)
neutrinoceros commentedon Sep 29, 2025
bisected to https://git.launchpad.net/beautifulsoup/commit/?id=951992740d04d1ee9d33c906e80b97cf8a197b05
neutrinoceros commentedon Sep 29, 2025
seems like a clear regression to me, so I'll report there
neutrinoceros commentedon Sep 29, 2025
upstream report: https://bugs.launchpad.net/beautifulsoup/+bug/2125906
as I mentioned there, I'll let bs4's maintainers decide wether they want to address this on their side or not. If they don't, I note that a fix is also possible on pandas' side, albeit a bit ugly.
leonardr commentedon Sep 29, 2025
The example code works (or at least doesn't raise an exception) using the recently released Beautiful Soup 4.14.2.
neutrinoceros commentedon Sep 29, 2025
Indeed it does. I suggest this issue could be closed by the addition of a test. Thanks @leonardr !
Dr-Irv commentedon Sep 29, 2025
I think some YAML files need to be updated to make sure that
beautifulsoup44.14.0 and 4.14.1 don't get installed, or just change it to havebeautifulsoup4 >= 4.14.2mroeschke commentedon Sep 29, 2025
We normally don't pin dependencies to avoid a bug in an optional dependency. In a unit testing context, the recommendation is just to xfail or skip the test based on the failing dependency version.
rhshadrach commentedon Sep 30, 2025
The linked bug report to Beautiful Soup indicates the issue has been resolved upstream. Closing. Thanks @neutrinoceros and @leonardr
rhshadrach commentedon Sep 30, 2025
Ah, I see that a test can be added, reopening.
neutrinoceros commentedon Sep 30, 2025
nevermind I did check that the existing test suite already captures the issue we saw with bs4 4.14.0. It can still happen that downstream testers find problems first but that's not something we can fix with more tests. Feel free to close !