BUG: to_json not warning about deprecated epoch date_format for datetime index/columns (#65868) by binggao1230 · Pull Request #65869 · pandas-dev/pandas

binggao1230 · 2026-06-12T18:37:04Z

closes BUG: missing deprecation warning about epoch timestamps in the index in to_json() #65868
Tests added and passed
All code checks passed
Added an entry in the latest doc/source/whatsnew/v3.1.0.rst file if fixing a bug

Problem

to_json() warns about the deprecated default date_format="epoch" only when a datetime-like value is in the column values. When the datetime is in the index (or the column labels), the output still contains epoch integers but no warning is emitted:

>>> df = pd.DataFrame({'A': [1, 2], 'B': [pd.Timestamp("2012-01-01"), pd.Timestamp("2012-01-02")]})
>>> df.to_json()                 # warns
>>> df.set_index("B").to_json()  # no warning, but still epoch output:
'{"A":{"1325376000000":1,"1325462400000":2}}'

As @jorisvandenbossche notes in the issue, to switch the default to "iso" in 4.0 we need to warn in both cases now.

Cause

In NDFrame.to_json the warning condition only inspects the column-value dtypes (self.dtypes / self.dtype). But pandas/io/json/_json.py epoch-encodes the index and the column labels too, so those axes can produce epoch output without triggering the warning.

Fix

Check the index dtype as well (and, for a DataFrame, the columns dtype) when deciding whether to emit the deprecation warning. Behaviour for the existing column-value case is unchanged, and no double warning is produced.

The existing parametrized test test_default_epoch_date_format_deprecated is extended with a datetime index, a timedelta index, datetime column labels, and a Series with a datetime index.

binggao1230 · 2026-06-12T19:10:34Z

Heads-up on CI: extending the warning to the index/columns (as intended) correctly surfaces the deprecated default epoch path in a number of existing round-trip tests that serialize datetime-like data through the index/columns and rely on date_format="epoch" — e.g. in tests/io/json/test_pandas.py: test_roundtrip_timestamp, test_series_roundtrip_timeseries, test_v12_compat, test_axis_dates, test_blocks_compat_GH9037, test_path, test_round_trip_exception, test_timedelta_to_json_fractional_precision, plus a few read-side tests that build their fixture via to_json(). The full matrix likely turns up a handful more outside io/json.

These now emit Pandas4Warning under filterwarnings=error. That's the same fallout the original column-only warning had in GH#57063, where the affected tests were wrapped in tm.assert_produces_warning(Pandas4Warning) (or switched to date_format="iso").

I'm happy to update all the affected tests the same way in this PR. Before I push a large test-only diff: do you prefer wrapping them in assert_produces_warning, or moving the round-trip tests to an explicit date_format so they keep asserting data integrity without the deprecation noise? I'll follow whichever convention you'd like for the rollout.

…ime index/columns (GH#65868) The default date_format="epoch" deprecation warning in to_json only checked the column-value dtypes, so a datetime-like index or set of column labels produced epoch integers without warning, even though those axes are also epoch-encoded. Check the index (and, for a DataFrame, the columns) dtype as well.

binggao1230 · 2026-06-12T19:34:53Z

Went ahead and updated the affected tests so CI is green. The blast radius turned out to be smaller than my first estimate — it's exactly six functions, all in tests/io/json/test_pandas.py (test_roundtrip_timestamp, test_blocks_compat_GH9037, test_series_roundtrip_timeseries, test_path, test_axis_dates, test_timedelta_to_json_fractional_precision). The other files I'd flagged earlier don't actually exercise the default-epoch path.

I kept these as epoch round-trip tests and wrapped the to_json() calls in tm.assert_produces_warning(Pandas4Warning, match=msg), matching the GH#57063 precedent already used in this module — rather than switching them to date_format="iso". That keeps the existing round-trip coverage intact while asserting the newly-expected warning. test_timedelta_to_json_fractional_precision also drops its warn = None for the object-dtype case, since the index is timedelta-like and now legitimately warns.

Happy to flip any of these to iso instead if you'd prefer that direction.

jorisvandenbossche

Thanks for the PR

jorisvandenbossche · 2026-06-12T21:13:25Z

            ser = ser.astype(object)
-            warn = None


Can you instead do ser.index = ser.index.astype(object) so we are actually testing with object dtype?

Can you still make this change?

jorisvandenbossche · 2026-06-12T21:16:48Z

 - :meth:`HDFStore.put` and :meth:`HDFStore.append` now support storing :class:`Series` and :class:`DataFrame` columns with :class:`PeriodDtype` in both ``"fixed"`` and ``"table"`` formats (:issue:`41978`)
 - Bug in :meth:`DataFrame.__repr__` raising ``TypeError`` for a column with a NumPy structured dtype (e.g. produced by :meth:`DataFrame.from_records` from a structured ``ndarray``) (:issue:`55011`)
 - Bug in :meth:`DataFrame.__repr__` where horizontally truncated output could exceed the terminal width by up to 4 characters (:issue:`32461`)
+- Bug in :meth:`DataFrame.to_json` and :meth:`Series.to_json` not emitting the default ``date_format="epoch"`` deprecation warning when the datetime-like values were in the index or column labels rather than in the column values (:issue:`65868`)


Can you move this to the deprecations section?

…H#65868)

binggao1230 · 2026-06-12T21:27:19Z

Thanks @jorisvandenbossche! Moved the note to the Deprecations section.

On the test suggestion (ser.index = ser.index.astype(object)) — trying it surfaced something worth flagging. With an object-dtype index the labels are still epoch-encoded, but the warning does not currently fire:

ser = Series([Timedelta(milliseconds=42)]).astype(object)
ser.index = ser.index.astype(object)
ser.to_json()      # -> {"42":42}   (epoch-encoded), but no Pandas4Warning

The reason is that this PR detects datetime-likes via dtype.kind in "mM", which an object-dtype axis holding Timestamp/Timedelta/datetime objects does not match, even though the C encoder still epoch-encodes those objects. The same holds for object-dtype column values, not just labels:

DataFrame({"a": [Timestamp("2020-01-01")]}, dtype=object).to_json()
# -> {"a":{"0":1577836800000}}   (epoch), no warning

So warning on object-dtype datetime-like axes is a real, slightly broader gap. I can extend the check to fall back to lib.infer_dtype(...) in {"datetime", "date", "timedelta"} for object-dtype axes (index, columns, and values), which would make the object-dtype test pass. Hesitations before I do:

it runs infer_dtype on object axes on the default to_json path (cheap when there are no object columns, but worth noting);
BUG: missing deprecation warning about epoch timestamps in the index in to_json() #65868 is framed around index/column labels — covering object-dtype column values is a wider scope;
a mixed object array (e.g. a Timestamp plus a string) still epoch-encodes the datetime element; unclear whether that should warn.

Happy to extend it here (I kept the test on the native m8[ns] index for now so CI stays green), or keep this PR to native dtypes and handle object-dtype separately. Which would you prefer?

jorisvandenbossche · 2026-06-16T09:00:59Z

On the test suggestion (ser.index = ser.index.astype(object)) — trying it surfaced something worth flagging. With an object-dtype index the labels are still epoch-encoded, but the warning does not currently fire:

Yes, I had noticed that as well. I was planning to open a separate issue about this to fix.

Your suggestion to use infer_dtype is I think the correct fix. I have similar reservations about using it since it makes the deprecation check more costly, but since the behaviour will change, I think we should do it.

But let's make that a focused change in a follow-up PR.

binggao1230 · 2026-06-16T09:08:38Z

Sounds good, agreed on keeping it focused. This PR stays on the native datetime-like dtypes (the test index remains m8[ns]), so the object-dtype infer_dtype change is left out of scope here. I'm happy to pick up that follow-up once you've opened the issue.

Only convert the index to object dtype, keeping the values as timedelta (m8[ns]). This better isolates the GH#65868 behavior — the epoch deprecation warning should fire when the index contains datetime-like values even with object-dtype index. Addresses review feedback from jorisvandenbossche on pandas-dev#65869. Signed-off-by: Vincent Gao <[email protected]>

binggao1230

Thanks for the review! I've updated the test to use ser.index = ser.index.astype(object) instead of ser = ser.astype(object), so we're testing with object-dtype index specifically. The whatsnew entry was already moved to the Deprecations section in an earlier commit.

…poch-index-warning

jorisvandenbossche · 2026-06-27T22:25:26Z

Thanks @gaoflow

binggao1230 force-pushed the fix-65868-to-json-epoch-index-warning branch from f1b67c2 to 45c681f Compare June 12, 2026 18:44

binggao1230 force-pushed the fix-65868-to-json-epoch-index-warning branch from 45c681f to cc092fb Compare June 12, 2026 19:34

jorisvandenbossche reviewed Jun 12, 2026

View reviewed changes

DOC: move to_json epoch index/columns warning note to Deprecations (G…

5972153

…H#65868)

jorisvandenbossche added IO JSON read_json, to_json, json_normalize Deprecate Functionality to remove in pandas labels Jun 16, 2026

jorisvandenbossche added this to the 3.1 milestone Jun 16, 2026

binggao1230 commented Jun 18, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into fix-65868-to-json-e…

3c9e321

…poch-index-warning

jorisvandenbossche approved these changes Jun 27, 2026

View reviewed changes

jorisvandenbossche merged commit 29819cb into pandas-dev:main Jun 27, 2026
45 of 46 checks passed

github-actions Bot mentioned this pull request Jun 27, 2026

DEPR: List of deprecations to be removed in 4.0 #56596

Open

Navigation Menu

Repository navigation

BUG: to_json not warning about deprecated epoch date_format for datetime index/columns (#65868) - #65869

BUG: to_json not warning about deprecated epoch date_format for datetime index/columns (#65868)#65869
jorisvandenbossche merged 4 commits into
pandas-dev:mainfrom
binggao1230:fix-65868-to-json-epoch-index-warning

binggao1230 commented Jun 12, 2026

Uh oh!

binggao1230 commented Jun 12, 2026

Uh oh!

binggao1230 commented Jun 12, 2026

Uh oh!

jorisvandenbossche left a comment

Uh oh!

jorisvandenbossche Jun 12, 2026

Uh oh!

jorisvandenbossche Jun 16, 2026

Uh oh!

jorisvandenbossche Jun 12, 2026

Uh oh!

binggao1230 commented Jun 12, 2026

Uh oh!

jorisvandenbossche commented Jun 16, 2026

Uh oh!

binggao1230 commented Jun 16, 2026

Uh oh!

binggao1230 left a comment

Uh oh!

Uh oh!

jorisvandenbossche commented Jun 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

binggao1230 commented Jun 12, 2026

Problem

Cause

Fix

Uh oh!

binggao1230 commented Jun 12, 2026

Uh oh!

binggao1230 commented Jun 12, 2026

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

binggao1230 commented Jun 12, 2026

Uh oh!

jorisvandenbossche commented Jun 16, 2026

Uh oh!

binggao1230 commented Jun 16, 2026

Uh oh!

binggao1230 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jorisvandenbossche commented Jun 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants