Fix datetime timezone handling in data frames #2784
Conversation
ghost
self-requested a review
ghost
changed the title
karriebear
left a comment
karriebear
left a comment
There was a problem hiding this comment.
Would be great to get some frontend tests, probably E2E(?) just to confirm that different time with/without timezones are doing what we want. Also do we want to release this with any updates to st.write? If so, perhaps a feature branch but then again I think we're already inconsistent so we wouldn't be making things worse?
| proto_array.datetimes.data.extend(pandas_array.astype(np.int64)) | ||
| # Just convert straight to ISO 8601, preserving timezone | ||
| # awareness/unawareness. The frontend will render it correctly. | ||
| proto_array.datetimes.data.extend(pandas_array.map(datetime.datetime.isoformat)) |
There was a problem hiding this comment.
Not 100% positive but I vaguely remember this happening in two places in the same doc. Perhaps for index? Might just want to double check?
There was a problem hiding this comment.
| return new Date(nanos / 1e6) | ||
| } | ||
|
|
||
| static iso8601ContainsTimezone(iso: string): boolean { |
There was a problem hiding this comment.
Could we get some unit tests, mostly because I'm lazy and I like to read those instead of code? 🙏
frontend/src/lib/format.ts
Outdated
| // it contains a timezone, the datetime part of the moment will change. | ||
| const withZone = moment.parseZone(iso) | ||
| const withoutZone = moment(iso) | ||
| return !withZone.utc(true).isSame(withoutZone.utc(true)) |
There was a problem hiding this comment.
Had to look up what the boolean did and totally looked up the wrong method and ended up at parsing instead of manipulating. Do you mind adding a comment for the boolean? 🙏
Passing true will change the time zone without changing the current time.
Also just because I hate timezones and had to look up moments documentation:
parseZone will keep the current time with timezone.
moment(iso) will convert to local or UTC time.
to UTC will take the time and strip the zone and localize to UTC
If both times are still equal then there is no timezone.
| */ | ||
| function toFormattedString(x: any): string { | ||
| if (moment.isMoment(x)) { | ||
| return x.format() |
There was a problem hiding this comment.
Not 100% from what I read on the docs but wouldn't this just end up parsing the time back into local timezone?
There was a problem hiding this comment.
This is formatting, not parsing — x itself is already a moment object, we're just rendering it.
Will do, I'll write some E2E tests.
I asked Abhi, he says the immediate (urgent!) priority is fixing this for dataframes (blocker for customer), and that fixing timezones everywhere else can be pushed to a later date. |
* save changes * remove redundant change * fix tests * fix quotes * add tests and other fixes * fix lint * update protobufs and tests * add e2e test * fix quotes * update test
* save changes * remove redundant change * fix tests * fix quotes * add tests and other fixes * fix lint * update protobufs and tests * add e2e test * fix quotes * update test
* develop: Color picker - show value (streamlit#2817) Minor improvements to local E2E testing (streamlit#2807) Fix datetime timezone handling in data frames (streamlit#2784) Remove nonexistent elements from widget state (streamlit#2760) Shared selectbox (streamlit#2795)
* created conditional rendering of controls & tests * reorganized and added tests * removed accidently added unneeded import * alter flex, padding and gap to align columns * adjusted spacing since not duplicated * alter flex, padding and gap to align columns * adjusted spacing since not duplicated * Fix checkbox spacing (#2738) * remove extra padding * remove snapshot completely * add snapshot back in * remove snapshots * remove rest of snapshots * add snapshots back in * Autofocus "clear cache" button (#2739) * autofocus clear cache button * add test * update test * fix lint * `allow_multiple_files` -> `accept_multiple_files` (#2761) * Update component-template submodule to latest (#2767) Point our `component-template` submodule to the latest commit in that repo. (Among other things, this will de-dupe a bunch of the `component-lib` code that used to live in `component-template` before being pulled into its own npm package.) * Slider thumb values are always visible (#2724) * Close #2699 * Override Thumb subcomponent completely * Fixed linting errors Co-authored-by: Ken McGrady <[email protected]> * Create base color picker for use with API and internally (#2778) * Copy to shared and cleanup * Cleanup * remove comment * config: client.showTracebacks (#2770) Adds a new config option, `"client.showTracebacks"`. By default, it's `True`. If the option is set to `False`, then uncaught exceptions in a Streamlit app will result in a generic "Something went wrong!" warning in the browser, rather than the exception and its traceback This logic happens in the `error_util.handle_uncaught_app_exception` function, rather than directly within ScriptRunner. `scriptrunner_test`, `caching_test`, and `streamlit_test` have new tests that verify the right thing happens when exceptions are thrown in different circumstances with this config option turned on and off. Closes #1032 * Shared selectbox (#2795) * Remove nonexistent elements from widget state (#2760) * remove nonexistent elements from widget states * fix typo * add test * Fix datetime timezone handling in data frames (#2784) * save changes * remove redundant change * fix tests * fix quotes * add tests and other fixes * fix lint * update protobufs and tests * add e2e test * fix quotes * update test * Revert "removed accidently added unneeded import" This reverts commit f886adc. * Revert "reorganized and added tests" This reverts commit d3632d9. * Revert "created conditional rendering of controls & tests" This reverts commit 7c4400e. * updated tests * fixed lint * correct percentages in spec for test * update st_image column image * added comment to HoizontalBlock * added Issue/PR to comment Co-authored-by: bh-streamlit <[email protected]> Co-authored-by: Simon Biggs <[email protected]> Co-authored-by: Tim Conkling <[email protected]> Co-authored-by: Henrikh Kantuni <[email protected]> Co-authored-by: Ken McGrady <[email protected]> Co-authored-by: karrie <[email protected]>
* develop: Update "showErrorDetails" config description and docs (streamlit#2841) Pause Dependabot updates for non-security-related issues (streamlit#2840) client.showTracebacks -> showErrorDetails (per product) (streamlit#2837) Color picker - show value (streamlit#2817) Minor improvements to local E2E testing (streamlit#2807) Fix datetime timezone handling in data frames (streamlit#2784) Remove nonexistent elements from widget state (streamlit#2760) Shared selectbox (streamlit#2795)
1 participant
Our new logic to handle datetimes in data frames is:
For example, if the user is in UTC-8:
Notice that the datetime displayed always stays the same. Only the timezone changes.
This fixes #1061 by moving the timezone-handling code to the frontend, i.e. the datetime is no longer coerced into a timezone on the backend. If a timezone-unaware datetime was provided, it remains timezone-unaware all the way to the frontend.
To make this change, we stopped representing dates as epoch timestamps, and started representing them as ISO 8601 strings. The frontend handles the timezone rendering logic.