Skip to content

Comments

refactor: Centralize date/time handling with Timestamp wrapper#6650

Merged
Xuanwo merged 23 commits intoapache:mainfrom
dami0806:refactor/jiff-timestamp-wrapper
Oct 14, 2025
Merged

refactor: Centralize date/time handling with Timestamp wrapper#6650
Xuanwo merged 23 commits intoapache:mainfrom
dami0806:refactor/jiff-timestamp-wrapper

Conversation

@dami0806
Copy link
Contributor

@dami0806 dami0806 commented Oct 12, 2025

Related Issue

Closes #6647

Rationale

This PR centralizes all date/time-related logic within OpenDAL. By introducing a new Timestamp wrapper in core/src/raw/time.rs, direct usage of jiff::Timestamp throughout the codebase is removed.

Key benefits:

  • Simplified API: Provides convenience methods like now(), from_millisecond(), and format_http_date().
  • Consistent error handling: Converts jiff errors into OpenDAL's Error type.
  • Type safety: Prevents external jiff types from leaking into the public API.
  • Future-proof design: Makes it easier to adapt to potential date/time library changes.

Changes included in this PR

Core Implementation (8 commits)

1. Add Timestamp wrapper (time.rs)

  • Introduced a new Timestamp wrapper around jiff::Timestamp in core/src/raw/time.rs.
  • Implemented conversion traits: From, TryFrom, FromStr.
  • Added formatting methods: format_http_date, format_rfc3339, format_iso8601.
  • Added helper constructors: now, from_millisecond, from_second.

2. Remove jiff_util module

  • Deleted the obsolete core/src/raw/jiff_util.rs.
  • Updated core/src/raw/mod.rs to expose the new Timestamp type.

3. Update raw layer utilities

  • Refactored adapters, http_util, and ops to use the Timestamp wrapper.

4. Update types layer

  • Updated metadata, options, and operator to use the Timestamp wrapper.

5. Migrate cloud storage services

  • Services: S3, GCS, Azure Blob/DLS/File, Tencent COS, Alibaba OSS, Backblaze B2, Aliyun Drive, Vercel Blob, Cloudflare KV.
  • Metadata parsing and formatting now use the Timestamp wrapper.

6. Migrate filesystem services

  • Services: Local FS, HDFS, HDFS Native, SFTP, WebHDFS, WebDAV, FTP, MonoioFS, Alluxio, CompFS.
  • Updated file metadata, system time conversions, and listers/writers to use Timestamp.

7. Migrate other services

  • Services: Dropbox, OneDrive, Google Drive, pCloud, GitHub, Hugging Face, Koofr, LakeFS, Seafile, Yandex Disk, UpYun, Swift, cacache, dashmap, mini_moka, DBFS.
  • Integrated the Timestamp wrapper.

8. Update Cargo.lock

  • Refreshed dependency lockfile to reflect the Timestamp migration.

Bindings & Integrations Fixes (5 commits)

9. Fix bindings and add as_millisecond() method

Files:

  • core/src/raw/time.rs - Added missing as_millisecond() method
  • bindings/c/src/metadata.rs - Use as_millisecond() instead of manual calculation
  • integrations/object_store/src/lib.rs - Use into_inner() for proper conversion
  • bindings/python/src/options.rs - Revert to Option<Timestamp> (was temporarily Option<String>)
  • bindings/python/src/metadata.rs - Revert to Option<Timestamp>

Rationale: During initial implementation, temporary workarounds (manual calculations, string conversions) were used to unblock testing. This commit properly adds the missing as_millisecond() method and removes all temporary code.

10. Add Python datetime integration

Files: bindings/python/src/lib.rs, bindings/python/src/metadata.rs, bindings/python/src/options.rsImplementation:

  • Created Python-specific Timestamp wrapper type in bindings layer
  • Implemented IntoPyObject<'py> trait to convert Timestamp → Python datetime
  • Implemented FromPyObject<'py> trait to convert Python datetimeTimestamp
  • Leverages PyO3's jiff-02 feature for automatic jiff::Timestamp ↔ Python datetime conversion
  • Core library remains free from PyO3 dependencies

11. Fix fuse3 integration

File: integrations/fuse3/src/file_system.rs

  • Updated metadata2file_attr() to use .into_inner() for TimestampSystemTime conversion

12. Format object_store integration

File: integrations/object_store/src/lib.rs

  • Applied cargo fmt to format multi-line function calls

13. Fix unftp-sbe integration

File: integrations/unftp-sbe/src/lib.rs

  • Updated OpendalMetadata::modified() to use .into_inner() for TimestampSystemTime conversion

Helper function replacements

Throughout the migration:

  • Replaced parse_datetime_from_rfc3339()str.parse::<Timestamp>().
  • Replaced parse_datetime_from_timestamp_millis()Timestamp::from_millisecond().
  • Replaced direct jiff::Timestamp usage → Timestamp wrapper.

Commit Summary

# Commit ID Description
1 f672ba0 Add Timestamp wrapper in time.rs
2 9e045d4 Update raw utilities to use Timestamp
3 44267e2 Update types layer to use Timestamp
4 3b5a2f9 Migrate cloud storage services
5 c524280 Migrate filesystem services
6 8ebbb36 Migrate other services
7 9640799 Update Cargo.lock
8 90b564a Fix Java bindings
9 cde0223 Fix Node.js bindings
10 b113811 Fix Python bindings initial attempt
11 d24fde0 Fix additional CI compilation errors
12 fa3f279 Refactor: add as_millisecond() and fix bindings properly
13 4dd5256 Feat: add Python Timestamp wrapper for datetime conversion
14 bd88579 Fix: fuse3 integration timestamp conversion
15 d0b9743 Chore: format object_store integration
16 2c4520c Fix: unftp-sbe integration timestamp conversion

Architecture Decisions

Why a wrapper instead of type alias?

  • Encapsulation: Hides jiff::Timestamp implementation details
  • Method additions: Can add OpenDAL-specific convenience methods
  • Future flexibility: Easy to swap underlying library if needed
  • API clarity: Makes timestamp handling explicit in function signatures

Language bindings strategy

Each language binding handles Timestamp conversion in its own layer:

  • Python: Custom wrapper type with PyO3 traits (IntoPyObject, FromPyObject)
  • C: Direct as_millisecond() for simple integer representation
  • Java/Node.js: Use existing datetime conversion patterns

This approach keeps the core library clean and allows each binding to use its language's native datetime types.

Local Verification

  • cargo fmt --check
  • cargo check -p opendal --features="blocking"
  • cargo clippy -p opendal --features="blocking" -- -D warnings
  • cargo test -p opendal --features="blocking,services-s3,services-fs" (All 319 tests passed)
  • cargo check (Python bindings) - Compilation successful
  • cargo clippy (Python bindings) - No warnings

Breaking Changes

⚠️ Public API Changes

Type Changes:

  • Metadata::last_modified() return type changed
    from Option<jiff::Timestamp> to
    Option<Timestamp> (our new wrapper type)
  • Read/write/stat options now use Timestamp
    wrapper instead of direct jiff::Timestamp

Impact:
If your code directly references jiff::Timestamp
types from OpenDAL's public API, you'll need to
update it to use the Timestamp wrapper after
this PR is merged.

Rationale:
This change prevents internal implementation
details (like jiff::Timestamp) from leaking into
OpenDAL's public API, improving type safety and
future extensibility.

Migration Example

// Before
let ts: Option<jiff::Timestamp> =
metadata.last_modified();

// After
let ts: Option<Timestamp> =
metadata.last_modified();
// Use wrapper methods like .as_millisecond(),
.format_http_date(), etc.

Review Request

1
Please share your feedback on this public API change.
If there are project policies or better approaches to consider, I'm happy to adjust accordingly.

2
I’ve looked into the current CI failures.

From the logs, it appears that the issue isn’t with the Rust code I modified. Instead, the failures occur during the Java binding build process when attempting to access the Maven Central repository. The build is consistently encountering 503 Service Temporarily Unavailable errors (and occasionally 500 Internal Server Error).

Specifically, the build fails when trying to download certain artifacts (such as plexus-classworlds or google-java-format) due to error responses from external Maven servers. This seems to be a temporary network issue or instability on the Maven Central server itself.

I’ve retried the CI a few times, but the same errors persist.
My Rust code compiles cleanly and passes all checks, so this Maven error does not appear to be related to my changes.
Could you advise if there’s anything I can do in this situation, or should we wait for the Maven servers to stabilize? Thank you.

  Introduce a new Timestamp newtype wrapper around jiff::Timestamp
  to provide a more ergonomic and type-safe API for time operations
  across OpenDAL.

  Key features:
  - Wraps jiff::Timestamp with a thin newtype pattern
  - Implements common conversion traits (From, TryFrom, FromStr)
  - Provides formatting methods (format_http_date, format_rfc3339, etc)
  - Adds helper constructors (now, from_millisecond, from_second)
  - Maintains compatibility with SystemTime and Duration

  This wrapper simplifies time handling and makes the API more consistent
  throughout the codebase while preserving all jiff functionality.
  Replace the old jiff_util module with the new time module.
  Update module exports to use the new Timestamp wrapper type.
  Update raw layer utilities to use the new Timestamp wrapper:
  - typed_kv/api: Use Timestamp for key-value metadata
  - http_util/header: Use Timestamp formatting methods
  - ops: Update operation metadata to use Timestamp
  Update types layer to use the new Timestamp wrapper:
  - metadata: Use Timestamp for file metadata timestamps
  - options: Use Timestamp for conditional request options
  - operator_futures: Use Timestamp in future option builders
  storage services to Timestamp wrapper

  Update AWS S3, Google GCS, Azure Blob/DLS/File,
  Tencent COS,
  Alibaba OSS, Backblaze B2, Aliyun Drive, Vercel
  Blob, and
  Cloudflare KV services to use the new Timestamp
  wrapper.

  Changes include:
  - Migrate core service logic to use Timestamp
  - Update listers to use Timestamp parsing
  - Replace parse_datetime_from_* helpers with
  Timestamp methods
  filesystem services to Timestamp wrapper

  Update local fs, HDFS, HDFS native, SFTP, WebHDFS,
   WebDAV, FTP,
  MonoioFS, Alluxio, and CompFS services to use the
  new Timestamp wrapper.

  Changes include:
  - Migrate SystemTime conversions to use Timestamp
  - Update file metadata handling with Timestamp
  - Update listers and writers to use Timestamp
  methods
  services to Timestamp wrapper

  Update Dropbox, OneDrive, Google Drive, pCloud,
  GitHub, Hugging Face,
  Koofr, LakeFS, Seafile, Yandex Disk, UpYun, Swift,
   cacache, dashmap,
  mini_moka, and DBFS services to use the new
  Timestamp wrapper.

  Changes include:
  - Migrate cloud drive services to use Timestamp
  - Update cache services with Timestamp handling
  - Update listers and writers across all services
  after Timestamp refactor
@dami0806 dami0806 requested a review from Xuanwo as a code owner October 12, 2025 10:31
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. releases-note/refactor The PR does a refactor on code or has a title that begins with "refactor" labels Oct 12, 2025
  use Timestamp wrapper
  parse_datetime_from_rfc3339 with
  Timestamp::from_str

  The parse_datetime_from_rfc3339 helper function
  was removed as part of
  the Timestamp wrapper introduction. Now using the
  FromStr trait
  implementation on Timestamp directly.
  to access jiff Timestamp methods

  The Timestamp wrapper doesn't expose all
  jiff::Timestamp methods directly.
  Use into_inner() to access the underlying
  jiff::Timestamp when needed,
  such as for as_millisecond().
  update timestamp conversion helpers

  Update timestamp_to_datetime and
  datetime_to_timestamp helper functions
  to use opendal::raw::Timestamp wrapper instead of
  jiff::Timestamp directly.
  Convert to/from jiff::Timestamp internally for
  chrono interoperability.
  CI compilation errors

  - C bindings: Calculate milliseconds from
  Timestamp using as_second() and
  subsec_nanosecond()
  - Python bindings: Change Timestamp to String for
  PyO3 compatibility
  - dav-server integration: Use as_system_time() for
   Timestamp to SystemTime conversion
Comment on lines 134 to 136
Some(time) => time.as_millisecond(),
Some(time) => {
time.as_second() * 1000 + (time.subsec_nanosecond() / 1_000_000) as i64
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like something vibe coding produced.

Please try to review your code to make it more concise for human review. Otherwise I don't think it's valuable for requesting for review.

Copy link
Contributor Author

@dami0806 dami0806 Oct 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I implemented the Timestamp wrapper, I added methods like as_second() and from_millisecond(), but initially missed as_millisecond().

While working on the C bindings, this caused a compilation error. To unblock testing, I temporarily calculated milliseconds manually (as_second() * 1000 + subsec_nanosecond() / 1_000_000).

I should have cleaned up this temporary code afterward. I’ll now properly add as_millisecond() to the Timestamp wrapper in core/src/raw/time.rs.

dami0806 and others added 7 commits October 12, 2025 22:51
Add as_millisecond() method to Timestamp wrapper and update C bindings
and object_store integration to use proper wrapper methods instead of
manual calculations. Revert Python String workaround back to Timestamp.
Add Timestamp wrapper in Python bindings to convert opendal::raw::Timestamp
to/from Python datetime using PyO3's jiff-02 feature.
…onversion

Use into_inner() to convert Timestamp to jiff::Timestamp, which can then
be converted to SystemTime.
Signed-off-by: tison <[email protected]>
@tisonkun tisonkun requested a review from silver-ymz as a code owner October 14, 2025 03:58
Signed-off-by: tison <[email protected]>
Signed-off-by: tison <[email protected]>
Copy link
Member

@tisonkun tisonkun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Xuanwo we may give it another look but this is generally LGTM.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Oct 14, 2025
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this!

#[getter]
pub fn last_modified(&self) -> Option<Timestamp> {
self.0.last_modified()
pub fn last_modified(&self) -> Option<jiff::Timestamp> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, jiff::Timestamp has native pyo3 support?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

PyO3/pyo3#4823 and others.

@Xuanwo Xuanwo merged commit ca54fab into apache:main Oct 14, 2025
343 checks passed
Comment on lines 138 to 150
let nano = env.call_method(&result, "getNano", "()I", &[])?.i()?;
Timestamp::new(epoch_second, nano).map(Some).map_err(|err| {
Error::new(
ErrorKind::Unexpected,
format!("invalid timestamp: seconds={epoch_second}, nanos={nano}"),
)
.set_source(err)
.into()
})

opendal::raw::Timestamp::from_second(epoch_second)
.map(Some)
.map_err(|err| {
Error::new(
ErrorKind::Unexpected,
format!("invalid timestamp: seconds={epoch_second}, nanos={nano}"),
)
.set_source(err)
.into()
})
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a bug I'll fix then.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow up at #6663

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer releases-note/refactor The PR does a refactor on code or has a title that begins with "refactor" size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wrap jiff::Timestamp and related logics under a wrapper Timestamp type

3 participants