refactor: Centralize date/time handling with Timestamp wrapper#6650
refactor: Centralize date/time handling with Timestamp wrapper#6650Xuanwo merged 23 commits intoapache:mainfrom
Conversation
Introduce a new Timestamp newtype wrapper around jiff::Timestamp to provide a more ergonomic and type-safe API for time operations across OpenDAL. Key features: - Wraps jiff::Timestamp with a thin newtype pattern - Implements common conversion traits (From, TryFrom, FromStr) - Provides formatting methods (format_http_date, format_rfc3339, etc) - Adds helper constructors (now, from_millisecond, from_second) - Maintains compatibility with SystemTime and Duration This wrapper simplifies time handling and makes the API more consistent throughout the codebase while preserving all jiff functionality.
Replace the old jiff_util module with the new time module. Update module exports to use the new Timestamp wrapper type.
Update raw layer utilities to use the new Timestamp wrapper: - typed_kv/api: Use Timestamp for key-value metadata - http_util/header: Use Timestamp formatting methods - ops: Update operation metadata to use Timestamp
Update types layer to use the new Timestamp wrapper: - metadata: Use Timestamp for file metadata timestamps - options: Use Timestamp for conditional request options - operator_futures: Use Timestamp in future option builders
storage services to Timestamp wrapper Update AWS S3, Google GCS, Azure Blob/DLS/File, Tencent COS, Alibaba OSS, Backblaze B2, Aliyun Drive, Vercel Blob, and Cloudflare KV services to use the new Timestamp wrapper. Changes include: - Migrate core service logic to use Timestamp - Update listers to use Timestamp parsing - Replace parse_datetime_from_* helpers with Timestamp methods
filesystem services to Timestamp wrapper Update local fs, HDFS, HDFS native, SFTP, WebHDFS, WebDAV, FTP, MonoioFS, Alluxio, and CompFS services to use the new Timestamp wrapper. Changes include: - Migrate SystemTime conversions to use Timestamp - Update file metadata handling with Timestamp - Update listers and writers to use Timestamp methods
services to Timestamp wrapper Update Dropbox, OneDrive, Google Drive, pCloud, GitHub, Hugging Face, Koofr, LakeFS, Seafile, Yandex Disk, UpYun, Swift, cacache, dashmap, mini_moka, and DBFS services to use the new Timestamp wrapper. Changes include: - Migrate cloud drive services to use Timestamp - Update cache services with Timestamp handling - Update listers and writers across all services
after Timestamp refactor
Timestamp wrapper
use Timestamp wrapper
parse_datetime_from_rfc3339 with Timestamp::from_str The parse_datetime_from_rfc3339 helper function was removed as part of the Timestamp wrapper introduction. Now using the FromStr trait implementation on Timestamp directly.
to access jiff Timestamp methods The Timestamp wrapper doesn't expose all jiff::Timestamp methods directly. Use into_inner() to access the underlying jiff::Timestamp when needed, such as for as_millisecond().
update timestamp conversion helpers Update timestamp_to_datetime and datetime_to_timestamp helper functions to use opendal::raw::Timestamp wrapper instead of jiff::Timestamp directly. Convert to/from jiff::Timestamp internally for chrono interoperability.
CI compilation errors - C bindings: Calculate milliseconds from Timestamp using as_second() and subsec_nanosecond() - Python bindings: Change Timestamp to String for PyO3 compatibility - dav-server integration: Use as_system_time() for Timestamp to SystemTime conversion
bindings/c/src/metadata.rs
Outdated
| Some(time) => time.as_millisecond(), | ||
| Some(time) => { | ||
| time.as_second() * 1000 + (time.subsec_nanosecond() / 1_000_000) as i64 | ||
| } |
There was a problem hiding this comment.
This seems like something vibe coding produced.
Please try to review your code to make it more concise for human review. Otherwise I don't think it's valuable for requesting for review.
There was a problem hiding this comment.
When I implemented the Timestamp wrapper, I added methods like as_second() and from_millisecond(), but initially missed as_millisecond().
While working on the C bindings, this caused a compilation error. To unblock testing, I temporarily calculated milliseconds manually (as_second() * 1000 + subsec_nanosecond() / 1_000_000).
I should have cleaned up this temporary code afterward. I’ll now properly add as_millisecond() to the Timestamp wrapper in core/src/raw/time.rs.
Add as_millisecond() method to Timestamp wrapper and update C bindings and object_store integration to use proper wrapper methods instead of manual calculations. Revert Python String workaround back to Timestamp.
Add Timestamp wrapper in Python bindings to convert opendal::raw::Timestamp to/from Python datetime using PyO3's jiff-02 feature.
…onversion Use into_inner() to convert Timestamp to jiff::Timestamp, which can then be converted to SystemTime.
Signed-off-by: tison <[email protected]>
Signed-off-by: tison <[email protected]>
Signed-off-by: tison <[email protected]>
Xuanwo
left a comment
There was a problem hiding this comment.
Thank you for working on this!
| #[getter] | ||
| pub fn last_modified(&self) -> Option<Timestamp> { | ||
| self.0.last_modified() | ||
| pub fn last_modified(&self) -> Option<jiff::Timestamp> { |
There was a problem hiding this comment.
Cool, jiff::Timestamp has native pyo3 support?
| let nano = env.call_method(&result, "getNano", "()I", &[])?.i()?; | ||
| Timestamp::new(epoch_second, nano).map(Some).map_err(|err| { | ||
| Error::new( | ||
| ErrorKind::Unexpected, | ||
| format!("invalid timestamp: seconds={epoch_second}, nanos={nano}"), | ||
| ) | ||
| .set_source(err) | ||
| .into() | ||
| }) | ||
|
|
||
| opendal::raw::Timestamp::from_second(epoch_second) | ||
| .map(Some) | ||
| .map_err(|err| { | ||
| Error::new( | ||
| ErrorKind::Unexpected, | ||
| format!("invalid timestamp: seconds={epoch_second}, nanos={nano}"), | ||
| ) | ||
| .set_source(err) | ||
| .into() | ||
| }) | ||
| } |
Related Issue
Closes #6647
Rationale
This PR centralizes all date/time-related logic within OpenDAL. By introducing a new
Timestampwrapper incore/src/raw/time.rs, direct usage ofjiff::Timestampthroughout the codebase is removed.Key benefits:
now(),from_millisecond(), andformat_http_date().jifferrors into OpenDAL'sErrortype.jifftypes from leaking into the public API.Changes included in this PR
Core Implementation (8 commits)
1. Add
Timestampwrapper (time.rs)Timestampwrapper aroundjiff::Timestampincore/src/raw/time.rs.From,TryFrom,FromStr.format_http_date,format_rfc3339,format_iso8601.now,from_millisecond,from_second.2. Remove
jiff_utilmodulecore/src/raw/jiff_util.rs.core/src/raw/mod.rsto expose the newTimestamptype.3. Update raw layer utilities
adapters,http_util, andopsto use theTimestampwrapper.4. Update types layer
metadata,options, andoperatorto use theTimestampwrapper.5. Migrate cloud storage services
Timestampwrapper.6. Migrate filesystem services
Timestamp.7. Migrate other services
Timestampwrapper.8. Update
Cargo.lockTimestampmigration.Bindings & Integrations Fixes (5 commits)
9. Fix bindings and add
as_millisecond()methodFiles:
core/src/raw/time.rs- Added missingas_millisecond()methodbindings/c/src/metadata.rs- Useas_millisecond()instead of manual calculationintegrations/object_store/src/lib.rs- Useinto_inner()for proper conversionbindings/python/src/options.rs- Revert toOption<Timestamp>(was temporarilyOption<String>)bindings/python/src/metadata.rs- Revert toOption<Timestamp>Rationale: During initial implementation, temporary workarounds (manual calculations, string conversions) were used to unblock testing. This commit properly adds the missing
as_millisecond()method and removes all temporary code.10. Add Python datetime integration
Files:
bindings/python/src/lib.rs,bindings/python/src/metadata.rs,bindings/python/src/options.rsImplementation:Timestampwrapper type in bindings layerIntoPyObject<'py>trait to convertTimestamp→ PythondatetimeFromPyObject<'py>trait to convert Pythondatetime→Timestampjiff-02feature for automaticjiff::Timestamp↔ Pythondatetimeconversion11. Fix fuse3 integration
File:
integrations/fuse3/src/file_system.rsmetadata2file_attr()to use.into_inner()forTimestamp→SystemTimeconversion12. Format object_store integration
File:
integrations/object_store/src/lib.rscargo fmtto format multi-line function calls13. Fix unftp-sbe integration
File:
integrations/unftp-sbe/src/lib.rsOpendalMetadata::modified()to use.into_inner()forTimestamp→SystemTimeconversionHelper function replacements
Throughout the migration:
parse_datetime_from_rfc3339()→str.parse::<Timestamp>().parse_datetime_from_timestamp_millis()→Timestamp::from_millisecond().jiff::Timestampusage →Timestampwrapper.Commit Summary
Timestampwrapper intime.rsTimestampTimestampCargo.lockas_millisecond()and fix bindings properlyTimestampwrapper for datetime conversionArchitecture Decisions
Why a wrapper instead of type alias?
jiff::Timestampimplementation detailsLanguage bindings strategy
Each language binding handles
Timestampconversion in its own layer:IntoPyObject,FromPyObject)as_millisecond()for simple integer representationThis approach keeps the core library clean and allows each binding to use its language's native datetime types.
Local Verification
cargo fmt --checkcargo check -p opendal --features="blocking"cargo clippy -p opendal --features="blocking" -- -D warningscargo test -p opendal --features="blocking,services-s3,services-fs"(All 319 tests passed)cargo check(Python bindings) - Compilation successfulcargo clippy(Python bindings) - No warningsBreaking Changes
Type Changes:
Metadata::last_modified()return type changedfrom
Option<jiff::Timestamp>toOption<Timestamp>(our new wrapper type)Timestampwrapper instead of direct
jiff::TimestampImpact:
If your code directly references
jiff::Timestamptypes from OpenDAL's public API, you'll need to
update it to use the
Timestampwrapper afterthis PR is merged.
Rationale:
This change prevents internal implementation
details (like
jiff::Timestamp) from leaking intoOpenDAL's public API, improving type safety and
future extensibility.
Migration Example
Review Request
1
Please share your feedback on this public API change.
If there are project policies or better approaches to consider, I'm happy to adjust accordingly.
2
I’ve looked into the current CI failures.
From the logs, it appears that the issue isn’t with the Rust code I modified. Instead, the failures occur during the Java binding build process when attempting to access the Maven Central repository. The build is consistently encountering
503 Service Temporarily Unavailableerrors (and occasionally500 Internal Server Error).Specifically, the build fails when trying to download certain artifacts (such as
plexus-classworldsorgoogle-java-format) due to error responses from external Maven servers. This seems to be a temporary network issue or instability on the Maven Central server itself.I’ve retried the CI a few times, but the same errors persist.
My Rust code compiles cleanly and passes all checks, so this Maven error does not appear to be related to my changes.
Could you advise if there’s anything I can do in this situation, or should we wait for the Maven servers to stabilize? Thank you.