Skip to content

💥 Read into DLPack tensor - #32

Merged
weiji14 merged 5 commits into
mainfrom
dlpack
Jun 13, 2025
Merged

weiji14 merged 5 commits into
mainfrom
dlpack

Conversation

@weiji14

@weiji14 weiji14 commented Jun 13, 2025 •

Copy link
Copy Markdown
Owner

Open up CogReader struct as public API with a new dlpack method to read from GeoTIFFs into a DLPack tensor! Corresponding PyCogReader struct/class has some new __dlpack__ and __dlpack_device__ methods conforming to the Python specification of DLPack (part of the Python array API standard).

Usage:

use std::io::Cursor;

use bytes::Bytes;
use cog3pio::io::geotiff::CogReader;
use dlpark::ffi::DataType;
use dlpark::prelude::TensorView;
use dlpark::SafeManagedTensorVersioned;
use object_store::path::Path;
use object_store::{parse_url, GetResult, ObjectStore};
use tokio;
use url::Url;

#[tokio::main]
async fn main() {
    let cog_url: &str =
        "https://github.com/cogeotiff/rio-tiler/raw/7.8.0/tests/fixtures/cog_dateline.tif";
    let tif_url: Url = Url::parse(cog_url).unwrap();
    let (store, location): (Box<dyn ObjectStore>, Path) = parse_url(&tif_url).unwrap();

    let stream: Cursor<Bytes> = {
        let result: GetResult = store.get(&location).await.unwrap();
        let bytes: Bytes = result.bytes().await.unwrap();
        Cursor::new(bytes)
    };

    // Read GeoTIFF into a dlpark::versioned::SafeManagedTensorVersioned
    let mut cog = CogReader::new(stream).unwrap();
    let tensor: SafeManagedTensorVersioned = cog.dlpack().unwrap();
    assert_eq!(tensor.shape(), [1, 2355, 2325]);
    assert_eq!(tensor.data_type(), &DataType::U16);
}

Changes:

  • Replace CogReader's ndarray method with dlpack method
  • Replace PyCogReader's as_numpy method with __dlpack__ and __dlpack_device__ methods
  • Refactored internals of read_geotiff functions to do TIFF -> DLPack -> (Py)Array3, while keeping backwards compatibility
  • Updated main README.md with:
    • New Rust example showing how to read a uint16 GeoTIFF into dlpark::versioned::SafeManagedTensorVersioned
    • Update Python example showing how to read uint16 GeoTIFF into xarray.DataArray object.

References:

weiji14 added 3 commits June 12, 2025 22:35
A Rust Library for High-Performance Tensor Exchange with Python!  Crate at https://crates.io/crates/dlpark
Open up `CogReader` struct as public API, with a new `dlpack` method to read into a DLPack tensor! Corresponding PyCogReader struct/class has some new `__dlpack__` and `__dlpack_device__` methods conforming to the Python specification of DLPack (part of the  Python array API standard). These methods have replaced the `ndarray`/`as_numpy` methods respectively.

The `read_geotiff` functions retain backwards compatibility to output ndarray::Array3/numpy::Pyarray3 objects. Updated unit tests as needed to handle all the refactorings to use DLPack. Wrote new documentation showing how to use CogReader to output a dlpark::versioned::SafeManagedTensorVersioned, and updated the xarray.open_dataarray example to show that a uint16 GeoTIFF can be read properly now! Also added new links to the async-tiff and tiff2 crates in the main README.md.
@weiji14 weiji14 added this to the 0.1.0 milestone Jun 13, 2025
@weiji14 weiji14 self-assigned this Jun 13, 2025
@weiji14 weiji14 added the feature New feature or request label Jun 13, 2025
weiji14 added 2 commits June 13, 2025 18:36
Found using `cargo msrv find`, the `1.85.0` version is determined by `dlpark=0.6.0` which uses Rust edition 2024.
Not needed anymore after replacing `num_traits::FromPrimitive` trait with `dlpark::traits::InferDataType`.
@weiji14
weiji14 marked this pull request as ready for review June 13, 2025 07:01
@weiji14
weiji14 enabled auto-merge (squash) June 13, 2025 07:01
@weiji14
weiji14 merged commit df33ffd into main Jun 13, 2025
@weiji14
weiji14 deleted the dlpack branch June 13, 2025 08:50
@weiji14 weiji14 mentioned this pull request Oct 17, 2025
5 of 6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant