Skip to content

Python bindings: implement __arrow_c_stream__() interface for ogr.Layer#9043

Merged
rouault merged 2 commits intoOSGeo:masterfrom
rouault:python_arrow_c_stream
Jan 11, 2024
Merged

Python bindings: implement __arrow_c_stream__() interface for ogr.Layer#9043
rouault merged 2 commits intoOSGeo:masterfrom
rouault:python_arrow_c_stream

Conversation

@rouault
Copy link
Member

@rouault rouault commented Jan 9, 2024

@kylebarron Just seeing your yesterday toot https://mapstodon.space/@kylebarron/111723196242600250 , I gave this a try. I don't necessarily have a way to test that beyond the simple unit tests I added

@rouault rouault added this to the 3.9.0 milestone Jan 9, 2024
@coveralls
Copy link
Collaborator

coveralls commented Jan 9, 2024 •
edited
Loading

Coverage Status

coverage: 68.739% (+0.003%) from 68.736%
when pulling e4f78be on rouault:python_arrow_c_stream
into 15751e3 on OSGeo:master.

@rouault rouault force-pushed the python_arrow_c_stream branch 2 times, most recently from 1377183 to 1c52f84 Compare January 9, 2024 15:31
kylebarron
kylebarron reviewed Jan 9, 2024
View reviewed changes
autotest/ogr/ogr_mem.py

with pytest.raises(Exception, match="requested_schema != None not implemented"):
# "something" should rather by a PyCapsule with an ArrowSchema...
lyr.__arrow_c_stream__(requested_schema="something")
Copy link

@kylebarron kylebarron Jan 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think requested_schema was added for future-proofing, but no one implements it yet. So it's fine to ignore

@kylebarron
Copy link

kylebarron commented Jan 9, 2024

This is awesome!

The pyarrow.table constructor will check for __arrow_c_stream__ and call on input as of pyarrow v14, so I think that's the easiest way to test that it works end to end.

cc @jorisvandenbossche @paleolimbot

@rouault
Copy link
Member Author

rouault commented Jan 9, 2024

The pyarrow.table constructor will check for arrow_c_stream

Confirmed:

$ python3
>>> from osgeo import gdal
>>> import pyarrow
>>> ds = gdal.OpenEx("poly.shp")
>>> lyr = ds.GetLayer(0)
>>> table = pyarrow.table(lyr)
>>> table
pyarrow.Table
OGC_FID: int64 not null
AREA: double
EAS_ID: int64
PRFEDEA: string
wkb_geometry: binary
----
OGC_FID: [[0,1,2,3,4,5,6,7,8,9]]
AREA: [[215229.266,247328.172,261752.781,547597.188,15775.758,101429.977,268597.625,1634833.375,-596610.313,5268.813]]
EAS_ID: [[168,179,171,173,172,169,166,158,165,170]]
PRFEDEA: [["35043411","35043423","35043414","35043416","35043415","35043412","35043409","35043369","35043408","35043413"]]
wkb_geometry: [[01030000000100000014000000000000602F491D41000000207F2D5241000000C028471D41000000E0922D5241000000007C461D4100000060AE2D524100000080C9471D4100000020B62D5241000000209C4C1D41000000E0D82D5241000000608D4C1D41000000A0DD2D5241000000207F4E1D41000000A0EA2D524100000020294F1D4100000080CA2D524100000000B4511D41000000E0552D5241000000C016521D4100000080452D5241000000E0174E1D41000000202E2D524100000020414D1D41000000E04C2D5241000000E04B4D1D41000000605E2D524100000040634D1D41000000E0742D5241000000A0EF4C1D41000000E08D2D5241000000E04E4C1D41000000E0A12D5241000000E0B04B1D4100000060B82D524100000080974A1D4100000080AE2D524100000080CF491D4100000080952D5241000000602F491D41000000207F2D5241,01030000000100000014000000000000608D4C1D41000000A0DD2D5241000000209C4C1D41000000E0D82D524100000080C9471D4100000020B62D5241000000007C461D4100000060AE2D5241000000C028471D41000000E0922D5241000000602F491D41000000207F2D5241000000608F481D41000000606D2D52410000002007471D4100000000432D524100000000B0431D41000000A0432D5241000000C0CD421D4100000020432D524100000000E4411D41000000E0422D5241000000A0A23F1D41000000A0342D524100000040733F1D41000000C0332D5241000000204D3E1D41000000C0552D5241000000201A3D1D41000000406F2D5241000000E0D63C1D41000000A06D2D5241000000C09B3C1D41000000E0762D5241000000C0B73E1D4100000020842D5241000000805E461D41000000E0B52D5241000000608D4C1D41000000A0DD2D5241,0103000000010000001D000000000000602F491D41000000207F2D524100000080CF491D4100000080952D524100000080974A1D4100000080AE2D5241000000E0B04B1D4100000060B82D5241000000E04E4C1D41000000E0A12D5241000000A0EF4C1D41000000E08D2D524100000040634D1D41000000E0742D5241000000E04B4D1D41000000605E2D524100000020014C1D41000000C05C2D5241000000E0FB4A1D4100000000572D5241000000805C4A1D4100000000522D524100000080D9471D4100000040302D52410000002001471D41000000002D2D52410000002073461D41000000E0262D52410000008057461D41000000C0202D5241000000E004461D4100000000162D5241000000208D451D41000000800B2D524100000020F0441D4100000060052D5241000000A043441D4100000060F02C5241000000401F401D4100000040D62C524100000040373E1D41000000C02B2D524100000040733F1D41000000C0332D5241000000A0A23F1D41000000A0342D524100000000E4411D41000000E0422D5241000000C0CD421D4100000020432D524100000000B0431D41000000A0432D52410000002007471D4100000000432D5241000000608F481D41000000606D2D5241000000602F491D41000000207F2D5241,0103000000010000001A000000000000C09B3C1D41000000E0762D5241000000E0D63C1D41000000A06D2D524100000040373E1D41000000C02B2D5241000000401F401D4100000040D62C52410000008027411D4100000040B22C52410000002060401D41000000A0A62C5241000000A0D33F1D41000000208D2C5241000000002A3E1D4100000080832C524100000020A93C1D41000000A0C02C5241000000C0C33A1D41000000C0B42C52410000004073381D4100000020A52C5241000000209C341D41000000808E2C524100000000F7331D4100000060AC2C5241000000C0AF331D4100000020BC2C524100000040BF331D4100000080C92C5241000000C0C2331D41000000E0DC2C5241000000C098341D4100000060DF2C5241000000006C371D41000000A00E2D52410000002074361D41000000201D2D52410000006015361D41000000A0172D52410000000012351D41000000E0052D524100000080F0331D41000000C0F12C52410000004020331D4100000080FB2C524100000020AE311D4100000000332D524100000000A53A1D41000000006B2D5241000000C09B3C1D41000000E0762D5241,01030000000100000006000000000000E0D63C1D41000000A06D2D5241000000201A3D1D41000000406F2D5241000000204D3E1D41000000C0552D524100000040733F1D41000000C0332D524100000040373E1D41000000C02B2D5241000000E0D63C1D41000000A06D2D5241,01030000000100000014000000000000E04B4D1D41000000605E2D524100000020414D1D41000000E04C2D5241000000E0174E1D41000000202E2D5241000000E0814B1D41000000001D2D5241000000C01A481D4100000080072D5241000000A0DF471D4100000000142D52410000006060461D41000000400C2D524100000060AA461D4100000080FF2C5241000000A043441D4100000060F02C524100000020F0441D4100000060052D5241000000208D451D41000000800B2D5241000000E004461D4100000000162D52410000008057461D41000000C0202D52410000002073461D41000000E0262D52410000002001471D41000000002D2D524100000080D9471D4100000040302D5241000000805C4A1D4100000000522D5241000000E0FB4A1D4100000000572D524100000020014C1D41000000C05C2D5241000000E04B4D1D41000000605E2D5241,0103000000010000001F000000000000C016521D4100000080452D5241000000A064541D4100000080552D5241000000E0DF541D41000000803D2D52410000000074551D41000000C0282D524100000040F4561D4100000080102D52410000000019571D41000000A0042D52410000006032551D4100000020112D5241000000E072541D4100000040112D5241000000800C541D41000000C0052D5241000000A06E571D4100000060E42C5241000000C0D0561D4100000060C92C52410000006097561D41000000C0C52C5241000000603D561D4100000060BB2C52410000008021551D4100000040B02C5241000000A0D1531D41000000A0A92C5241000000209A511D41000000409D2C524100000040014F1D41000000C0852C524100000080744E1D41000000A09A2C524100000040A94F1D4100000000A42C52410000008084501D4100000020AF2C524100000000F0501D41000000C0BA2C5241000000405E511D4100000040CF2C5241000000E05E511D4100000020DD2C52410000002008501D41000000C0172D5241000000A0C44E1D4100000080112D524100000000774D1D41000000C0072D5241000000002C4C1D41000000A0002D524100000040C44B1D41000000000F2D5241000000E0814B1D41000000001D2D5241000000E0174E1D41000000202E2D5241000000C016521D4100000080452D5241,0103000000010000003D00000000000040F4561D4100000080102D5241000000E0E5571D41000000801A2D5241000000E0E3581D4100000000252D524100000020685A1D41000000A02C2D5241000000E08A5A1D41000000002D2D524100000000615B1D41000000C0322D5241000000C0005D1D4100000000422D524100000060C35D1D41000000A0502D52410000004005601D4100000020332D5241000000602C601D4100000080222D5241000000A0E7621D4100000020322D524100000080C5621D4100000040422D5241000000A096631D41000000C0492D5241000000A00B641D41000000A04D2D5241000000009C641D41000000E0512D52410000008008651D41000000E03C2D52410000006077651D4100000060272D524100000040B5651D41000000601F2D5241000000E08F651D41000000E01E2D5241000000404D5F1D4100000080FA2C524100000040E75E1D4100000060F62C5241000000E0BC5E1D41000000E0F32C524100000000865E1D41000000C0EF2C5241000000C01F5E1D4100000000E92C5241000000E0D15D1D41000000A0D72C524100000080D65D1D4100000020D02C524100000060BF5E1D4100000000852C5241000000C0DD5D1D41000000A05A2C524100000040A95D1D41000000E0522C5241000000C03F5C1D41000000603C2C5241000000C0CA591D4100000080052C524100000040E8581D41000000A0002C5241000000C0A4571D4100000020FB2B5241000000C039561D41000000E0FA2B5241000000405F551D4100000000032C5241000000C067501D4100000020EE2B524100000000764F1D4100000060E32B5241000000C01E4F1D4100000040DD2B5241000000600C4F1D4100000080C92B5241000000C046501D4100000060AA2B5241000000C0D6501D41000000E06D2B524100000060274F1D41000000A0542B524100000040BD491D4100000020402B524100000020A2491D4100000040442B524100000080C3471D4100000060942B5241000000C012441D4100000080322C524100000040F8441D41000000E0432C5241000000E0E04D1D4100000020872C524100000040014F1D41000000C0852C5241000000209A511D41000000409D2C5241000000A0D1531D41000000A0A92C52410000008021551D4100000040B02C5241000000603D561D4100000060BB2C52410000006097561D41000000C0C52C5241000000C0D0561D4100000060C92C5241000000A06E571D4100000060E42C5241000000800C541D41000000C0052D5241000000E072541D4100000040112D52410000006032551D4100000020112D52410000000019571D41000000A0042D524100000040F4561D4100000080102D5241,0103000000010000001B000000000000C01A481D4100000080072D5241000000E0814B1D41000000001D2D524100000040C44B1D41000000000F2D5241000000002C4C1D41000000A0002D524100000000774D1D41000000C0072D5241000000A0C44E1D4100000080112D52410000002008501D41000000C0172D5241000000E05E511D4100000020DD2C5241000000405E511D4100000040CF2C524100000000F0501D41000000C0BA2C52410000008084501D4100000020AF2C524100000040A94F1D4100000000A42C524100000080744E1D41000000A09A2C524100000040014F1D41000000C0852C5241000000E0E04D1D4100000020872C524100000040F8441D41000000E0432C5241000000C012441D4100000080322C524100000000FF431D4100000020362C5241000000004B431D4100000080552C52410000000030431D41000000C05D2C5241000000C09B421D4100000000712C524100000080D6411D4100000080912C52410000008027411D4100000040B22C5241000000401F401D4100000040D62C5241000000A043441D4100000060F02C524100000060AA461D4100000080FF2C5241000000C01A481D4100000080072D5241,01030000000100000005000000000000C01A481D4100000080072D524100000060AA461D4100000080FF2C52410000006060461D41000000400C2D5241000000A0DF471D4100000000142D5241000000C01A481D4100000080072D5241]]

jorisvandenbossche
jorisvandenbossche reviewed Jan 10, 2024
View reviewed changes
Copy link
Contributor

@jorisvandenbossche jorisvandenbossche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@rouault rouault force-pushed the python_arrow_c_stream branch from a535ca2 to e4f78be Compare January 10, 2024 16:30
@rouault rouault merged commit f8270f8 into OSGeo:master Jan 11, 2024
@kylebarron kylebarron mentioned this pull request Jan 24, 2024
Closed
@jorisvandenbossche jorisvandenbossche mentioned this pull request Jan 25, 2024
Open
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@jorisvandenbossche jorisvandenbossche jorisvandenbossche left review comments

@kylebarron kylebarron kylebarron left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

3.9.0

Development

Successfully merging this pull request may close these issues.

4 participants

@rouault @coveralls @kylebarron @jorisvandenbossche