ARROW-819: Public Cython and C++ API in the style of lxml, arrow::py::import_pyarrow method#680
Closed
wesm wants to merge 5 commits intoapache:masterfrom
Closed
ARROW-819: Public Cython and C++ API in the style of lxml, arrow::py::import_pyarrow method#680wesm wants to merge 5 commits intoapache:masterfrom
wesm wants to merge 5 commits intoapache:masterfrom
Conversation
…c C API easier Change-Id: I3879a8f0546c88959f4468f69f41044455c7978a
Change-Id: Ifc6b97f7bae539c7c072e6c4ddbba8b8dbd06bc5
Change-Id: I8eb4db35868420d374e03ebb9305e2884b3cac44
Change-Id: I21e6a311273cda7eb793f8430581ba06b2393913
Member
Author
|
OK, now we have $ nm -g debug/libarrow_python.so | c++filt | grep import
000000000010e802 T arrow::py::import_pyarrow()
$ nm -g debug/libarrow_python.so | c++filt | grep wrap
000000000010e885 T arrow::py::wrap_array(std::shared_ptr<arrow::Array> const&)
000000000010e849 T arrow::py::wrap_field(std::shared_ptr<arrow::Field> const&)
000000000010e8df T arrow::py::wrap_table(std::shared_ptr<arrow::Table> const&)
000000000010e80d T arrow::py::wrap_buffer(std::shared_ptr<arrow::Buffer> const&)
000000000010e8c1 T arrow::py::wrap_column(std::shared_ptr<arrow::Column> const&)
000000000010e867 T arrow::py::wrap_schema(std::shared_ptr<arrow::Schema> const&)
000000000010e8a3 T arrow::py::wrap_tensor(std::shared_ptr<arrow::Tensor> const&)
000000000010e82b T arrow::py::wrap_data_type(std::shared_ptr<arrow::DataType> const&)
000000000010e8fd T arrow::py::wrap_record_batch(std::shared_ptr<arrow::RecordBatch> const&)I haven't tried using these in a C extension yet (open to ideas how to test this), but I think this works! |
wesm
changed the title
Member
Author
|
This conflicts with #679; I can rebase this after that is merged. |
xhochy
approved these changes
May 13, 2017
Member
xhochy
left a comment
xhochy
left a comment
There was a problem hiding this comment.
👍 , amazing work! This is probably the last piece I needed for the turbodbc work to continue.
| return ::import_pyarrow__lib(); | ||
| } | ||
|
|
||
| PyObject* wrap_buffer(const std::shared_ptr<Buffer>& buffer) { |
Member
There was a problem hiding this comment.
You won't be able to call these from C without wrapping them into extern "C" { .. }" due to the name mangling. Also I would be astonished if plain C code could instantiate std::shared_ptr`.
Member
Author
There was a problem hiding this comment.
I updated the PR title and comments to indicate this is a C++ API and not a C API (because the arguments are STL types)
wesm
changed the title
Change-Id: Icad57e6d5e9ee5302e8623664c3c58ac363bdd69
Member
Author
|
I will merge this as soon as I get a green build, then rebase #679 since there may be some more discussion about names for the IPC reader/writer classes |
jeffknupp
pushed a commit
to jeffknupp/arrow
that referenced
this pull request
Jun 3, 2017
…:import_pyarrow method I have been looking at LXML's approach to creating both a public Cython API and C++ API https://github.com/lxml/lxml While this may seem like a somewhat radical reorganization of the code, putting all of the main symbols in a single Cython extension makes generating a C++ API for them significantly simpler. By using `.pxi` files we can break the codebase into as small pieces as we like (as long as there are no circular dependencies). As a convenient side effect, the build times are shorter. Author: Wes McKinney <[email protected]> Closes apache#680 from wesm/ARROW-819 and squashes the following commits: 9e6ee24 [Wes McKinney] Fix up optional extensions cff757d [Wes McKinney] Expose pyarrow C API in arrow/python/pyarrow.h b39d19c [Wes McKinney] Fix test suite. Move _config into lib ff1b5e5 [Wes McKinney] Rename things a bit d4a8391 [Wes McKinney] Reorganize Cython code in the style of lxml so make declaring a public C API easier
pribor
pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
Bumps `dep.junit.jupiter.version` from 5.12.0 to 5.12.1. Updates `org.junit.jupiter:junit-jupiter-engine` from 5.12.0 to 5.12.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/junit-team/junit5/releases">org.junit.jupiter:junit-jupiter-engine's releases</a>.</em></p> <blockquote> <p>JUnit 5.12.1 = Platform 1.12.1 + Jupiter 5.12.1 + Vintage 5.12.1</p> <p>See <a href="https://junit.org/junit5/docs/5.12.1/release-notes/">Release Notes</a>.</p> <p><strong>Full Changelog</strong>: <a href="https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1">https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/junit-team/junit5/commit/ba9c9ae111f5f06645ebcda2bb831c8e093cc002"><code>ba9c9ae</code></a> Release 5.12.1</li> <li><a href="https://github.com/junit-team/junit5/commit/e28ad4a89047da0aca6aca32b0e45ac8309954e1"><code>e28ad4a</code></a> Finalize 5.12.1 release notes</li> <li><a href="https://github.com/junit-team/junit5/commit/1044e2c63c435f735ab78dbbca09e78b13fb5f80"><code>1044e2c</code></a> Move entry to 5.12.1 release notes</li> <li><a href="https://github.com/junit-team/junit5/commit/bea821d69ae8848aa21f7ee10692c6185abfc9ac"><code>bea821d</code></a> Fix Javadoc formatting</li> <li><a href="https://github.com/junit-team/junit5/commit/1a1a67daee1507ae85a25ec0f57bce98d6824b39"><code>1a1a67d</code></a> Set stable module name for the standalone JAR file</li> <li><a href="https://github.com/junit-team/junit5/commit/062d3fa175d94010c24054fb44b2566955ba95ae"><code>062d3fa</code></a> Remove internal packages from API reports</li> <li><a href="https://github.com/junit-team/junit5/commit/74521ab55ca19269e4cf628ed738d0382a1699c9"><code>74521ab</code></a> Introduce <code>ExtensionContext.getEnclosingTestClasses()</code></li> <li><a href="https://github.com/junit-team/junit5/commit/1e135b9d37a88bc0319b98ddcfc050fc0cf89a98"><code>1e135b9</code></a> Create initial 5.12.1 release notes from template</li> <li><a href="https://github.com/junit-team/junit5/commit/d540e4ff3aeb13a3b3167515efa6ed1ec00b0773"><code>d540e4f</code></a> Link to correct method</li> <li><a href="https://github.com/junit-team/junit5/commit/31d60b7751e415853b582a275fe626b0358ab939"><code>31d60b7</code></a> Split API reports by module and package</li> <li>Additional commits viewable in <a href="https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1">compare view</a></li> </ul> </details> <br /> Updates `org.junit.jupiter:junit-jupiter-api` from 5.12.0 to 5.12.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/junit-team/junit5/releases">org.junit.jupiter:junit-jupiter-api's releases</a>.</em></p> <blockquote> <p>JUnit 5.12.1 = Platform 1.12.1 + Jupiter 5.12.1 + Vintage 5.12.1</p> <p>See <a href="https://junit.org/junit5/docs/5.12.1/release-notes/">Release Notes</a>.</p> <p><strong>Full Changelog</strong>: <a href="https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1">https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/junit-team/junit5/commit/ba9c9ae111f5f06645ebcda2bb831c8e093cc002"><code>ba9c9ae</code></a> Release 5.12.1</li> <li><a href="https://github.com/junit-team/junit5/commit/e28ad4a89047da0aca6aca32b0e45ac8309954e1"><code>e28ad4a</code></a> Finalize 5.12.1 release notes</li> <li><a href="https://github.com/junit-team/junit5/commit/1044e2c63c435f735ab78dbbca09e78b13fb5f80"><code>1044e2c</code></a> Move entry to 5.12.1 release notes</li> <li><a href="https://github.com/junit-team/junit5/commit/bea821d69ae8848aa21f7ee10692c6185abfc9ac"><code>bea821d</code></a> Fix Javadoc formatting</li> <li><a href="https://github.com/junit-team/junit5/commit/1a1a67daee1507ae85a25ec0f57bce98d6824b39"><code>1a1a67d</code></a> Set stable module name for the standalone JAR file</li> <li><a href="https://github.com/junit-team/junit5/commit/062d3fa175d94010c24054fb44b2566955ba95ae"><code>062d3fa</code></a> Remove internal packages from API reports</li> <li><a href="https://github.com/junit-team/junit5/commit/74521ab55ca19269e4cf628ed738d0382a1699c9"><code>74521ab</code></a> Introduce <code>ExtensionContext.getEnclosingTestClasses()</code></li> <li><a href="https://github.com/junit-team/junit5/commit/1e135b9d37a88bc0319b98ddcfc050fc0cf89a98"><code>1e135b9</code></a> Create initial 5.12.1 release notes from template</li> <li><a href="https://github.com/junit-team/junit5/commit/d540e4ff3aeb13a3b3167515efa6ed1ec00b0773"><code>d540e4f</code></a> Link to correct method</li> <li><a href="https://github.com/junit-team/junit5/commit/31d60b7751e415853b582a275fe626b0358ab939"><code>31d60b7</code></a> Split API reports by module and package</li> <li>Additional commits viewable in <a href="https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1">compare view</a></li> </ul> </details> <br /> Updates `org.junit.jupiter:junit-jupiter-params` from 5.12.0 to 5.12.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/junit-team/junit5/releases">org.junit.jupiter:junit-jupiter-params's releases</a>.</em></p> <blockquote> <p>JUnit 5.12.1 = Platform 1.12.1 + Jupiter 5.12.1 + Vintage 5.12.1</p> <p>See <a href="https://junit.org/junit5/docs/5.12.1/release-notes/">Release Notes</a>.</p> <p><strong>Full Changelog</strong>: <a href="https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1">https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/junit-team/junit5/commit/ba9c9ae111f5f06645ebcda2bb831c8e093cc002"><code>ba9c9ae</code></a> Release 5.12.1</li> <li><a href="https://github.com/junit-team/junit5/commit/e28ad4a89047da0aca6aca32b0e45ac8309954e1"><code>e28ad4a</code></a> Finalize 5.12.1 release notes</li> <li><a href="https://github.com/junit-team/junit5/commit/1044e2c63c435f735ab78dbbca09e78b13fb5f80"><code>1044e2c</code></a> Move entry to 5.12.1 release notes</li> <li><a href="https://github.com/junit-team/junit5/commit/bea821d69ae8848aa21f7ee10692c6185abfc9ac"><code>bea821d</code></a> Fix Javadoc formatting</li> <li><a href="https://github.com/junit-team/junit5/commit/1a1a67daee1507ae85a25ec0f57bce98d6824b39"><code>1a1a67d</code></a> Set stable module name for the standalone JAR file</li> <li><a href="https://github.com/junit-team/junit5/commit/062d3fa175d94010c24054fb44b2566955ba95ae"><code>062d3fa</code></a> Remove internal packages from API reports</li> <li><a href="https://github.com/junit-team/junit5/commit/74521ab55ca19269e4cf628ed738d0382a1699c9"><code>74521ab</code></a> Introduce <code>ExtensionContext.getEnclosingTestClasses()</code></li> <li><a href="https://github.com/junit-team/junit5/commit/1e135b9d37a88bc0319b98ddcfc050fc0cf89a98"><code>1e135b9</code></a> Create initial 5.12.1 release notes from template</li> <li><a href="https://github.com/junit-team/junit5/commit/d540e4ff3aeb13a3b3167515efa6ed1ec00b0773"><code>d540e4f</code></a> Link to correct method</li> <li><a href="https://github.com/junit-team/junit5/commit/31d60b7751e415853b582a275fe626b0358ab939"><code>31d60b7</code></a> Split API reports by module and package</li> <li>Additional commits viewable in <a href="https://github.com/junit-team/junit5/compare/r5.12.0...r5.12.1">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I have been looking at LXML's approach to creating both a public Cython API and C++ API
https://github.com/lxml/lxml
While this may seem like a somewhat radical reorganization of the code, putting all of the main symbols in a single Cython extension makes generating a C++ API for them significantly simpler. By using
.pxifiles we can break the codebase into as small pieces as we like (as long as there are no circular dependencies). As a convenient side effect, the build times are shorter.