fix(python/adbc_driver_postgresql): handle kwargs in dbapi connect #2700

fwojciec · 2025-04-13T00:16:55Z

But I've encountered some issues when writing a test for the change:

there are currently no supported db_kwargs beyond driver and uri - which are set explicitly
regarding conn_kwargs - only two appear to be supported:
- adbc.connection.autocommit - which is configurable explicitly
- adbc.connection.db_schema - which seems to cause a SEGFAULT when I add a test that tries to set this using conn_kwargs

The SEGFAULT happens only when I'm trying to set this specific option, when I change the key I get the expected adbc_driver_manager.NotSupportedError: NOT_IMPLEMENTED: [libpq] Unknown option adbc.connection.db_schema2 error. All other tests in the suite pass, so I don't think it's a general setup issue.

the test is commented out in the tests/test_dbapi.py file. pytest output:

========================================================================= test session starts ==========================================================================
platform darwin -- Python 3.11.12, pytest-8.3.5, pluggy-1.5.0 -- /opt/homebrew/Caskroom/miniforge/base/envs/adbc/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/filip/code/github/arrow-adbc/python/adbc_driver_postgresql
configfile: pyproject.toml
collected 26 items

tests/test_dbapi.py::test_conn_current_catalog PASSED                                                                                                            [  3%]
tests/test_dbapi.py::test_conn_current_db_schema PASSED                                                                                                          [  7%]
tests/test_dbapi.py::test_conn_change_db_schema PASSED                                                                                                           [ 11%]
tests/test_dbapi.py::test_conn_get_info PASSED                                                                                                                   [ 15%]
tests/test_dbapi.py::test_query_batch_size PASSED                                                                                                                [ 19%]
tests/test_dbapi.py::test_query_cancel PASSED                                                                                                                    [ 23%]
tests/test_dbapi.py::test_query_execute_schema PASSED                                                                                                            [ 26%]
tests/test_dbapi.py::test_query_invalid PASSED                                                                                                                   [ 30%]
tests/test_dbapi.py::test_query_trivial PASSED                                                                                                                   [ 34%]
tests/test_dbapi.py::test_stmt_ingest PASSED                                                                                                                     [ 38%]
tests/test_dbapi.py::test_stmt_ingest_dataset PASSED                                                                                                             [ 42%]
tests/test_dbapi.py::test_stmt_ingest_multi PASSED                                                                                                               [ 46%]
tests/test_dbapi.py::test_ddl PASSED                                                                                                                             [ 50%]
tests/test_dbapi.py::test_crash PASSED                                                                                                                           [ 53%]
tests/test_dbapi.py::test_reuse PASSED                                                                                                                           [ 57%]
tests/test_dbapi.py::test_ingest PASSED                                                                                                                          [ 61%]
tests/test_dbapi.py::test_ingest_schema PASSED                                                                                                                   [ 65%]
tests/test_dbapi.py::test_ingest_temporary PASSED                                                                                                                [ 69%]
tests/test_dbapi.py::test_timestamp_txn PASSED                                                                                                                   [ 73%]
tests/test_dbapi.py::test_connect_conn_kwargs_db_schema Fatal Python error: Segmentation fault

Current thread 0x00000001eba94840 (most recent call first):
  File "/Users/filip/code/github/arrow-adbc/python/adbc_driver_postgresql/adbc_driver_postgresql/dbapi.py", line 122 in connect
  File "/Users/filip/code/github/arrow-adbc/python/adbc_driver_postgresql/tests/test_dbapi.py", line 463 in test_connect_conn_kwargs_db_schema
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/python.py", line 1627 in runtest
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/runner.py", line 242 in <lambda>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/runner.py", line 241 in call_and_report
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/main.py", line 337 in _main
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/lib/python3.11/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/opt/homebrew/Caskroom/miniforge/base/envs/adbc/bin/pytest", line 8 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pyarrow.lib, pyarrow._compute, pyarrow._acero, pyarrow._fs, pyarrow._csv, pyarrow._json, pyarrow._substrait, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, adbc_driver_manager._lib, adbc_driver_manager._reader, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs (total: 70)
zsh: segmentation fault  ADBC_POSTGRESQL_TEST_URI= pytest -vvx

I'm wondering if it even makes sense to "fix" this given the above?

lidavidm · 2025-04-14T00:53:01Z

Thanks for the PR. Let me debug what's going on and I'll file a new PR.

lidavidm · 2025-04-14T06:38:00Z

See #2701 (I need to add some tests)

Also, I had to tweak the test:

+    with dbapi.connect(postgres_uri, conn_kwargs={schema_key: schema_name}) as conn:
+        option_value = conn.adbc_connection.get_option(schema_key)
+        assert option_value == schema_name

fwojciec · 2025-04-14T18:32:46Z

I've updated the test in this PR to include your change from the comment above - and uncommented it. Thank you!

lidavidm · 2025-04-15T02:48:35Z

Ok, if you rebase things should be fixed

fwojciec · 2025-04-15T06:27:00Z

Hi @lidavidm,

Done! I've rebased this branch onto the latest main and force-pushed the update. Confirmed locally that all tests are passing now. Thanks!

lidavidm

Thanks!

…pache#2700) Closes apache#2696.

fwojciec requested a review from lidavidm as a code owner April 13, 2025 00:16

github-actions bot modified the milestone: ADBC Libraries 18 Apr 13, 2025

fwojciec changed the title ~~fix(python/driver/postgresql): handle kwargs in dbapi connect~~ fix(python/adbc_driver_postgresql): handle kwargs in dbapi connect Apr 13, 2025

rebase main

ded2474

fwojciec force-pushed the fix-pgsql-connect-kwargs branch from 7c1ce74 to ded2474 Compare April 15, 2025 06:22

lidavidm approved these changes Apr 15, 2025

View reviewed changes

lidavidm merged commit 8184845 into apache:main Apr 15, 2025
62 checks passed

colin-rogers-dbt pushed a commit to dbt-labs/arrow-adbc that referenced this pull request Jun 10, 2025

fix(python/adbc_driver_postgresql): handle kwargs in dbapi connect (a…

be72643

…pache#2700) Closes apache#2696.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(python/adbc_driver_postgresql): handle kwargs in dbapi connect #2700

fix(python/adbc_driver_postgresql): handle kwargs in dbapi connect #2700

Uh oh!

fwojciec commented Apr 13, 2025

Uh oh!

lidavidm commented Apr 14, 2025

Uh oh!

lidavidm commented Apr 14, 2025

Uh oh!

fwojciec commented Apr 14, 2025

Uh oh!

lidavidm commented Apr 15, 2025

Uh oh!

fwojciec commented Apr 15, 2025

Uh oh!

lidavidm left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fix(python/adbc_driver_postgresql): handle kwargs in dbapi connect #2700

fix(python/adbc_driver_postgresql): handle kwargs in dbapi connect #2700

Uh oh!

Conversation

fwojciec commented Apr 13, 2025

Uh oh!

lidavidm commented Apr 14, 2025

Uh oh!

lidavidm commented Apr 14, 2025

Uh oh!

fwojciec commented Apr 14, 2025

Uh oh!

lidavidm commented Apr 15, 2025

Uh oh!

fwojciec commented Apr 15, 2025

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants