Skip to content

Commit f199d53

Browse files
Keep import duckdb from opening a database and fix the default connection race (duckdb#680)
Fixes duckdb#612. Importing duckdb opened the default in-memory database and started its thread pools, because the DB API NUMBER type object parsed the strings "BIGNUM" and "DECIMAL" at module scope, and parsing a type string without a connection runs on the default connection. Logical types belong to a connection's catalog, so the DB API categories no longer hold type instances at all: STRING, NUMBER, DATETIME and BINARY are sets of type ids, and `column_type == duckdb.NUMBER` tests the id of the type a connection's result handed back. Import now starts no thread. Consequences: - Every DECIMAL is a NUMBER. Before, only DECIMAL(18,3) matched, because "DECIMAL" parsed to that one width and scale. - A JSON column is a STRING (its id is varchar), and an aggregate state matches by the id of its value. - `DBAPITypeObject` takes a collection of type ids; its `types` list is gone. Comparing a column type with a DB API type object, column type on the left, printed a nanobind implicit conversion failure to stderr on every call. A third equality overload handles that operand; every equality result is unchanged. Removing the import-time connection exposed a pre-existing deadlock: the default connection holder kept its mutex while connecting, and connecting releases the GIL, so two threads whose first `duckdb.sql()` created the default connection hung the process (before, only `duckdb.close()` followed by such a race reached it). `set_default_connection` racing a reader deadlocked the same way, on 1.5.6 too. The holder now keeps its mutex only around pointer reads and swaps; connecting and destroying happen outside it. A finalizer that fetches the default connection no longer deadlocks when the default is replaced, and once the module's cleanup has run while the interpreter finalizes, the holder neither opens nor keeps a connection, so nothing is left open at exit. A first-use race may briefly open one database per racing thread.
2 parents 6118c58 + 5fa1cca commit f199d53

5 files changed

Lines changed: 643 additions & 57 deletions

File tree

‎duckdb/_dbapi_type_object.py‎

Lines changed: 53 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,9 @@
2525
- DuckDB Type System: https://duckdb.org/docs/sql/data_types/overview
2626
"""
2727

28-
from duckdb import sqltypes
28+
import collections.abc
29+
30+
from duckdb.sqltypes import DuckDBPyType
2931

3032

3133
class DBAPITypeObject:
@@ -35,36 +37,40 @@ class DBAPITypeObject:
3537
It allows checking whether a specific DuckDB type belongs to a broader
3638
category like STRING, NUMBER, DATETIME, etc.
3739
38-
The type object supports equality comparison with DuckDBPyType instances,
39-
returning True if the type belongs to this category.
40+
A category is a set of type ids, as DuckDBPyType.id gives them. A DuckDBPyType
41+
belongs to the category when its id is in the set, whatever its parameters
42+
(such as the width and scale of a DECIMAL) or its alias (such as JSON, a VARCHAR).
4043
4144
Args:
42-
types: A list of DuckDBPyType instances that belong to this type category.
45+
type_ids: Lowercase type ids of the types that belong to this category.
4346
4447
Example:
45-
>>> string_types = DBAPITypeObject([sqltypes.VARCHAR, sqltypes.CHAR])
46-
>>> result = sqltypes.VARCHAR == string_types # True
47-
>>> result = sqltypes.INTEGER == string_types # False
48+
>>> string_types = DBAPITypeObject({"varchar"})
49+
>>> result = duckdb.sqltypes.VARCHAR == string_types # True
50+
>>> result = duckdb.sqltypes.INTEGER == string_types # False
4851
4952
Note:
5053
This follows the DB API 2.0 specification where type objects are compared
5154
using equality operators rather than isinstance() checks.
5255
"""
5356

54-
def __init__(self, types: list[sqltypes.DuckDBPyType]) -> None:
57+
def __init__(self, type_ids: collections.abc.Iterable[str]) -> None:
5558
"""Initialize a DB API type object.
5659
5760
Args:
58-
types: List of DuckDB types that belong to this category.
61+
type_ids: Lowercase type ids, as DuckDBPyType.id gives them, of the types that belong to this category.
5962
"""
60-
self.types = types
63+
if isinstance(type_ids, str):
64+
msg = "type_ids must be a collection of type ids, not a single str"
65+
raise TypeError(msg)
66+
self.type_ids = frozenset(type_ids)
6167

6268
def __eq__(self, other: object) -> bool:
6369
"""Check if a DuckDB type belongs to this type category.
6470
6571
This method implements the DB API 2.0 type checking mechanism.
66-
It returns True if the other object is a DuckDBPyType that
67-
is contained in this type category.
72+
It returns True if the other object is a DuckDBPyType whose id
73+
is in this type category.
6874
6975
Args:
7076
other: The object to compare, typically a DuckDBPyType instance.
@@ -73,34 +79,32 @@ def __eq__(self, other: object) -> bool:
7379
True if other is a DuckDBPyType in this category, False otherwise.
7480
7581
Example:
76-
>>> NUMBER == sqltypes.INTEGER # True
77-
>>> NUMBER == sqltypes.VARCHAR # False
82+
>>> NUMBER == duckdb.sqltypes.INTEGER # True
83+
>>> NUMBER == duckdb.sqltypes.VARCHAR # False
7884
"""
79-
if isinstance(other, sqltypes.DuckDBPyType):
80-
return other in self.types
81-
return False
85+
return isinstance(other, DuckDBPyType) and other.id in self.type_ids
8286

8387
def __repr__(self) -> str:
8488
"""Return a string representation of this type object.
8589
8690
Returns:
87-
A string showing the type object and its contained DuckDB types.
91+
A string listing the type ids of this category, uppercase and sorted.
8892
8993
Example:
9094
>>> repr(STRING)
9195
'<DBAPITypeObject [VARCHAR]>'
9296
"""
93-
return f"<DBAPITypeObject [{','.join(str(x) for x in self.types)}]>"
97+
return f"<DBAPITypeObject [{','.join(sorted(type_id.upper() for type_id in self.type_ids))}]>"
9498

9599

96100
# Define the standard DB API 2.0 type objects for DuckDB
97101

98-
STRING = DBAPITypeObject([sqltypes.VARCHAR])
102+
STRING = DBAPITypeObject({"varchar"})
99103
"""
100104
STRING type object for text-based database columns.
101105
102106
This type object represents all string/text types in DuckDB. Currently includes:
103-
- VARCHAR: Variable-length character strings
107+
- VARCHAR: Variable-length character strings, and the types that alias it, such as JSON
104108
105109
Use this to check if a column contains textual data that should be handled
106110
as Python strings.
@@ -113,22 +117,22 @@ def __repr__(self) -> str:
113117
"""
114118

115119
NUMBER = DBAPITypeObject(
116-
[
117-
sqltypes.TINYINT,
118-
sqltypes.UTINYINT,
119-
sqltypes.SMALLINT,
120-
sqltypes.USMALLINT,
121-
sqltypes.INTEGER,
122-
sqltypes.UINTEGER,
123-
sqltypes.BIGINT,
124-
sqltypes.UBIGINT,
125-
sqltypes.HUGEINT,
126-
sqltypes.UHUGEINT,
127-
sqltypes.DuckDBPyType("BIGNUM"),
128-
sqltypes.DuckDBPyType("DECIMAL"),
129-
sqltypes.FLOAT,
130-
sqltypes.DOUBLE,
131-
]
120+
{
121+
"tinyint",
122+
"utinyint",
123+
"smallint",
124+
"usmallint",
125+
"integer",
126+
"uinteger",
127+
"bigint",
128+
"ubigint",
129+
"hugeint",
130+
"uhugeint",
131+
"bignum",
132+
"decimal",
133+
"float",
134+
"double",
135+
}
132136
)
133137
"""
134138
NUMBER type object for numeric database columns.
@@ -144,7 +148,7 @@ def __repr__(self) -> str:
144148
145149
Decimal Types:
146150
- BIGNUM: Arbitrary precision integers
147-
- DECIMAL: Fixed-point decimal numbers
151+
- DECIMAL: Fixed-point decimal numbers, of every width and scale
148152
149153
Floating Point Types:
150154
- FLOAT: 32-bit floating point
@@ -161,16 +165,16 @@ def __repr__(self) -> str:
161165
"""
162166

163167
DATETIME = DBAPITypeObject(
164-
[
165-
sqltypes.DATE,
166-
sqltypes.TIME,
167-
sqltypes.TIME_TZ,
168-
sqltypes.TIMESTAMP,
169-
sqltypes.TIMESTAMP_TZ,
170-
sqltypes.TIMESTAMP_NS,
171-
sqltypes.TIMESTAMP_MS,
172-
sqltypes.TIMESTAMP_S,
173-
]
168+
{
169+
"date",
170+
"time",
171+
"time with time zone",
172+
"timestamp",
173+
"timestamp with time zone",
174+
"timestamp_ns",
175+
"timestamp_ms",
176+
"timestamp_s",
177+
}
174178
)
175179
"""
176180
DATETIME type object for date and time database columns.
@@ -201,7 +205,7 @@ def __repr__(self) -> str:
201205
>>> cursor.description[2][1] == DATETIME # Check if third column is date/time
202206
"""
203207

204-
BINARY = DBAPITypeObject([sqltypes.BLOB])
208+
BINARY = DBAPITypeObject({"blob"})
205209
"""
206210
BINARY type object for binary data database columns.
207211

‎src/include/duckdb_python/pyconnection/pyconnection.hpp‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,9 +49,13 @@ struct DefaultConnectionHolder {
4949
public:
5050
std::shared_ptr<DuckDBPyConnection> Get();
5151
void Set(std::shared_ptr<DuckDBPyConnection> conn);
52+
//! Once the module's cleanup ran and the interpreter is finalizing, the holder neither opens nor keeps a
53+
//! connection, because nothing would close it
54+
void Release();
5255

5356
private:
5457
std::shared_ptr<DuckDBPyConnection> connection;
58+
bool released = false;
5559
mutex l;
5660
};
5761

‎src/pyconnection.cpp‎

Lines changed: 67 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2155,18 +2155,77 @@ void DuckDBPyConnection::LoadExtension(const string &extension) {
21552155
ExtensionHelper::LoadExternalExtension(*connection.context, extension_opts);
21562156
}
21572157

2158+
// sys.is_finalizing is asked because Py_IsFinalizing is public only from Python 3.13; without sys the interpreter is
2159+
// finalizing.
2160+
static bool InterpreterIsFinalizing() {
2161+
PyObject *is_finalizing = PySys_GetObject("is_finalizing");
2162+
if (!is_finalizing) {
2163+
return true;
2164+
}
2165+
PyObject *result = PyObject_CallNoArgs(is_finalizing);
2166+
if (!result) {
2167+
PyErr_Clear();
2168+
return true;
2169+
}
2170+
const int truth = PyObject_IsTrue(result);
2171+
Py_DECREF(result);
2172+
if (truth < 0) {
2173+
PyErr_Clear();
2174+
return true;
2175+
}
2176+
return truth != 0;
2177+
}
2178+
2179+
// The mutex guards only reads and swaps of the pointer: opening or destroying a connection releases the GIL and can run
2180+
// Python code that fetches the default connection again, so both happen after the mutex is released.
21582181
std::shared_ptr<DuckDBPyConnection> DefaultConnectionHolder::Get() {
2159-
lock_guard<mutex> guard(l);
2160-
if (!connection || connection->con.ConnectionIsClosed()) {
2161-
nb::dict config_dict;
2162-
connection = DuckDBPyConnection::Connect(nb::str(":memory:"), false, config_dict);
2182+
bool was_released;
2183+
{
2184+
lock_guard<mutex> guard(l);
2185+
if (connection && !connection->con.ConnectionIsClosed()) {
2186+
return connection;
2187+
}
2188+
was_released = released;
2189+
}
2190+
if (was_released && InterpreterIsFinalizing()) {
2191+
throw ConnectionException("the default connection is not available during interpreter shutdown");
2192+
}
2193+
nb::dict config_dict;
2194+
auto opened = DuckDBPyConnection::Connect(nb::str(":memory:"), false, config_dict);
2195+
std::shared_ptr<DuckDBPyConnection> replaced;
2196+
{
2197+
lock_guard<mutex> guard(l);
2198+
if (connection && !connection->con.ConnectionIsClosed()) {
2199+
return connection;
2200+
}
2201+
replaced = std::move(connection);
2202+
connection = opened;
21632203
}
2164-
return connection;
2204+
return opened;
21652205
}
21662206

21672207
void DefaultConnectionHolder::Set(std::shared_ptr<DuckDBPyConnection> conn) {
2168-
lock_guard<mutex> guard(l);
2169-
connection = conn;
2208+
bool was_released;
2209+
{
2210+
lock_guard<mutex> guard(l);
2211+
was_released = released;
2212+
}
2213+
if (was_released && InterpreterIsFinalizing()) {
2214+
return;
2215+
}
2216+
{
2217+
lock_guard<mutex> guard(l);
2218+
connection.swap(conn);
2219+
}
2220+
}
2221+
2222+
void DefaultConnectionHolder::Release() {
2223+
std::shared_ptr<DuckDBPyConnection> released_connection;
2224+
{
2225+
lock_guard<mutex> guard(l);
2226+
released = true;
2227+
connection.swap(released_connection);
2228+
}
21702229
}
21712230

21722231
void DuckDBPyConnection::Cursors::AddCursor(std::shared_ptr<DuckDBPyConnection> conn) {
@@ -2519,7 +2578,7 @@ void DuckDBPyConnection::Exit(DuckDBPyConnection &self, const nb::object &exc_ty
25192578
}
25202579

25212580
void DuckDBPyConnection::Cleanup() {
2522-
GetModuleState().default_connection.Set(nullptr);
2581+
GetModuleState().default_connection.Release();
25232582
GetModuleState().import_cache.reset();
25242583
GetModuleState().consumed_arrow_readers.reset();
25252584
}

‎src/typing/pytype.cpp‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -364,6 +364,19 @@ void DuckDBPyType::Initialize(nb::handle &m) {
364364
nb::is_operator());
365365
type_module.def("__eq__", &DuckDBPyType::EqualsString, "Compare two types for equality", nb::arg("other"),
366366
nb::is_operator());
367+
// Binds every other operand in nanobind's strict first pass, because its implicit conversion pass prints to
368+
// stderr each time the conversion to DuckDBPyType fails; registered after the str overload, so a plain str
369+
// operand is compared by its text and never parsed.
370+
type_module.def(
371+
"__eq__",
372+
[](const DuckDBPyType &self, const nb::object &other) -> nb::object {
373+
std::unique_ptr<DuckDBPyType> converted;
374+
if (!DuckDBPyType::TryConvert(other, converted)) {
375+
return nb::borrow(nb::handle(Py_NotImplemented));
376+
}
377+
return nb::bool_(self.Equals(*converted));
378+
},
379+
"Compare two types for equality", nb::arg("other").none(), nb::is_operator());
367380
type_module.def("__hash__", [](const DuckDBPyType &type) {
368381
auto s = type.ToString();
369382
return nb::hash(nb::str(s.c_str(), s.size()));

0 commit comments

Comments
 (0)