Skip to content

Commit d25e184

Browse files
timsaucerclaude
timsaucer
and
claude
committed
Name the codecs that actually park live objects
The durable-metadata section named `datafusion-ffi-example` and `datafusion-ffi-query-planner-example` as the two codecs that keep a process-local `HashMap` of live providers. The second one does not: `grep -rn "HashMap\|OnceLock\|TOKEN"` over its `src/` is empty. It claims `node.is::<DistributedExec>()` narrowly and writes a fixed durable marker, which is the behaviour this section recommends. The guide was holding up a correct codec as a counter-example. The count was right and the attribution was wrong. Both offending codecs are in `datafusion-ffi-example` -- the logical one for table providers and the physical one for execution plan nodes -- so the paragraph now names the crate once and distinguishes its two codecs. "Providers" becomes "objects", since one of the two parks plans rather than providers. Also connects the broad-claim discussion 130 lines further down to the registry it forces. The two were written as separate faults, but the registry is downstream of the claim: a codec handed a node it cannot inspect has nothing to write down about it, so parking the object is all that is left. Saying so is what stops a reader treating the registry as an independent shortcut they might take deliberately. Deliberately minimal. Fixing the logical codec, quarantining the `ForeignExecutionPlan` arm, and rewriting this section around the result are tracked in #1723 -- the rewrite has to describe facts the first two change, so patching further here would only be undone. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent e26520d commit d25e184

1 file changed

Lines changed: 17 additions & 12 deletions

File tree

‎docs/source/extension-guide/codecs.md‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -58,17 +58,17 @@ Your payload has to be enough to rebuild the object somewhere your process is
5858
not. Write the metadata a fresh instance can be constructed from — a path, a
5959
connection string, a schema, the options the object was created with.
6060

61-
Two of the example codecs in this repository do not do this, and it is worth
62-
knowing before copying them. `datafusion-ffi-example` and
63-
`datafusion-ffi-query-planner-example` keep a process-local `HashMap` of live
64-
providers and encode an integer token into it: encoding inserts, decoding
65-
removes. That makes Rust type identity observable across three separately
66-
loaded libraries in one test, which is what those examples exist to show. It
67-
also means a decode consumes its token, so the same bytes cannot be decoded
68-
twice, one encoded plan cannot fan out to several readers, and a plan that
69-
never reaches a decoder keeps its provider alive for the life of the process.
70-
A real codec has none of those properties because it does not park the object
71-
anywhere.
61+
Two codecs in this repository do not do this, and it is worth knowing before
62+
copying them. Both are in `datafusion-ffi-example`: its logical codec, for
63+
table providers, and its physical codec, for execution plan nodes. Each keeps
64+
a process-local `HashMap` of live objects and encodes an integer token into
65+
it: encoding inserts, decoding removes. That makes Rust type identity
66+
observable across three separately loaded libraries in one test, which is what
67+
that example exists to show. It also means a decode consumes its token, so the
68+
same bytes cannot be decoded twice, one encoded plan cannot fan out to several
69+
readers, and a plan that never reaches a decoder keeps its object alive for
70+
the life of the process. A real codec has none of those properties because it
71+
does not park the object anywhere.
7272

7373
For one that does it properly, read
7474
[`examples/distributed/storage-library`](https://github.com/apache/datafusion-python/tree/main/examples/distributed/storage-library).
@@ -183,7 +183,12 @@ after it. The query still succeeds. What changes is which library wrote the
183183
bytes — so a plan that has to decode in another process now needs whichever
184184
library happened to win, not the one whose node it is.
185185
`MyPhysicalExtensionCodec` in [`datafusion-ffi-example`] claims this way, and
186-
the query-planner example's test suite pins the consequence.
186+
the query-planner example's test suite pins the consequence. It also pays for
187+
it with the token registry described in
188+
{ref}`extension_codec_durable_metadata`: a codec handed a node it cannot
189+
inspect has nothing to write down about it, so parking the object is all that
190+
is left. The registry is a consequence of the broad claim, not a separate
191+
choice.
187192

188193
Two rules of thumb:
189194

0 commit comments

Comments
 (0)