You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3ecd78d
Browse filesBrowse the repository at this point in the historyBrowse files
* feat(udf): chunk_parts and json_pack — the etl bundles' runtime functions
Generated etl bundles (skardi-cloud crates/skardi-etl) run their ingest
jobs on THIS server, and the design's runtime audit concluded today's
engine carries everything they need except two small UDFs. This adds
exactly those, plus the pack enumeration surface:
- chunk_parts(mode, text, size[, overlap]) ->
List<Struct<chunk_idx Int32, chunk_text Utf8>> (feature: chunking,
registered beside chunk). The 0-based split ordinal is assigned
inside the UDF in one deterministic pass per value, because
DataFusion cannot plan `UNNEST ... WITH ORDINALITY`
(apache/datafusion#11419) — the plannability pin test locks the
projection-position UNNEST spelling that does work, with per-row
ordinal restart and overlap-0 reassembly asserted end to end.
- json_pack(key, value, ...) -> Utf8: the SQL-callable JSON encoder
(core has none through 54.x; datafusion-functions-json is read-only).
serde_json does the encoding, making it an injection boundary —
quotes/backslashes/control chars round-trip byte-exact in tests;
keys are literal, insertion-ordered (deterministic output); JSON
scalars + Timestamp (epoch ms) + List<Utf8> supported; non-finite
floats refuse. Registered on both server session contexts.
- SourcePackRegistry::packs(): name-sorted deterministic enumeration.
Both UDFs are also generally useful standalone (inline chunked
ingestion with stable ordinals; JSON object packing in any pipeline).
839 lib tests green with --features chunking (822 default); the
consumer-side plan-check and hybrid e2e suites live with the generator
in skardi-cloud#232.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(udf): close the runtime delivery contract per PR #193 review
- chunking joins skardi-server's DEFAULT features: generated etl
bundles call chunk_parts at job run time, and the etl PRD's contract
is that a stock server runs them — the plain cargo build and the
no-FEATURES Docker artifact now carry it (the skardi LIBRARY keeps
default = []; opt out with --no-default-features).
- Server-level wiring regression test: a pipeline calling both UDFs
must PLAN through load_server_config (the planning context) and the
same statement must EXECUTE through setup_app_state's runtime
context — the failure mode the UDF unit tests (which register
directly) cannot catch.
- serde_json's preserve_order enabled explicitly on skardi: the
argument-order-is-object-order contract previously held only through
transitive feature unification (bson), one dep change away from
silent lexicographic re-sorting. New test pins ordering with keys
where insertion order disagrees with lexicographic order (z, a, m —
the existing id/title/updated assertion passes under either).
- User docs: chunk_parts joins docs/chunk.md (signature, ordinal
semantics, the plannable UNNEST spelling and why), json_pack gets
docs/json_pack.md (contract, injection-boundary rationale, examples);
README links both.
Co-Authored-By: Claude Fable 5 <[email protected]>
* fix(udf): PR #193 round-2 — merge-safe packs pin, view-string + list-literal support, chunk arg dedup
- packs pin: main gained the feishu pack while this branch was in
flight, so the roster assertion would have gone red on the merged
tree. Absorbed main; the test now asserts sortedness independently of
the roster (survives future pack additions) plus the explicit
completeness list including feishu.
- json_pack: Utf8View accepted for keys and values (DataFusion 52
carries computed string expressions as view types — the filters.rs
precedent); a const-folded make_array literal (ScalarValue::List)
now encodes through the same utf8_list_value conversion as list
columns, honoring the documented "column or literal" contract.
- json_pack coverage: every timestamp granularity pinned by value
(seconds ×1000 checked, micro/nano floor, pre-epoch sign), the
seconds-overflow refusal, List<Utf8> with null element and null
list, the make_array literal spelling, non-Utf8 list elements
refused with the type named, and view-typed keys/values byte-exact.
- chunking: the ~55 duplicated argument-parsing lines collapse into
one parse_chunk_args shared by chunk and chunk_parts, so the two
UDFs' argument contracts cannot drift; the shared text reader also
accepts Utf8View columns and scalars.
- style: register_json_pack_udf imported via `use` in server
config.rs/server.rs instead of inline crate paths.
Co-Authored-By: Claude Fable 5 <[email protected]>
---------
Co-authored-by: Claude Fable 5 <[email protected]>
Copy file name to clipboardExpand all lines: README.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -439,7 +439,8 @@ We're **building in public**. `[x]` means shipped today, `[ ]` means open for co
439
439
- [x] Hybrid search — combine keyword and semantic search results in one SQL query (RRF merge), no Python re-ranking layer
440
440
- [x] Inline embeddings — `candle()` UDF (local GGUF / Candle models, or remote embedding APIs) called inside SQL, so content + vector stay on the same row atomically
441
441
- [x] ONNX inference — `onnx_predict` UDF for inline model predictions in SQL
442
-
- [x] Chunking UDF — `chunk()` with character / markdown splitters (via [`text-splitter`](https://crates.io/crates/text-splitter)) so ingestion can chunk inline in SQL ([docs](docs/chunk.md)); token / code splitters next
442
+
- [x] Chunking UDFs — `chunk()` and index-carrying `chunk_parts()` with character / markdown splitters (via [`text-splitter`](https://crates.io/crates/text-splitter)) so ingestion can chunk inline in SQL ([docs](docs/chunk.md)); token / code splitters next
443
+
- [x] JSON encoding UDF — `json_pack()` builds JSON objects in SQL through `serde_json` (deterministic key order, no injection path) ([docs](docs/json_pack.md))
443
444
- [ ] Memory primitive — give your agent a memory store (keyword + semantic recall, TTL/expiration, per-session provenance) defined in one YAML block
0 commit comments