Skip to content

Commit 3ecd78d

Browse files
bakeyclaude
andauthored
feat(udf): chunk_parts + json_pack — runtime functions for generated etl bundles (#193)
* feat(udf): chunk_parts and json_pack — the etl bundles' runtime functions Generated etl bundles (skardi-cloud crates/skardi-etl) run their ingest jobs on THIS server, and the design's runtime audit concluded today's engine carries everything they need except two small UDFs. This adds exactly those, plus the pack enumeration surface: - chunk_parts(mode, text, size[, overlap]) -> List<Struct<chunk_idx Int32, chunk_text Utf8>> (feature: chunking, registered beside chunk). The 0-based split ordinal is assigned inside the UDF in one deterministic pass per value, because DataFusion cannot plan `UNNEST ... WITH ORDINALITY` (apache/datafusion#11419) — the plannability pin test locks the projection-position UNNEST spelling that does work, with per-row ordinal restart and overlap-0 reassembly asserted end to end. - json_pack(key, value, ...) -> Utf8: the SQL-callable JSON encoder (core has none through 54.x; datafusion-functions-json is read-only). serde_json does the encoding, making it an injection boundary — quotes/backslashes/control chars round-trip byte-exact in tests; keys are literal, insertion-ordered (deterministic output); JSON scalars + Timestamp (epoch ms) + List<Utf8> supported; non-finite floats refuse. Registered on both server session contexts. - SourcePackRegistry::packs(): name-sorted deterministic enumeration. Both UDFs are also generally useful standalone (inline chunked ingestion with stable ordinals; JSON object packing in any pipeline). 839 lib tests green with --features chunking (822 default); the consumer-side plan-check and hybrid e2e suites live with the generator in skardi-cloud#232. Co-Authored-By: Claude Fable 5 <[email protected]> * fix(udf): close the runtime delivery contract per PR #193 review - chunking joins skardi-server's DEFAULT features: generated etl bundles call chunk_parts at job run time, and the etl PRD's contract is that a stock server runs them — the plain cargo build and the no-FEATURES Docker artifact now carry it (the skardi LIBRARY keeps default = []; opt out with --no-default-features). - Server-level wiring regression test: a pipeline calling both UDFs must PLAN through load_server_config (the planning context) and the same statement must EXECUTE through setup_app_state's runtime context — the failure mode the UDF unit tests (which register directly) cannot catch. - serde_json's preserve_order enabled explicitly on skardi: the argument-order-is-object-order contract previously held only through transitive feature unification (bson), one dep change away from silent lexicographic re-sorting. New test pins ordering with keys where insertion order disagrees with lexicographic order (z, a, m — the existing id/title/updated assertion passes under either). - User docs: chunk_parts joins docs/chunk.md (signature, ordinal semantics, the plannable UNNEST spelling and why), json_pack gets docs/json_pack.md (contract, injection-boundary rationale, examples); README links both. Co-Authored-By: Claude Fable 5 <[email protected]> * fix(udf): PR #193 round-2 — merge-safe packs pin, view-string + list-literal support, chunk arg dedup - packs pin: main gained the feishu pack while this branch was in flight, so the roster assertion would have gone red on the merged tree. Absorbed main; the test now asserts sortedness independently of the roster (survives future pack additions) plus the explicit completeness list including feishu. - json_pack: Utf8View accepted for keys and values (DataFusion 52 carries computed string expressions as view types — the filters.rs precedent); a const-folded make_array literal (ScalarValue::List) now encodes through the same utf8_list_value conversion as list columns, honoring the documented "column or literal" contract. - json_pack coverage: every timestamp granularity pinned by value (seconds ×1000 checked, micro/nano floor, pre-epoch sign), the seconds-overflow refusal, List<Utf8> with null element and null list, the make_array literal spelling, non-Utf8 list elements refused with the type named, and view-typed keys/values byte-exact. - chunking: the ~55 duplicated argument-parsing lines collapse into one parse_chunk_args shared by chunk and chunk_parts, so the two UDFs' argument contracts cannot drift; the shared text reader also accepts Utf8View columns and scalars. - style: register_json_pack_udf imported via `use` in server config.rs/server.rs instead of inline crate paths. Co-Authored-By: Claude Fable 5 <[email protected]> --------- Co-authored-by: Claude Fable 5 <[email protected]>
1 parent fe4395f commit 3ecd78d

11 files changed

Lines changed: 1185 additions & 56 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -439,7 +439,8 @@ We're **building in public**. `[x]` means shipped today, `[ ]` means open for co
439439
- [x] Hybrid search — combine keyword and semantic search results in one SQL query (RRF merge), no Python re-ranking layer
440440
- [x] Inline embeddings — `candle()` UDF (local GGUF / Candle models, or remote embedding APIs) called inside SQL, so content + vector stay on the same row atomically
441441
- [x] ONNX inference — `onnx_predict` UDF for inline model predictions in SQL
442-
- [x] Chunking UDF — `chunk()` with character / markdown splitters (via [`text-splitter`](https://crates.io/crates/text-splitter)) so ingestion can chunk inline in SQL ([docs](docs/chunk.md)); token / code splitters next
442+
- [x] Chunking UDFs — `chunk()` and index-carrying `chunk_parts()` with character / markdown splitters (via [`text-splitter`](https://crates.io/crates/text-splitter)) so ingestion can chunk inline in SQL ([docs](docs/chunk.md)); token / code splitters next
443+
- [x] JSON encoding UDF — `json_pack()` builds JSON objects in SQL through `serde_json` (deterministic key order, no injection path) ([docs](docs/json_pack.md))
443444
- [ ] Memory primitive — give your agent a memory store (keyword + semantic recall, TTL/expiration, per-session provenance) defined in one YAML block
444445

445446
`3` Online serving (pipelines)

‎crates/server/Cargo.toml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,14 @@ name = "skardi-server"
1313
path = "src/main.rs"
1414

1515
[features]
16-
default = []
16+
# `chunking` is default: generated etl bundles (skardi-cloud) call
17+
# chunk_parts at job run time, and the etl PRD's contract is that a
18+
# STOCK server runs them — a feature-gated runtime function would fail
19+
# the plain `cargo build -p skardi-server` / no-FEATURES Docker artifact
20+
# at `chunk_parts`. The skardi LIBRARY keeps `default = []`; this is a
21+
# server-artifact decision only (text-splitter is a small pure-Rust
22+
# dep). Opt out with --no-default-features.
23+
default = ["chunking"]
1724
candle = ["skardi/candle"]
1825
onnx = ["skardi/onnx"]
1926
embedding = ["candle", "gguf", "onnx", "remote-embed"]

‎crates/server/src/config.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ use skardi::sources::providers::seekdb::register_seekdb_tables;
2020
use skardi::sources::providers::sqlite::register_sqlite_tables;
2121
use skardi::sources::providers::sqlx::postgres::register_postgres_tables;
2222
use skardi::sources::sql_validator::{AdhocSqlPolicy, SqlValidatorConfig, validate_sql};
23+
use skardi::util::json_pack::register_json_pack_udf;
2324
use std::collections::HashMap;
2425
use std::path::Path;
2526
use std::path::PathBuf;
@@ -443,6 +444,9 @@ pub async fn load_server_config(args: CliArgs) -> Result<ServerConfig> {
443444
// Register chunk UDF (text-splitter wrapper for inline ingestion)
444445
#[cfg(feature = "chunking")]
445446
register_chunk_udf(&mut session_ctx);
447+
// Register json_pack UDF (SQL-side JSON encoding; the etl generator's
448+
// metadata/frontmatter serialization boundary)
449+
register_json_pack_udf(&mut session_ctx);
446450

447451
// This auth layer is used only for SQL planning and is discarded after current function returns.
448452
// The live auth layer is built separately in setup_app_state.

‎crates/server/src/server.rs‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ use skardi::engine::datafusion::DataFusionEngine;
88
use skardi::jobs::{JobExecutor, JobStore, SqliteJobStore};
99
use skardi::sources::DataSourceType;
1010
use skardi::sources::sql_validator::AdhocSqlPolicy;
11+
use skardi::util::json_pack::register_json_pack_udf;
1112
use std::collections::HashMap;
1213
use std::path::PathBuf;
1314
use std::sync::{Arc, RwLock};
@@ -188,6 +189,9 @@ pub async fn setup_app_state(config: ServerConfig) -> Result<AppState> {
188189
// Register chunk UDF (text-splitter wrapper for inline ingestion)
189190
#[cfg(feature = "chunking")]
190191
register_chunk_udf(&mut session_ctx);
192+
// Register json_pack UDF (SQL-side JSON encoding; the etl generator's
193+
// metadata/frontmatter serialization boundary)
194+
register_json_pack_udf(&mut session_ctx);
191195

192196
// Build auth layer and register auth.users / auth.sessions on the runtime SessionContext.
193197
let auth_layer = AuthLayer::build(&AuthMode::from_env()).await?;
@@ -558,6 +562,68 @@ spec:
558562
assert!(!state.auth_layer.is_enabled());
559563
}
560564

565+
/// Both PRODUCTION registration paths for the etl runtime UDFs,
566+
/// exercised end to end: the planning context (`load_server_config` —
567+
/// a pipeline calling both UDFs must plan during schema inference)
568+
/// and the runtime context (`setup_app_state` — the same call must
569+
/// execute). The UDF unit tests register the functions directly, so
570+
/// they stay green if either production call is removed; THIS test
571+
/// is what fails in that case.
572+
#[cfg(feature = "chunking")]
573+
#[tokio::test]
574+
async fn production_contexts_register_the_etl_runtime_udfs() {
575+
let temp_dir = TempDir::new().unwrap();
576+
let pipeline_path = temp_dir.path().join("etl-udf-wiring.yaml");
577+
fs::write(
578+
&pipeline_path,
579+
r#"
580+
kind: pipeline
581+
metadata:
582+
name: "etl-udf-wiring"
583+
version: "1.0.0"
584+
spec:
585+
query: |
586+
SELECT json_pack('z', 'v') AS metadata,
587+
chunk_parts('character', 'hello world and beyond', 8) AS parts
588+
"#,
589+
)
590+
.unwrap();
591+
let args = CliArgs {
592+
pipeline_path: Some(pipeline_path),
593+
jobs_path: None,
594+
jobs_db_path: None,
595+
ctx_file: None,
596+
semantics_path: None,
597+
port: 8080,
598+
query_audit_db: None,
599+
query_audit_retention_days: None,
600+
};
601+
let config = crate::config::load_server_config(args)
602+
.await
603+
.expect("the PLANNING context plans chunk_parts + json_pack");
604+
assert!(config.pipelines.contains_key("etl-udf-wiring"));
605+
606+
let state = setup_app_state(config).await.expect("app state");
607+
let batches = state
608+
.session_ctx
609+
.sql(
610+
"SELECT json_pack('z', 'v') AS m, chunk_parts('character', 'hello world and beyond', 8) AS p",
611+
)
612+
.await
613+
.expect("the RUNTIME context plans both UDFs")
614+
.collect()
615+
.await
616+
.expect("and executes them");
617+
assert_eq!(batches[0].num_rows(), 1);
618+
let packed = batches[0]
619+
.column(0)
620+
.as_any()
621+
.downcast_ref::<arrow::array::StringArray>()
622+
.unwrap()
623+
.value(0);
624+
assert_eq!(packed, r#"{"z":"v"}"#);
625+
}
626+
561627
#[tokio::test]
562628
async fn test_setup_app_state_with_data_sources() {
563629
let (config, _temp_dir) = create_test_config_with_data_sources_and_temp_dir().await;

‎crates/skardi/Cargo.toml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,12 @@ tracing = { workspace = true }
4444
# pipeline
4545
log = { workspace = true }
4646
regex = { workspace = true }
47-
serde_json = { workspace = true }
47+
# preserve_order is LOAD-BEARING: json_pack's argument-order-is-object-order
48+
# contract (and the etl golden bundles' byte determinism) rides on
49+
# serde_json::Map preserving insertion order. Without the explicit feature
50+
# it held only through transitive unification (bson enables it) — a dep
51+
# change away from silently becoming lexicographic.
52+
serde_json = { workspace = true, features = ["preserve_order"] }
4853
serde_yaml = { workspace = true }
4954
# model
5055
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "stream"] }

0 commit comments

Comments
 (0)