Repository navigation
[PG] add DDL and INSERT support with compatibility and benchmark coverage - #1249
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I60005b72765058a41c7722793149e3786a6a6964
Signed-off-by: mag1cian <[email protected]> Change-Id: I173773455d5641f7bd29861de4ae0b916a6a6964
Pre-register every outcome label of lakesoul_query_requests_total, lakesoul_query_rows_total and lakesoul_query_duration_seconds at 0 when the coordinator starts, so increase()/rate() see the first burst of an outcome instead of the series appearing at its first nonzero sample. lakesoul_memory_reservation_failures_total was already registered at pool creation; use increment(0) instead of absolute(0) so a second pool in the same process cannot reset it. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: Icb291d45a0dcadcda90f6f6f7f3ed92c6a6a6964
report.py looked up increase(lakesoul_memory_reservation_failures_total[15s]) while bench.py collects the [1m] window (a [15s] window holds at most one sample at a 15 s scrape interval), so the memory table always showed an empty failures column. report.py now imports the memory queries from bench.py instead of retyping them, and the column says it is the max per 1m window. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I54e4ceb6686155de1582d6fc302de2fa6a6a6964
…igible workers
The r07 discovery outage was diagnosable only from coordinator logs. The
Kubernetes resolver now exports
lakesoul_worker_discovery_refresh_failures_total{kind="hard"|"transient"},
lakesoul_worker_discovery_snapshot_age_seconds (since the last successful
refresh, updated after every attempt, +Inf before the first) and
lakesoul_worker_discovery_eligible_workers (also set by static discovery).
The series are registered when the resolver starts, so the first outage shows
in increase(). The benchmark collects them.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I6a99af0d7882d6c2bc2ccd09501b5bb76a6a6964
…ervability overlay The processes cannot report their own OOM kills, restarts or CPU throttling, and the overlay could not show them. lakesoul-prometheus now scrapes every kubelet's cAdvisor (ClusterRole with nodes list/watch and nodes/metrics, not nodes/proxy), keeping only the LakeSoul containers' memory and CFS series, and a namespaced kube-state-metrics deployed by the overlay (--namespaces=lakesoul --resources=pods) for restart counts and last termination reasons. The LakeSoul Overview dashboard gains panels for container RSS / working set / limit vs. lakesoul_memory_pool_reserved_bytes and the pool limit, restarts and OOM kills (last_terminated_reason, since cAdvisor's OOM event series vanishes with the killed container), and CPU throttling. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: Ia3628d12ea9c8bc897592e3951c048de6a6a6964
lsbench gains a `console` scenario that runs the TPC-H queries through one lakesoul-console process (single node or distributed over --worker URLs) and turns its per-statement Elapsed output into the standard sample records, with row-count answer checks. bench.py console runs it in a Job whose containers have requests equal to limits, a native sampler sidecar, worker URLs from the lakesoul-worker EndpointSlices, and samples read from the container log so an OOM-killed console still reports finished queries. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: Ie8978dc91b36dc8845cbb1eb3b9c494f6a6a6964
…enchmarks lakesoul-console --format json prints, per statement, a column line with names and Arrow types, then one JSON array per row with the values in the table format and NULL as null. lsbench console uses it to compare results with answers.compare, as the tpch scenario does, instead of row counts. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: Ib9fd809244ee3e44e6c07663239376b26a6a6964
Huawei OBS endpoints such as cn-north-4 reject path-style requests with VirtualHostDomainRequired, and LakeSoul defaults to path style. The coordinator, worker, datagen and console manifests now read AWS_S3_VIRTUAL_HOST_STYLE from the optional secret key aws-s3-virtual-host-style; without the key, behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I4497426534454ef038903bd286dfa1046a6a6964
List the six tpch_sf{1,10}{,_pkmonth,_nopk} schemas with the datagen flags
used, and record that the SF10 pkmonth and nopk loads needed lineitem
rerun alone with a higher memory limit.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I78a683aede606cd31bcbb0b2bece9d706a6a6964
The nopk sink writes one file per tpchgen part. Modulo chunks put a slice of every part in every statement, so CHUNKS=4 wrote 4x the files at 1/4 the size and changed the scan layout. For LAYOUT=nopk, chunk k is now the key range of parts [k*PARTS/N, (k+1)*PARTS/N), computed from tpchgen's part boundaries and order key mapping, so each part is still one file. PARTS must be a multiple of CHUNKS; nation/region stay single statements. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I122cbb9acd1c7c70f3e40ed427a6f7846a6a6964
lakesoul-console built its session factory with LakeSoulSessionFactory::new, whose pool is ExecutionMemoryArgs::default(): LAKESOUL_MEMORY_LIMIT was ignored, execution was unbounded and never spilled. The console now takes ExecutionMemoryArgs (--memory-limit, --memory-pool, --vortex-decode-budget and their env) and installs the pool and Vortex decode budget like the coordinator. Sessions register the warehouse S3 store with register_s3_object_store, which skipped the disk cache; only register_object_store honored LAKESOUL_CACHE, and it returns early once a store exists. The check now lives in register_s3_object_store, so coordinator, worker and console sessions read the warehouse through the shared cache when it is set. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I463757c12ebd13d2307c0f708ad0f6866a6a6964
Under a memory limit, lakesoul-console hung on TPC-H q17 over tpch_sf1_pkmonth (2 CPU, LAKESOUL_MEMORY_LIMIT=1Gi, LAKESOUL_CACHE=1) in about one run in four: every task parked, no error. EXPLAIN ANALYZE showed the hash RepartitionExec spilling. This is apache/datafusion#24883: two input tasks spilling at once leave the shared spill pool with two open files, the reader waits on the drained older file while the batch it needs is in the newer one, and the writers park on the closed distributor gate. The fix (apache/datafusion#24891, merge commit 09a2aff7) is only in DataFusion 56. Vendor datafusion-physical-plan 55.2.0 under rust/third_party with that commit's src/spill/spill_pool.rs change and use it via [patch.crates-io]. With the patch the same configuration still spills in RepartitionExec and completed 20 of 20 runs; the crate's 20 spill_pool tests, including the upstream scenario fuzzer, pass. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I6f24b7436d4bcd3ee74bfd3cccaff5ae6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
create_lakesoul_session_config read the DataFusion environment with SessionConfig::from_env() and then forced .with_batch_size(8192), so DATAFUSION_EXECUTION_BATCH_SIZE was silently ignored and the batch size could only be changed per session with SET. 8192 is DataFusion's default, so dropping the override changes nothing when the variable is unset. The batch size governs memory held outside LAKESOUL_MEMORY_LIMIT: at a 1Gi budget, TPC-H SF10 q09 peaked at 2.2 GiB RSS with 8192 rows per batch and 0.76 GiB with 2048. Document the variable, and that workers do not read it (datafusion-distributed builds worker tasks from DataFusion's defaults plus the coordinator's distributed.shuffle_batch_size). Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I4c829aed5a8c706bf7cadbaab4c4fec06a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
A merge-on-read scan built one MergeParquetExec per range partition. Its parent pulls every input partition at once, so a table with 84 range partitions opened 84 merges together, each holding a non-spillable LakeSoulMerge reservation, whatever target_partitions was: pkmonth q01 exhausted a 1 GiB pool at target_partitions = 2. The scan now packs whole range partitions into one merge per task the scan stage may get (scan_task_budget: the distributed per-stage cap, or the listed workers; 1 for a local plan or the local fallback), each with at most target_partitions lanes. A lane runs its bucket work units one at a time, so a local scan, or one distributed task, holds at most target_partitions merges of the scan open. Work units keep their own files and range constants, so MOR version order, partial updates and range defaults are unchanged, and a range never spans two tasks. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I408225af368ecbe11b680ae84615d2236a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
…op test CoalescePartitionsExec producers keep polling after sending a batch, so a merge may already be closed when the consumer receives the first batch. Assert that a merge executed instead; the lakesoul-io lane test pins the release of a merge that is open mid-unit. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I8b91213ab7d12b1e0724d99273afed5b6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
DataFusion default 8192 without an override, DATAFUSION_EXECUTION_BATCH_SIZE when set, a session SET over both, and LakeSoul-fixed options such as prefer_hash_join over the environment. A separate test binary with one test, so changing the process environment races nothing else. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: Ibd0fa4abe6e75932f37620b250a70b456a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
TPC-H q11 orders by `value` alone, and its SF10 answer has two pairs of parts with equal values. Every q11 "mismatch" of the SF10 console runs swaps one of these pairs, and the ordered check stopped at the first difference, so the rest of the result was never checked. When every ORDER BY key is a bare output column, rows with equal keys now match as a multiset at their positions; other keys keep exact row order. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: Iac26b16e21c686e17b450273eb5c00616a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
…d profiling lakesoul-io no longer installs a global allocator. Its `jemalloc` feature installed jemalloc with heap profiling forced on via `malloc_conf` and a developer-specific `prof_prefix` path. The feature is now stats-only (`resident_memory()` for LoggedMemoryPool); the unused flamegraph and stats-print helpers and the jemalloc_pprof dependency are gone. postgres-lakesoul and lakesoul-console now own `#[global_allocator]`: jemalloc 5.3 with built-in defaults (no profiling compiled in, no decay tuning) behind a default `jemalloc` feature; `--no-default-features` gives the system allocator. `unprefixed_malloc_on_supported_platforms` routes C `malloc` through jemalloc too. Each binary logs the active allocator at startup (version, config.prof, decay options, stats.allocated). lakesoul-worker, lakesoul-io-c (JNI), lakesoul-metadata-c, Python and the libraries do not link jemalloc and keep the system allocator. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I18781042bb92889610b11c1cd17290c16a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
Interleaved q09 A/B against the glibc control image e04e8205703e on the THP=always node (2 CPU, 3 GiB, 1 GiB fair pool, batch 8192), plus the full pk suite against the 20261024 glibc baseline. jemalloc lowers peak anon RSS from 2.63 to 1.81 GiB on q09 and from 2.87 to 1.84 GiB over the suite, with identical results, spill volume and total CPU, at ~16x (q09) to ~41x (suite) more minor faults. Document the allocator, its startup log line and rollback in the k8s and benchmark READMEs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I8b2b289b138b33281aa7972c697190a96a6a6964 Co-Authored-By: ChatGPT <[email protected]> Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
Retain the existing empty workspace state on the rewritten benchmark history without changing its files. Co-Authored-By: ChatGPT <[email protected]> Change-Id: I160644a5aa0de15e35107080b1c887566a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
Change-Id: I1a2a0de511ae2a12d29b96f5eb856ec06a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
Change-Id: I9a3d72a98723593953ea84e0f71c31d66a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
Stage 3 contracts with physical proofs: - storage: append duplicates, merge superseded values, partial columns, partition-local merge, CDC tombstone removal - catalog: namespace -> database mapping, public schema, no cross-database read via search_path - vector: codec, exact fallback, ANN route and recall, probes isolation Stage 4 strict TPC-H gate: - lsbench strict driver with SF 0.1/1/10 answers and negative exits - tpch-compat workflow; PK single coordinator, nopk/pkmonth dual worker Session and protocol fixes: - connection-local SET/RESET/DISCARD TIME ZONE with startup defaults and plan invalidation on settings generation - SHOW Describe reports the column it executes: hooks claim SHOW at Parse, responses carry text (OID 25) like PostgreSQL Change-Id: Ia029cbf7b6799a40c97170ba3b90152e6a6a6964 Co-Authored-By: DeepSeek V4 Flash <[email protected]> Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
The RemoveSubqueryFromProjection removal now lives on the datafusion-postgres fork (mag1c1an1/datafusion-postgres, lakesoul-shared-static-tables, commit 03441f8) instead of in a local copy of the crate. datafusion-pg-catalog is a path dependency inside that fork workspace, so pinning a newer rev is enough. - bump the datafusion-postgres rev to 03441f8 - drop the [patch."https://github.com/mag1c1an1/datafusion-postgres.git"] entry - drop the workspace exclude for rust/third_party/datafusion-pg-catalog - delete rust/third_party/datafusion-pg-catalog (96 files, -9k lines) Verified after the switch: --bins 121 passed / 1 ignored and the six targets 16 passed / 1 ignored in both merge configurations on PostgreSQL 17.10; clippy --bin postgres-lakesoul --tests --no-deps -- -D warnings clean; metadata, reference schemas and connections back to zero. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I0a2387bb227ba54c1feb3b75c529132c6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
Add src/ddl.rs: a statement hook, installed per connection ahead of the
read-only guard, that answers PostgreSQL `CREATE TABLE` and
`DROP TABLE [IF EXISTS]` instead of DataFusion's own DDL planning. A table
is built with `LakeSoulTableProvider::new_from_create_external_table` and
published through `LakeSoulNamespace::register_table`, so its metadata,
primary-key partition layout and `WITH (hashbucketnum, file_format)`
properties are exactly what the LakeSoul management SQL produces, while the
location stays `{warehouse}/{database}/{table}` and cannot be chosen by the
client.
Supported: the PG column subset smallint/int/bigint/real/double/boolean/
varchar/char/text/date/timestamp/timestamptz/decimal(p,s)/bytea, one
primary key (column or table constraint), `IF NOT EXISTS`, `DROP TABLE
IF EXISTS` with several names, and extended-protocol Parse/Execute. Refused
with an explicit SQLSTATE: CREATE TABLE AS (needs the INSERT slice),
TEMPORARY/EXTERNAL/OR REPLACE, LIKE/CLONE, STORED AS/LOCATION,
UNIQUE/CHECK/FOREIGN KEY, column defaults, other types and options (0A000),
out-of-range hashbucketnum (22023), unknown schema (3F000), and any DDL
inside a transaction block, because LakeSoul metadata commits immediately
and would silently ignore ROLLBACK. Duplicate tables are 42P07; dropping an
absent table is 42P01.
tests/ddl_contract.rs proves each step through the wire protocol against
the real child process and the metadata database: table_info namespace, path
and properties, pg_class, column OIDs, duplicate/IF NOT EXISTS identity,
cascaded drop counts, the refusal matrix (nothing published, connection
usable), extended protocol, and the transaction boundary. The harness
cleanup now enumerates table_info per namespace before deleting it, so
tables a suite created through the server are reclaimed too.
Verified on PostgreSQL 17.10 in both merge configurations: seven targets
18 passed/1 ignored each, --bins 122 passed/1 ignored, clippy -D warnings
and rustfmt clean, metadata residue zero.
Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: Ic4040e641174f8e41d299ae1820b93dc6a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Address the reviewer findings on the CREATE/DROP TABLE slice. Identifiers
are folded the way PostgreSQL folds them (unquoted to lower case), and a
name that folding would change, or that would alter the data path or the
partition description (`/`, `\`, `?`, `#`, `%`, `,`, `;`, control
characters, `.`/`..`), is refused instead of producing a location outside
`{warehouse}/{database}` or a `partitions` string that decodes to different
keys. This also makes the metadata identity consistent: the provider folds
column and table names itself, so a quoted mixed-case name used to publish
`id` as the field and `;ID` as the key.
CREATE TABLE now refuses every parsed clause LakeSoul cannot honour
(DEFERRABLE and other constraint characteristics, USING/INCLUDE on the key,
INHERITS, PARTITION BY/PARTITION OF, ON COMMIT, ON CLUSTER, Hive
distribution/row format, WITHOUT ROWID, STRICT, VERSION, COMMENT, ...)
instead of silently ignoring it, rejects duplicate columns (42701),
repeated key columns (42P16), a key naming a missing column (42703),
conflicting NULL/NOT NULL declarations (42601), FLOAT(0) and non-default
timestamp precision, and reports `hashbucketnum = -1` as 22023.
DROP TABLE resolves and validates every name before the first deletion, so
a statement that PostgreSQL would reject leaves all named tables and their
metadata in place.
A simple-query message that mixes DDL with other statements is refused with
0A000 before its first statement runs: PostgreSQL would execute the batch in
an implicit transaction and roll the CREATE back when a later statement
fails, while LakeSoul table metadata is committed immediately.
Tests: the drop cascade is now counted by the table_id captured before the
DROP, pg_class queries are parameterized, and the new cases cover the
identifier rules, all-or-nothing multi-name DROP, the batch refusal,
IF NOT EXISTS keeping the existing schema, extended-protocol refusals, and
the tightened error matrix.
Verified on PostgreSQL 17.10 in both merge configurations: seven targets
18 passed/1 ignored each, --bins 122 passed/1 ignored, clippy -D warnings
and rustfmt clean, metadata residue zero.
Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: I29b3887605aea551c3ef411943edf7a36a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Add the PostgreSQL front end's INSERT hook and the write-side changes it relies on: - accept INSERT in the read-only guard; the hook resolves the target namespace from the parsed DML target, maps ON CONFLICT onto LakeSoul write modes, refuses unsupported clauses with PostgreSQL SQLSTATEs and reports INSERT 0 n. - add a per-statement conflict mode to the session and typed WriteViolation/NotNullViolation errors mapped to 23505/23502. - publish every range partition of one statement in a single metadata transaction instead of one commit per partition. - do not publish when the result stream was cancelled before the commit started; remove the attempted files instead. Follow-ups from review: - delete a statement's attempted files when metadata publication fails, but only after checking by commit id that no partition became visible: a dropped connection during COMMIT leaves the outcome unknown. - map InsertConflict/CommitConflict to SQLSTATE 40001 (retryable) instead of XX000; the loser of a write race cannot be classified as 23505 until it retries. - publish multi-partition commits in partition_desc order so concurrent statements cannot take opposite partition orders. - document that index maintenance after a successful commit still reports an error while the rows are committed and visible. - add a multi-partition INSERT case over a non-PK range table (three partitions, one statement). - rustfmt two pre-existing files in lakesoul-datafusion. Covered by tests/insert_contract.rs against a real PostgreSQL metadata database, including authoritative snapshot checks around every refusal. Verified on PostgreSQL 17.10 in both merge configurations: eight targets 20 passed/1 ignored each, --bins 123 passed/1 ignored, rustfmt clean, metadata residue zero. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Icaae5770900ea866036e8bec268b04196a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
Zero DECIMAL values lost their scale on the extended-protocol binary path: rust_decimal serializes zero with dscale 0, while PostgreSQL keeps the column scale (0.00 decodes as scale 2). Re-encode top-level Decimal128 zero cells in the query response stream using the statement's Arrow schema before they reach the client; nonzero cells stay byte-identical to the upstream encoder. Decimal256 results and negative DECIMAL scales are rejected with 0A000 instead of being truncated, and a scale beyond PostgreSQL's 14-bit dscale is 22003. The strict reproduction becomes a normal gate and now diffs 0, 0.00, 1e-20 and -0.00 against PostgreSQL; sql_compat no longer has an ignored case. Verified on PostgreSQL 17.10 in both merge configurations: sql_compat 5 passed/0 ignored, --bins 127 passed/1 ignored (the remaining ignored case is the unrelated routing timezone test), rustfmt and scoped clippy clean. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: If0bd4670e9b387ce6a529aad57fcd5626a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
PostgreSQL ignores statements in a failed transaction block with SQLSTATE 25P02 (in_failed_sql_transaction); the upstream transaction hook reports 25P01 instead. Add an in-repo guard immediately in front of that hook: it reads the same per-connection transaction status and answers first for simple queries, extended Parse and Execute of statements prepared while the block was healthy. COMMIT and ROLLBACK still pass through, so a failed block ends as PostgreSQL ends it (COMMIT is accepted and tagged ROLLBACK). The two failed-transaction contracts become differential against PostgreSQL, asserting the same 25P02 for the same SQL on both sides and covering COMMIT-in-failed-block; the state is read from the connection, never matched on error text. Verified on PostgreSQL 17.10 in both merge configurations: session_contract 3 passed/0 ignored, sql_compat 5 passed/0 ignored, --bins 127 passed/1 ignored (unrelated routing timezone case), rustfmt and scoped clippy clean. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I5b5e85e3f61ca40dc1aa882e1500ca226a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
…nding A prepared statement binds its TableProvider at Parse. A successful CREATE TABLE or DROP TABLE on the same connection now advances the connection's settings generation, so the next Execute replans: after the connection drops a table the bound scan reports 42P01 like PostgreSQL instead of the generic catalog-resolution XX000, and DROP + same-name CREATE binds the new table. DDL performed by another connection does not invalidate this connection's plans: the old binding keeps executing (zero rows for a dropped table, and a prepared INSERT into a dropped table fails with 42P01 and leaves no metadata or file residue). That is a documented LakeSoul catalog-binding boundary, not PostgreSQL equivalence. Committed data still needs no re-Prepare: the physical scan re-lists files on every Execute, as read-committed-first-select already proves. Tests: same-connection DDL replans a prepared statement, cross-connection drop and drop+recreate keep the bound plan, prepared INSERT into a dropped table leaves no state. Verified on PostgreSQL 17.10 in both merge configurations: ddl_contract 2 passed, insert_contract 2 passed, session_contract 3 passed, sql_compat 5 passed/0 ignored, --bins 127 passed/1 ignored, rustfmt and scoped clippy clean. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I22e66f0de7857d6241c64029dbee00446a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
An INSERT is analysed before planning, the way PostgreSQL analyses it. The source width must match the exported columns: too many expressions is 42601, and an explicit column list with too few expressions is 42601; a no-list source shorter than the table is completed with NULLs for the unlisted columns, in VALUES rows and in SELECT projections, which is PostgreSQL's default semantics for those columns. An untyped string literal is converted against the target column (22P02 when it does not convert), while every other expression must be assignment-castable (42804) under a matrix derived from PostgreSQL 17.10: numeric families interchange, text accepts every type, temporal types interchange, and boolean and binary stay within their own family. `SELECT` expression casts keep their documented XX000 boundary, and a wildcard projection is checked through the planned source type because positions cannot be mapped from the AST. Tests: differential cases against the reference PostgreSQL for 22P02, 42804 and 42601 in both protocols, the NULL-fill success paths for VALUES and SELECT, a mixed literal/column SELECT, and a control insert; every rejection keeps the authoritative snapshot unchanged. Verified on PostgreSQL 17.10 in both merge configurations: insert_contract 2 passed, sql_compat 5 passed/0 ignored, ddl_contract 2 passed, session_contract 3 passed, --bins 127 passed/1 ignored, rustfmt and scoped clippy clean. Signed-off-by: mag1cian <[email protected]> Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I8d05f208eef2cedbd147ac08bff8d6706a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
An `INSERT ... ON CONFLICT DO UPDATE` whose source produces the same primary key twice is now rejected with PostgreSQL's cardinality violation (21000, "cannot affect row a second time"). The statement's source is grouped by key before the write; a duplicated key fails the statement before any file is written or metadata published, so a rejected upsert leaves no state behind. The check runs at Execute in both protocols, because parameters must be bound before the source can be evaluated. `ON CONFLICT DO NOTHING` still drops the duplicate instead of failing, a plain `INSERT` still reports 23505 from the sink, and distinct keys still replace as before. Tests: duplicate source in both protocols against the reference PostgreSQL (21000 on both), authoritative snapshot unchanged, DO NOTHING accepted with one surviving row, and a distinct-key control that updates an existing row and inserts a new one. Verified on PostgreSQL 17.10 in both merge configurations: insert_contract 2 passed, sql_compat 5 passed/0 ignored, ddl_contract 2 passed, session_contract 3 passed, --bins 127 passed/1 ignored, rustfmt and scoped clippy clean. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Ibde0ea1390534716531a5990eeddcf576a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
`CREATE TABLE ... WITH (hashbucketnum = N)` stored the bucket count but the spelling was inconsistent with the other `WITH` options (`file_format`, `use_cdc`, `cdc_column`). Rename the PG-facing option to `hash_bucket_num`. - `ddl.rs`: match `hash_bucket_num` in `WITH` options and in the default injection key. The injection key must move in lockstep: leaving the old spelling as the default would overwrite a user-supplied `hash_bucket_num` with the provider default (`table_provider.rs` reads `hash_bucket_num` before `format.hash_bucket_num`). - Old spellings `hashbucketnum` and `hashBucketNum` are rejected `0A000` (no alias shim); out-of-range values remain `22023`. - Metadata property stays `hashBucketNum` (Spark/Flink persisted key); no storage migration. Contract tests: `options_and_column_types` asserts a non-default bucket count is stored; the refusal matrix and extended-protocol rejection list cover the old spellings; zero/negative/bound tests use the new spelling. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Ie093110c9c0a1774f66b159c158fd64f6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
`CREATE TABLE` without `WITH (file_format = ...)` previously defaulted to `parquet`; the B7 product decision is `vortex-compact`. The write path already supported it (`writer/mod.rs` dispatches `PhysicalFormat::VortexCompact` to `VortexSink::new(..., compact=true)` = ZSTD) and the read path auto-detects `.vortex`, so the default is a one-line injection change. - `ddl.rs`: default injection key now `vortex-compact`. - `ddl_contract::create_table_publishes_metadata`: asserts the default is `vortex-compact`; `options_and_column_types` keeps an explicit `file_format = 'parquet'` to prove the default does not override an explicit choice. - `insert_contract` wording no longer names parquet for the default-format in-test tables. - README stage-7 semantics reflect the new default. Verified: legacy and v2 merge `ddl_contract` + `insert_contract` pass; end-to-end smoke writes real `.vortex` files (metadata `vortex-compact`) with exact 96-row readback, and explicit `parquet`/`vortex`/`vortex-compact` each produce their own layout. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I956d3065f07ae6633657e43093528eb36a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
`CREATE TABLE ... WITH (append_only = true|false)` now persists the Spark/Flink `appendOnly` table property. The value is validated at the PG boundary (case-insensitive `true`/`false`, normalised to lowercase, invalid `22023`) and also accepts a bare boolean literal. Persistence goes through the typed property struct: `LakeSoulTableProperty` gains an `append_only` field (`serde(rename = "appendOnly")`), and `table_provider` maps `format.append_only`/`append_only` into it. Without both, `serde_json::to_string` would silently drop the option. The PG layer does not enforce the property yet: `UPDATE`/`DELETE` remain `0A000`, and the Rust read/write paths do not consult `appendOnly`. The README states it is persisted but not yet enforced. Verified: legacy and v2 merge `ddl_contract` pass (explicit `append_only = true`, quoted `FALSE`, invalid `yes` -> 22023, extended protocol), and `insert_contract` + bins pass under legacy. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I868037803deca51bc48764e2a39ff1856a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
`CREATE TABLE ... WITH (use_cdc = true|false, lakesoul_cdc_change_column = 'col')` now persists the two LakeSoul CDC properties. Both are live in this front end, unlike `append_only`: - `use_cdc` enables `stable_sort` on write (input order decides the newest version of a repeated key). - `lakesoul_cdc_change_column` enables the read-side tombstone filter: `col != 'delete'` is applied after merge-on-read, so a delete marker is the newest version of a key, participates in the merge, and is then dropped. A later insert revives the key. At CREATE TABLE the change column is resolved against the declared schema: it must name a declared column (`42703` otherwise), and `use_cdc = true` without an explicit column falls back to `rowKinds`, which must also be declared (the front end folds names, so the property stores `rowkinds`). The PG layer never injects a hidden column into the schema. `UPDATE`/`DELETE` statements remain `0A000`: CDC here is data-driven, the user writes the change-column value. Verified: legacy and v2 merge `ddl_contract` + `insert_contract` pass, including the tombstone/revival case. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Ia3b1f2e1977be54b64cd485bfbf8de226a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
mag1c1an1
force-pushed
the
feature/pg-ddl-insert
branch
from
November 3, 2026 14:06
e9688bc to
ec15786
Compare
Format the PostgreSQL benchmark driver, scripts, TPC-H answer generator, workload mixes, the TPC-H compatibility workflow, and the lakesoul-io manifest with the repository formatter. Python changes are AST-identical; this only satisfies the Format Check CI gate.
Exclude the unrendered console-job.yaml template from prettier: its bare ${EXTRA_ENV} placeholder is substituted with an indented env block by scripts/bench.py, so the template is not valid YAML on its own.
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
The multi-partition publish path resolves the table id up front so one transaction can publish every range partition. A statement that wrote no files (zero-row INSERT, or an empty sink) has nothing to publish, but it still resolved the table and failed with `table ... not found` when the name was not in the metadata store. Upstream main iterated the partitions and committed nothing in that case. Resolve and publish only when at least one partition has files, restoring the no-op behaviour; the commit metric still finalizes on the same path. Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
mag1c1an1
force-pushed
the
feature/pg-ddl-insert
branch
2 times, most recently
from
November 3, 2026 15:00
f100f8e to
1eb6b63
Compare
v2 merges run the merger on a spawned task, and the plan-level LakeSoulMerge consumer was dropped as soon as the stream was built, so a lane held no consumer while its merge was open: memory accounting missed the merge and the one-merge-per-lane guard saw nothing. Thread the reservation through create_merger! and have WindowSlidingMerger hold it for its task lifetime. A v2 unit can still finish and release before its first batch is read (the task buffers the unit), so the lane test now asserts the durable invariants: one merge opened, none for later units, and release after the lane is dropped (awaited, because the task is aborted asynchronously). Verified: physical_plan::merge tests 27/27 under both merge modes; lakesoul-io lib 190 passed / 1 ignored under both modes; clippy -D warnings on lakesoul-io. Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
This PR contains the 37-commit stack through jj revision
pwoykxly(e9688bc871e8e4b72da6934ee67bbfff2ab05bd8), based on upstreammain.The later CDC tombstone regression-test commit (
kkmwpyrt) and its working-copy changes are intentionally excluded.Changes
CREATE TABLE,DROP TABLE, andINSERTsupport, including merge-on-read conflict handling and PostgreSQL-compatible SQLSTATE errors.vortex-compact, rename the PG bucket option tohash_bucket_num, persistappend_only, and supportuse_cdc/lakesoul_cdc_change_columnwith schema validation.Semantics and limitations
append_onlyis persisted asappendOnlybut is not enforced by the Rust read/write paths.use_cdcenables stable sorting on writes; the configured CDC change column enables post-merge delete-tombstone filtering.UPDATEandDELETESQL statements remain unsupported (0A000); CDC is driven by inserted change-column values.Validation
cargo metadata --lockedpassed after resolving the inherited lockfile conflict.jj log -r 'conflicts()').