Skip to content

[PG] add DDL and INSERT support with compatibility and benchmark coverage - #1249

Merged
xuchen-plus merged 40 commits into
lakesoul-io:mainfrom
mag1c1an1:feature/pg-ddl-insert
Nov 4, 2026
Merged

xuchen-plus merged 40 commits into
lakesoul-io:mainfrom
mag1c1an1:feature/pg-ddl-insert

Conversation

@mag1c1an1

Copy link
Copy Markdown
Contributor

Scope

This PR contains the 37-commit stack through jj revision pwoykxly (e9688bc871e8e4b72da6934ee67bbfff2ab05bd8), based on upstream main.

The later CDC tombstone regression-test commit (kkmwpyrt) and its working-copy changes are intentionally excluded.

Changes

  • Add PostgreSQL-front-end CREATE TABLE, DROP TABLE, and INSERT support, including merge-on-read conflict handling and PostgreSQL-compatible SQLSTATE errors.
  • Add compatibility fixtures, contract coverage, and CI; fix NUMERIC zero encoding, failed-transaction errors, and prepared-plan invalidation after local DDL.
  • Default new PG tables to vortex-compact, rename the PG bucket option to hash_bucket_num, persist append_only, and support use_cdc / lakesoul_cdc_change_column with schema validation.
  • Add Kubernetes TPC-H benchmark tooling, console JSON output, result comparison, and observability updates.
  • Include supporting Rust changes for bounded merge scheduling, session batch size, memory/cache configuration, a DataFusion repartition spill-pool deadlock fix, and executable-owned jemalloc configuration.

Semantics and limitations

  • append_only is persisted as appendOnly but is not enforced by the Rust read/write paths.
  • use_cdc enables stable sorting on writes; the configured CDC change column enables post-merge delete-tombstone filtering.
  • UPDATE and DELETE SQL statements remain unsupported (0A000); CDC is driven by inserted change-column values.

Validation

  • cargo metadata --locked passed after resolving the inherited lockfile conflict.
  • No conflicted revisions remain (jj log -r 'conflicts()').
  • Existing commit descriptions record the contract, compatibility, and benchmark verification performed during implementation. Test suites were not rerun for PR creation.

mag1c1an1 and others added 30 commits November 3, 2026 18:02
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I60005b72765058a41c7722793149e3786a6a6964
Signed-off-by: mag1cian <[email protected]>
Change-Id: I173773455d5641f7bd29861de4ae0b916a6a6964
Pre-register every outcome label of lakesoul_query_requests_total,
lakesoul_query_rows_total and lakesoul_query_duration_seconds at 0 when the
coordinator starts, so increase()/rate() see the first burst of an outcome
instead of the series appearing at its first nonzero sample.
lakesoul_memory_reservation_failures_total was already registered at pool
creation; use increment(0) instead of absolute(0) so a second pool in the
same process cannot reset it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: Icb291d45a0dcadcda90f6f6f7f3ed92c6a6a6964
report.py looked up increase(lakesoul_memory_reservation_failures_total[15s])
while bench.py collects the [1m] window (a [15s] window holds at most one
sample at a 15 s scrape interval), so the memory table always showed an empty
failures column. report.py now imports the memory queries from bench.py
instead of retyping them, and the column says it is the max per 1m window.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I54e4ceb6686155de1582d6fc302de2fa6a6a6964
…igible workers

The r07 discovery outage was diagnosable only from coordinator logs. The
Kubernetes resolver now exports
lakesoul_worker_discovery_refresh_failures_total{kind="hard"|"transient"},
lakesoul_worker_discovery_snapshot_age_seconds (since the last successful
refresh, updated after every attempt, +Inf before the first) and
lakesoul_worker_discovery_eligible_workers (also set by static discovery).
The series are registered when the resolver starts, so the first outage shows
in increase(). The benchmark collects them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I6a99af0d7882d6c2bc2ccd09501b5bb76a6a6964
…ervability overlay

The processes cannot report their own OOM kills, restarts or CPU throttling,
and the overlay could not show them. lakesoul-prometheus now scrapes every
kubelet's cAdvisor (ClusterRole with nodes list/watch and nodes/metrics,
not nodes/proxy), keeping only the LakeSoul containers' memory and CFS
series, and a namespaced kube-state-metrics deployed by the overlay
(--namespaces=lakesoul --resources=pods) for restart counts and last
termination reasons.

The LakeSoul Overview dashboard gains panels for container RSS / working
set / limit vs. lakesoul_memory_pool_reserved_bytes and the pool limit,
restarts and OOM kills (last_terminated_reason, since cAdvisor's OOM event
series vanishes with the killed container), and CPU throttling.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: Ia3628d12ea9c8bc897592e3951c048de6a6a6964
lsbench gains a `console` scenario that runs the TPC-H queries through one
lakesoul-console process (single node or distributed over --worker URLs) and
turns its per-statement Elapsed output into the standard sample records,
with row-count answer checks. bench.py console runs it in a Job whose
containers have requests equal to limits, a native sampler sidecar, worker
URLs from the lakesoul-worker EndpointSlices, and samples read from the
container log so an OOM-killed console still reports finished queries.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: Ie8978dc91b36dc8845cbb1eb3b9c494f6a6a6964
…enchmarks

lakesoul-console --format json prints, per statement, a column line with
names and Arrow types, then one JSON array per row with the values in the
table format and NULL as null. lsbench console uses it to compare results
with answers.compare, as the tpch scenario does, instead of row counts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: Ib9fd809244ee3e44e6c07663239376b26a6a6964
Huawei OBS endpoints such as cn-north-4 reject path-style requests with
VirtualHostDomainRequired, and LakeSoul defaults to path style. The
coordinator, worker, datagen and console manifests now read
AWS_S3_VIRTUAL_HOST_STYLE from the optional secret key
aws-s3-virtual-host-style; without the key, behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I4497426534454ef038903bd286dfa1046a6a6964
List the six tpch_sf{1,10}{,_pkmonth,_nopk} schemas with the datagen flags
used, and record that the SF10 pkmonth and nopk loads needed lineitem
rerun alone with a higher memory limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I78a683aede606cd31bcbb0b2bece9d706a6a6964
The nopk sink writes one file per tpchgen part. Modulo chunks put a slice
of every part in every statement, so CHUNKS=4 wrote 4x the files at 1/4
the size and changed the scan layout. For LAYOUT=nopk, chunk k is now the
key range of parts [k*PARTS/N, (k+1)*PARTS/N), computed from tpchgen's
part boundaries and order key mapping, so each part is still one file.
PARTS must be a multiple of CHUNKS; nation/region stay single statements.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I122cbb9acd1c7c70f3e40ed427a6f7846a6a6964
lakesoul-console built its session factory with LakeSoulSessionFactory::new,
whose pool is ExecutionMemoryArgs::default(): LAKESOUL_MEMORY_LIMIT was
ignored, execution was unbounded and never spilled. The console now takes
ExecutionMemoryArgs (--memory-limit, --memory-pool, --vortex-decode-budget
and their env) and installs the pool and Vortex decode budget like the
coordinator.

Sessions register the warehouse S3 store with register_s3_object_store,
which skipped the disk cache; only register_object_store honored
LAKESOUL_CACHE, and it returns early once a store exists. The check now
lives in register_s3_object_store, so coordinator, worker and console
sessions read the warehouse through the shared cache when it is set.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I463757c12ebd13d2307c0f708ad0f6866a6a6964
Under a memory limit, lakesoul-console hung on TPC-H q17 over
tpch_sf1_pkmonth (2 CPU, LAKESOUL_MEMORY_LIMIT=1Gi, LAKESOUL_CACHE=1) in
about one run in four: every task parked, no error. EXPLAIN ANALYZE showed
the hash RepartitionExec spilling. This is apache/datafusion#24883: two
input tasks spilling at once leave the shared spill pool with two open
files, the reader waits on the drained older file while the batch it needs
is in the newer one, and the writers park on the closed distributor gate.

The fix (apache/datafusion#24891, merge commit 09a2aff7) is only in
DataFusion 56. Vendor datafusion-physical-plan 55.2.0 under rust/third_party
with that commit's src/spill/spill_pool.rs change and use it via
[patch.crates-io]. With the patch the same configuration still spills in
RepartitionExec and completed 20 of 20 runs; the crate's 20 spill_pool
tests, including the upstream scenario fuzzer, pass.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: I6f24b7436d4bcd3ee74bfd3cccaff5ae6a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
create_lakesoul_session_config read the DataFusion environment with
SessionConfig::from_env() and then forced .with_batch_size(8192), so
DATAFUSION_EXECUTION_BATCH_SIZE was silently ignored and the batch size
could only be changed per session with SET. 8192 is DataFusion's default,
so dropping the override changes nothing when the variable is unset.

The batch size governs memory held outside LAKESOUL_MEMORY_LIMIT: at a 1Gi
budget, TPC-H SF10 q09 peaked at 2.2 GiB RSS with 8192 rows per batch and
0.76 GiB with 2048. Document the variable, and that workers do not read it
(datafusion-distributed builds worker tasks from DataFusion's defaults plus
the coordinator's distributed.shuffle_batch_size).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: I4c829aed5a8c706bf7cadbaab4c4fec06a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
A merge-on-read scan built one MergeParquetExec per range partition. Its
parent pulls every input partition at once, so a table with 84 range
partitions opened 84 merges together, each holding a non-spillable
LakeSoulMerge reservation, whatever target_partitions was: pkmonth q01
exhausted a 1 GiB pool at target_partitions = 2.

The scan now packs whole range partitions into one merge per task the
scan stage may get (scan_task_budget: the distributed per-stage cap, or
the listed workers; 1 for a local plan or the local fallback), each with
at most target_partitions lanes. A lane runs its bucket work units one at
a time, so a local scan, or one distributed task, holds at most
target_partitions merges of the scan open. Work units keep their own
files and range constants, so MOR version order, partial updates and
range defaults are unchanged, and a range never spans two tasks.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: I408225af368ecbe11b680ae84615d2236a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
…op test

CoalescePartitionsExec producers keep polling after sending a batch, so a
merge may already be closed when the consumer receives the first batch.
Assert that a merge executed instead; the lakesoul-io lane test pins the
release of a merge that is open mid-unit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: I8b91213ab7d12b1e0724d99273afed5b6a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
DataFusion default 8192 without an override, DATAFUSION_EXECUTION_BATCH_SIZE
when set, a session SET over both, and LakeSoul-fixed options such as
prefer_hash_join over the environment. A separate test binary with one
test, so changing the process environment races nothing else.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: Ibd0fa4abe6e75932f37620b250a70b456a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
TPC-H q11 orders by `value` alone, and its SF10 answer has two pairs of
parts with equal values. Every q11 "mismatch" of the SF10 console runs
swaps one of these pairs, and the ordered check stopped at the first
difference, so the rest of the result was never checked. When every
ORDER BY key is a bare output column, rows with equal keys now match as
a multiset at their positions; other keys keep exact row order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: Iac26b16e21c686e17b450273eb5c00616a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
…d profiling

lakesoul-io no longer installs a global allocator. Its `jemalloc` feature
installed jemalloc with heap profiling forced on via `malloc_conf` and a
developer-specific `prof_prefix` path. The feature is now stats-only
(`resident_memory()` for LoggedMemoryPool); the unused flamegraph and
stats-print helpers and the jemalloc_pprof dependency are gone.

postgres-lakesoul and lakesoul-console now own `#[global_allocator]`:
jemalloc 5.3 with built-in defaults (no profiling compiled in, no decay
tuning) behind a default `jemalloc` feature; `--no-default-features` gives
the system allocator. `unprefixed_malloc_on_supported_platforms` routes C
`malloc` through jemalloc too. Each binary logs the active allocator at
startup (version, config.prof, decay options, stats.allocated).

lakesoul-worker, lakesoul-io-c (JNI), lakesoul-metadata-c, Python and the
libraries do not link jemalloc and keep the system allocator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: I18781042bb92889610b11c1cd17290c16a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Interleaved q09 A/B against the glibc control image e04e8205703e on the
THP=always node (2 CPU, 3 GiB, 1 GiB fair pool, batch 8192), plus the full
pk suite against the 20261024 glibc baseline. jemalloc lowers peak anon RSS
from 2.63 to 1.81 GiB on q09 and from 2.87 to 1.84 GiB over the suite, with
identical results, spill volume and total CPU, at ~16x (q09) to ~41x (suite)
more minor faults. Document the allocator, its startup log line and rollback
in the k8s and benchmark READMEs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: I8b2b289b138b33281aa7972c697190a96a6a6964
Co-Authored-By: ChatGPT <[email protected]>
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Retain the existing empty workspace state on the rewritten benchmark
history without changing its files.

Co-Authored-By: ChatGPT <[email protected]>

Change-Id: I160644a5aa0de15e35107080b1c887566a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I1a2a0de511ae2a12d29b96f5eb856ec06a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Change-Id: I9a3d72a98723593953ea84e0f71c31d66a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Stage 3 contracts with physical proofs:
- storage: append duplicates, merge superseded values, partial columns,
  partition-local merge, CDC tombstone removal
- catalog: namespace -> database mapping, public schema, no cross-database
  read via search_path
- vector: codec, exact fallback, ANN route and recall, probes isolation

Stage 4 strict TPC-H gate:
- lsbench strict driver with SF 0.1/1/10 answers and negative exits
- tpch-compat workflow; PK single coordinator, nopk/pkmonth dual worker

Session and protocol fixes:
- connection-local SET/RESET/DISCARD TIME ZONE with startup defaults and
  plan invalidation on settings generation
- SHOW Describe reports the column it executes: hooks claim SHOW at Parse,
  responses carry text (OID 25) like PostgreSQL

Change-Id: Ia029cbf7b6799a40c97170ba3b90152e6a6a6964
Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
The RemoveSubqueryFromProjection removal now lives on the datafusion-postgres fork (mag1c1an1/datafusion-postgres, lakesoul-shared-static-tables, commit 03441f8) instead of in a local copy of the crate. datafusion-pg-catalog is a path dependency inside that fork workspace, so pinning a newer rev is enough.

- bump the datafusion-postgres rev to 03441f8
- drop the [patch."https://github.com/mag1c1an1/datafusion-postgres.git"] entry
- drop the workspace exclude for rust/third_party/datafusion-pg-catalog
- delete rust/third_party/datafusion-pg-catalog (96 files, -9k lines)

Verified after the switch: --bins 121 passed / 1 ignored and the six targets 16 passed / 1 ignored in both merge configurations on PostgreSQL 17.10; clippy --bin postgres-lakesoul --tests --no-deps -- -D warnings clean; metadata, reference schemas and connections back to zero.

Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: I0a2387bb227ba54c1feb3b75c529132c6a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Add src/ddl.rs: a statement hook, installed per connection ahead of the
read-only guard, that answers PostgreSQL `CREATE TABLE` and
`DROP TABLE [IF EXISTS]` instead of DataFusion's own DDL planning. A table
is built with `LakeSoulTableProvider::new_from_create_external_table` and
published through `LakeSoulNamespace::register_table`, so its metadata,
primary-key partition layout and `WITH (hashbucketnum, file_format)`
properties are exactly what the LakeSoul management SQL produces, while the
location stays `{warehouse}/{database}/{table}` and cannot be chosen by the
client.

Supported: the PG column subset smallint/int/bigint/real/double/boolean/
varchar/char/text/date/timestamp/timestamptz/decimal(p,s)/bytea, one
primary key (column or table constraint), `IF NOT EXISTS`, `DROP TABLE
IF EXISTS` with several names, and extended-protocol Parse/Execute. Refused
with an explicit SQLSTATE: CREATE TABLE AS (needs the INSERT slice),
TEMPORARY/EXTERNAL/OR REPLACE, LIKE/CLONE, STORED AS/LOCATION,
UNIQUE/CHECK/FOREIGN KEY, column defaults, other types and options (0A000),
out-of-range hashbucketnum (22023), unknown schema (3F000), and any DDL
inside a transaction block, because LakeSoul metadata commits immediately
and would silently ignore ROLLBACK. Duplicate tables are 42P07; dropping an
absent table is 42P01.

tests/ddl_contract.rs proves each step through the wire protocol against
the real child process and the metadata database: table_info namespace, path
and properties, pg_class, column OIDs, duplicate/IF NOT EXISTS identity,
cascaded drop counts, the refusal matrix (nothing published, connection
usable), extended protocol, and the transaction boundary. The harness
cleanup now enumerates table_info per namespace before deleting it, so
tables a suite created through the server are reclaimed too.

Verified on PostgreSQL 17.10 in both merge configurations: seven targets
18 passed/1 ignored each, --bins 122 passed/1 ignored, clippy -D warnings
and rustfmt clean, metadata residue zero.

Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: Ic4040e641174f8e41d299ae1820b93dc6a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Address the reviewer findings on the CREATE/DROP TABLE slice. Identifiers
are folded the way PostgreSQL folds them (unquoted to lower case), and a
name that folding would change, or that would alter the data path or the
partition description (`/`, `\`, `?`, `#`, `%`, `,`, `;`, control
characters, `.`/`..`), is refused instead of producing a location outside
`{warehouse}/{database}` or a `partitions` string that decodes to different
keys. This also makes the metadata identity consistent: the provider folds
column and table names itself, so a quoted mixed-case name used to publish
`id` as the field and `;ID` as the key.

CREATE TABLE now refuses every parsed clause LakeSoul cannot honour
(DEFERRABLE and other constraint characteristics, USING/INCLUDE on the key,
INHERITS, PARTITION BY/PARTITION OF, ON COMMIT, ON CLUSTER, Hive
distribution/row format, WITHOUT ROWID, STRICT, VERSION, COMMENT, ...)
instead of silently ignoring it, rejects duplicate columns (42701),
repeated key columns (42P16), a key naming a missing column (42703),
conflicting NULL/NOT NULL declarations (42601), FLOAT(0) and non-default
timestamp precision, and reports `hashbucketnum = -1` as 22023.

DROP TABLE resolves and validates every name before the first deletion, so
a statement that PostgreSQL would reject leaves all named tables and their
metadata in place.

A simple-query message that mixes DDL with other statements is refused with
0A000 before its first statement runs: PostgreSQL would execute the batch in
an implicit transaction and roll the CREATE back when a later statement
fails, while LakeSoul table metadata is committed immediately.

Tests: the drop cascade is now counted by the table_id captured before the
DROP, pg_class queries are parameterized, and the new cases cover the
identifier rules, all-or-nothing multi-name DROP, the batch refusal,
IF NOT EXISTS keeping the existing schema, extended-protocol refusals, and
the tightened error matrix.

Verified on PostgreSQL 17.10 in both merge configurations: seven targets
18 passed/1 ignored each, --bins 122 passed/1 ignored, clippy -D warnings
and rustfmt clean, metadata residue zero.

Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: I29b3887605aea551c3ef411943edf7a36a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Add the PostgreSQL front end's INSERT hook and the write-side changes it
relies on:
- accept INSERT in the read-only guard; the hook resolves the target
  namespace from the parsed DML target, maps ON CONFLICT onto LakeSoul
  write modes, refuses unsupported clauses with PostgreSQL SQLSTATEs and
  reports INSERT 0 n.
- add a per-statement conflict mode to the session and typed
  WriteViolation/NotNullViolation errors mapped to 23505/23502.
- publish every range partition of one statement in a single metadata
  transaction instead of one commit per partition.
- do not publish when the result stream was cancelled before the commit
  started; remove the attempted files instead.

Follow-ups from review:
- delete a statement's attempted files when metadata publication fails, but
  only after checking by commit id that no partition became visible: a
  dropped connection during COMMIT leaves the outcome unknown.
- map InsertConflict/CommitConflict to SQLSTATE 40001 (retryable) instead of
  XX000; the loser of a write race cannot be classified as 23505 until it
  retries.
- publish multi-partition commits in partition_desc order so concurrent
  statements cannot take opposite partition orders.
- document that index maintenance after a successful commit still reports an
  error while the rows are committed and visible.
- add a multi-partition INSERT case over a non-PK range table (three
  partitions, one statement).
- rustfmt two pre-existing files in lakesoul-datafusion.

Covered by tests/insert_contract.rs against a real PostgreSQL metadata
database, including authoritative snapshot checks around every refusal.
Verified on PostgreSQL 17.10 in both merge configurations: eight targets
20 passed/1 ignored each, --bins 123 passed/1 ignored, rustfmt clean,
metadata residue zero.
Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: Icaae5770900ea866036e8bec268b04196a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
Zero DECIMAL values lost their scale on the extended-protocol binary path:
rust_decimal serializes zero with dscale 0, while PostgreSQL keeps the
column scale (0.00 decodes as scale 2). Re-encode top-level Decimal128
zero cells in the query response stream using the statement's Arrow schema
before they reach the client; nonzero cells stay byte-identical to the
upstream encoder.

Decimal256 results and negative DECIMAL scales are rejected with 0A000
instead of being truncated, and a scale beyond PostgreSQL's 14-bit dscale
is 22003.

The strict reproduction becomes a normal gate and now diffs 0, 0.00, 1e-20
and -0.00 against PostgreSQL; sql_compat no longer has an ignored case.

Verified on PostgreSQL 17.10 in both merge configurations: sql_compat
5 passed/0 ignored, --bins 127 passed/1 ignored (the remaining ignored case
is the unrelated routing timezone test), rustfmt and scoped clippy clean.
Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: If0bd4670e9b387ce6a529aad57fcd5626a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
PostgreSQL ignores statements in a failed transaction block with SQLSTATE
25P02 (in_failed_sql_transaction); the upstream transaction hook reports
25P01 instead. Add an in-repo guard immediately in front of that hook: it
reads the same per-connection transaction status and answers first for
simple queries, extended Parse and Execute of statements prepared while
the block was healthy. COMMIT and ROLLBACK still pass through, so a failed
block ends as PostgreSQL ends it (COMMIT is accepted and tagged ROLLBACK).

The two failed-transaction contracts become differential against
PostgreSQL, asserting the same 25P02 for the same SQL on both sides and
covering COMMIT-in-failed-block; the state is read from the connection,
never matched on error text.

Verified on PostgreSQL 17.10 in both merge configurations:
session_contract 3 passed/0 ignored, sql_compat 5 passed/0 ignored,
--bins 127 passed/1 ignored (unrelated routing timezone case), rustfmt
and scoped clippy clean.
Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: I5b5e85e3f61ca40dc1aa882e1500ca226a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
mag1c1an1 and others added 7 commits November 3, 2026 14:02
…nding

A prepared statement binds its TableProvider at Parse. A successful
CREATE TABLE or DROP TABLE on the same connection now advances the
connection's settings generation, so the next Execute replans: after the
connection drops a table the bound scan reports 42P01 like PostgreSQL
instead of the generic catalog-resolution XX000, and DROP + same-name
CREATE binds the new table.

DDL performed by another connection does not invalidate this connection's
plans: the old binding keeps executing (zero rows for a dropped table, and
a prepared INSERT into a dropped table fails with 42P01 and leaves no
metadata or file residue). That is a documented LakeSoul catalog-binding
boundary, not PostgreSQL equivalence. Committed data still needs no
re-Prepare: the physical scan re-lists files on every Execute, as
read-committed-first-select already proves.

Tests: same-connection DDL replans a prepared statement, cross-connection
drop and drop+recreate keep the bound plan, prepared INSERT into a dropped
table leaves no state.
Verified on PostgreSQL 17.10 in both merge configurations: ddl_contract
2 passed, insert_contract 2 passed, session_contract 3 passed, sql_compat
5 passed/0 ignored, --bins 127 passed/1 ignored, rustfmt and scoped clippy
clean.
Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: I22e66f0de7857d6241c64029dbee00446a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
An INSERT is analysed before planning, the way PostgreSQL analyses it.
The source width must match the exported columns: too many expressions is
42601, and an explicit column list with too few expressions is 42601; a
no-list source shorter than the table is completed with NULLs for the
unlisted columns, in VALUES rows and in SELECT projections, which is
PostgreSQL's default semantics for those columns. An untyped string
literal is converted against the target column (22P02 when it does not
convert), while every other expression must be assignment-castable
(42804) under a matrix derived from PostgreSQL 17.10: numeric families
interchange, text accepts every type, temporal types interchange, and
boolean and binary stay within their own family.
`SELECT` expression casts keep their documented XX000 boundary, and a
wildcard projection is checked through the planned source type because
positions cannot be mapped from the AST.
Tests: differential cases against the reference PostgreSQL for 22P02,
42804 and 42601 in both protocols, the NULL-fill success paths for
VALUES and SELECT, a mixed literal/column SELECT, and a control insert;
every rejection keeps the authoritative snapshot unchanged.
Verified on PostgreSQL 17.10 in both merge configurations:
insert_contract 2 passed, sql_compat 5 passed/0 ignored, ddl_contract
2 passed, session_contract 3 passed, --bins 127 passed/1 ignored,
rustfmt and scoped clippy clean.
Signed-off-by: mag1cian <[email protected]>
Co-Authored-By: DeepSeek V4 Flash <[email protected]>

Change-Id: I8d05f208eef2cedbd147ac08bff8d6706a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
An `INSERT ... ON CONFLICT DO UPDATE` whose source produces the same
primary key twice is now rejected with PostgreSQL's cardinality
violation (21000, "cannot affect row a second time"). The statement's
source is grouped by key before the write; a duplicated key fails the
statement before any file is written or metadata published, so a
rejected upsert leaves no state behind. The check runs at Execute in
both protocols, because parameters must be bound before the source can
be evaluated.

`ON CONFLICT DO NOTHING` still drops the duplicate instead of failing,
a plain `INSERT` still reports 23505 from the sink, and distinct keys
still replace as before.

Tests: duplicate source in both protocols against the reference
PostgreSQL (21000 on both), authoritative snapshot unchanged, DO NOTHING
accepted with one surviving row, and a distinct-key control that updates
an existing row and inserts a new one.
Verified on PostgreSQL 17.10 in both merge configurations:
insert_contract 2 passed, sql_compat 5 passed/0 ignored, ddl_contract
2 passed, session_contract 3 passed, --bins 127 passed/1 ignored,
rustfmt and scoped clippy clean.
Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: Ibde0ea1390534716531a5990eeddcf576a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
`CREATE TABLE ... WITH (hashbucketnum = N)` stored the bucket count but the
spelling was inconsistent with the other `WITH` options (`file_format`,
`use_cdc`, `cdc_column`). Rename the PG-facing option to `hash_bucket_num`.

- `ddl.rs`: match `hash_bucket_num` in `WITH` options and in the default
  injection key. The injection key must move in lockstep: leaving the old
  spelling as the default would overwrite a user-supplied `hash_bucket_num`
  with the provider default (`table_provider.rs` reads `hash_bucket_num`
  before `format.hash_bucket_num`).
- Old spellings `hashbucketnum` and `hashBucketNum` are rejected `0A000`
  (no alias shim); out-of-range values remain `22023`.
- Metadata property stays `hashBucketNum` (Spark/Flink persisted key); no
  storage migration.

Contract tests: `options_and_column_types` asserts a non-default bucket
count is stored; the refusal matrix and extended-protocol rejection list
cover the old spellings; zero/negative/bound tests use the new spelling.

Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: Ie093110c9c0a1774f66b159c158fd64f6a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
`CREATE TABLE` without `WITH (file_format = ...)` previously defaulted to
`parquet`; the B7 product decision is `vortex-compact`. The write path already
supported it (`writer/mod.rs` dispatches `PhysicalFormat::VortexCompact` to
`VortexSink::new(..., compact=true)` = ZSTD) and the read path auto-detects
`.vortex`, so the default is a one-line injection change.

- `ddl.rs`: default injection key now `vortex-compact`.
- `ddl_contract::create_table_publishes_metadata`: asserts the default is
  `vortex-compact`; `options_and_column_types` keeps an explicit
  `file_format = 'parquet'` to prove the default does not override an explicit
  choice.
- `insert_contract` wording no longer names parquet for the default-format
  in-test tables.
- README stage-7 semantics reflect the new default.

Verified: legacy and v2 merge `ddl_contract` + `insert_contract` pass;
end-to-end smoke writes real `.vortex` files (metadata `vortex-compact`) with
exact 96-row readback, and explicit `parquet`/`vortex`/`vortex-compact` each
produce their own layout.

Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: I956d3065f07ae6633657e43093528eb36a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
`CREATE TABLE ... WITH (append_only = true|false)` now persists the
Spark/Flink `appendOnly` table property. The value is validated at the PG
boundary (case-insensitive `true`/`false`, normalised to lowercase, invalid
`22023`) and also accepts a bare boolean literal.

Persistence goes through the typed property struct: `LakeSoulTableProperty`
gains an `append_only` field (`serde(rename = "appendOnly")`), and
`table_provider` maps `format.append_only`/`append_only` into it. Without
both, `serde_json::to_string` would silently drop the option.

The PG layer does not enforce the property yet: `UPDATE`/`DELETE` remain
`0A000`, and the Rust read/write paths do not consult `appendOnly`. The
README states it is persisted but not yet enforced.

Verified: legacy and v2 merge `ddl_contract` pass (explicit `append_only =
true`, quoted `FALSE`, invalid `yes` -> 22023, extended protocol), and
`insert_contract` + bins pass under legacy.

Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: I868037803deca51bc48764e2a39ff1856a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
`CREATE TABLE ... WITH (use_cdc = true|false, lakesoul_cdc_change_column =
'col')` now persists the two LakeSoul CDC properties. Both are live in
this front end, unlike `append_only`:

- `use_cdc` enables `stable_sort` on write (input order decides the newest
  version of a repeated key).
- `lakesoul_cdc_change_column` enables the read-side tombstone filter:
  `col != 'delete'` is applied after merge-on-read, so a delete marker is
  the newest version of a key, participates in the merge, and is then
  dropped. A later insert revives the key.

At CREATE TABLE the change column is resolved against the declared schema:
it must name a declared column (`42703` otherwise), and `use_cdc = true`
without an explicit column falls back to `rowKinds`, which must also be
declared (the front end folds names, so the property stores `rowkinds`).
The PG layer never injects a hidden column into the schema.

`UPDATE`/`DELETE` statements remain `0A000`: CDC here is data-driven, the
user writes the change-column value.

Verified: legacy and v2 merge `ddl_contract` + `insert_contract` pass,
including the tombstone/revival case.

Co-Authored-By: DeepSeek V4 Flash <[email protected]>
Change-Id: Ia3b1f2e1977be54b64cd485bfbf8de226a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
@mag1c1an1
mag1c1an1 force-pushed the feature/pg-ddl-insert branch from e9688bc to ec15786 Compare November 3, 2026 14:06
mag1c1an1 and others added 2 commits November 3, 2026 14:25
Format the PostgreSQL benchmark driver, scripts, TPC-H answer generator, workload mixes, the TPC-H compatibility workflow, and the lakesoul-io manifest with the repository formatter. Python changes are AST-identical; this only satisfies the Format Check CI gate.

Exclude the unrendered console-job.yaml template from prettier: its bare ${EXTRA_ENV} placeholder is substituted with an indented env block by scripts/bench.py, so the template is not valid YAML on its own.

Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
The multi-partition publish path resolves the table id up front so one
transaction can publish every range partition. A statement that wrote no
files (zero-row INSERT, or an empty sink) has nothing to publish, but it
still resolved the table and failed with `table ... not found` when the
name was not in the metadata store. Upstream main iterated the partitions
and committed nothing in that case.

Resolve and publish only when at least one partition has files, restoring
the no-op behaviour; the commit metric still finalizes on the same path.

Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
@mag1c1an1
mag1c1an1 force-pushed the feature/pg-ddl-insert branch 2 times, most recently from f100f8e to 1eb6b63 Compare November 3, 2026 15:00
v2 merges run the merger on a spawned task, and the plan-level
LakeSoulMerge consumer was dropped as soon as the stream was built, so a
lane held no consumer while its merge was open: memory accounting missed
the merge and the one-merge-per-lane guard saw nothing.

Thread the reservation through create_merger! and have
WindowSlidingMerger hold it for its task lifetime. A v2 unit can still
finish and release before its first batch is read (the task buffers the
unit), so the lane test now asserts the durable invariants: one merge
opened, none for later units, and release after the lane is dropped
(awaited, because the task is aborted asynchronously).

Verified: physical_plan::merge tests 27/27 under both merge modes; lakesoul-io lib 190 passed / 1 ignored under both modes; clippy -D warnings on lakesoul-io.

Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>
@xuchen-plus
xuchen-plus merged commit c593cdf into lakesoul-io:main Nov 4, 2026
24 checks passed
@mag1c1an1
mag1c1an1 deleted the feature/pg-ddl-insert branch November 4, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants