Skip to content

Commit c593cdf

Browse files
mag1c1an1claudecodexDeepSeek V4 Flash
authored
[PG] add DDL and INSERT support with compatibility and benchmark coverage (#1249)
* [PG]: add Kubernetes benchmark harness for postgres-lakesoul Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I60005b72765058a41c7722793149e3786a6a6964 * [PG]: bench add pkmonth layout Signed-off-by: mag1cian <[email protected]> Change-Id: I173773455d5641f7bd29861de4ae0b916a6a6964 * [PG]: export query outcome and memory failure metrics at zero on startup Pre-register every outcome label of lakesoul_query_requests_total, lakesoul_query_rows_total and lakesoul_query_duration_seconds at 0 when the coordinator starts, so increase()/rate() see the first burst of an outcome instead of the series appearing at its first nonzero sample. lakesoul_memory_reservation_failures_total was already registered at pool creation; use increment(0) instead of absolute(0) so a second pool in the same process cannot reset it. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: Icb291d45a0dcadcda90f6f6f7f3ed92c6a6a6964 * [PG]: report memory failures from the series bench collects report.py looked up increase(lakesoul_memory_reservation_failures_total[15s]) while bench.py collects the [1m] window (a [15s] window holds at most one sample at a 15 s scrape interval), so the memory table always showed an empty failures column. report.py now imports the memory queries from bench.py instead of retyping them, and the column says it is the max per 1m window. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I54e4ceb6686155de1582d6fc302de2fa6a6a6964 * [Rust]: export worker discovery refresh failures, snapshot age and eligible workers The r07 discovery outage was diagnosable only from coordinator logs. The Kubernetes resolver now exports lakesoul_worker_discovery_refresh_failures_total{kind="hard"|"transient"}, lakesoul_worker_discovery_snapshot_age_seconds (since the last successful refresh, updated after every attempt, +Inf before the first) and lakesoul_worker_discovery_eligible_workers (also set by static discovery). The series are registered when the resolver starts, so the first outage shows in increase(). The benchmark collects them. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I6a99af0d7882d6c2bc2ccd09501b5bb76a6a6964 * [PG]: scrape container memory, restarts and throttling in the k8s observability overlay The processes cannot report their own OOM kills, restarts or CPU throttling, and the overlay could not show them. lakesoul-prometheus now scrapes every kubelet's cAdvisor (ClusterRole with nodes list/watch and nodes/metrics, not nodes/proxy), keeping only the LakeSoul containers' memory and CFS series, and a namespaced kube-state-metrics deployed by the overlay (--namespaces=lakesoul --resources=pods) for restart counts and last termination reasons. The LakeSoul Overview dashboard gains panels for container RSS / working set / limit vs. lakesoul_memory_pool_reserved_bytes and the pool limit, restarts and OOM kills (last_terminated_reason, since cAdvisor's OOM event series vanishes with the killed container), and CPU throttling. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: Ia3628d12ea9c8bc897592e3951c048de6a6a6964 * [PG]: run lakesoul-console TPC-H benchmarks as Guaranteed-QoS k8s Jobs lsbench gains a `console` scenario that runs the TPC-H queries through one lakesoul-console process (single node or distributed over --worker URLs) and turns its per-statement Elapsed output into the standard sample records, with row-count answer checks. bench.py console runs it in a Job whose containers have requests equal to limits, a native sampler sidecar, worker URLs from the lakesoul-worker EndpointSlices, and samples read from the container log so an OOM-killed console still reports finished queries. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: Ie8978dc91b36dc8845cbb1eb3b9c494f6a6a6964 * [Rust]: add JSON output to lakesoul-console and value-check console benchmarks lakesoul-console --format json prints, per statement, a column line with names and Arrow types, then one JSON array per row with the values in the table format and NULL as null. lsbench console uses it to compare results with answers.compare, as the tpch scenario does, instead of row counts. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: Ib9fd809244ee3e44e6c07663239376b26a6a6964 * [PG]: pass AWS_S3_VIRTUAL_HOST_STYLE from lakesoul-secrets to k8s pods Huawei OBS endpoints such as cn-north-4 reject path-style requests with VirtualHostDomainRequired, and LakeSoul defaults to path style. The coordinator, worker, datagen and console manifests now read AWS_S3_VIRTUAL_HOST_STYLE from the optional secret key aws-s3-virtual-host-style; without the key, behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I4497426534454ef038903bd286dfa1046a6a6964 * [PG]: document the regenerated TPC-H benchmark data set List the six tpch_sf{1,10}{,_pkmonth,_nopk} schemas with the datagen flags used, and record that the SF10 pkmonth and nopk loads needed lineitem rerun alone with a higher memory limit. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I78a683aede606cd31bcbb0b2bece9d706a6a6964 * [PG]: chunk nopk TPC-H loads by whole tpchgen parts The nopk sink writes one file per tpchgen part. Modulo chunks put a slice of every part in every statement, so CHUNKS=4 wrote 4x the files at 1/4 the size and changed the scan layout. For LAYOUT=nopk, chunk k is now the key range of parts [k*PARTS/N, (k+1)*PARTS/N), computed from tpchgen's part boundaries and order key mapping, so each part is still one file. PARTS must be a multiple of CHUNKS; nation/region stay single statements. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I122cbb9acd1c7c70f3e40ed427a6f7846a6a6964 * [Rust]: honor memory limit and LAKESOUL_CACHE in lakesoul-console lakesoul-console built its session factory with LakeSoulSessionFactory::new, whose pool is ExecutionMemoryArgs::default(): LAKESOUL_MEMORY_LIMIT was ignored, execution was unbounded and never spilled. The console now takes ExecutionMemoryArgs (--memory-limit, --memory-pool, --vortex-decode-budget and their env) and installs the pool and Vortex decode budget like the coordinator. Sessions register the warehouse S3 store with register_s3_object_store, which skipped the disk cache; only register_object_store honored LAKESOUL_CACHE, and it returns early once a store exists. The check now lives in register_s3_object_store, so coordinator, worker and console sessions read the warehouse through the shared cache when it is set. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: mag1cian <[email protected]> Change-Id: I463757c12ebd13d2307c0f708ad0f6866a6a6964 * [Rust]: patch DataFusion RepartitionExec spill-pool deadlock Under a memory limit, lakesoul-console hung on TPC-H q17 over tpch_sf1_pkmonth (2 CPU, LAKESOUL_MEMORY_LIMIT=1Gi, LAKESOUL_CACHE=1) in about one run in four: every task parked, no error. EXPLAIN ANALYZE showed the hash RepartitionExec spilling. This is apache/datafusion#24883: two input tasks spilling at once leave the shared spill pool with two open files, the reader waits on the drained older file while the batch it needs is in the newer one, and the writers park on the closed distributor gate. The fix (apache/datafusion#24891, merge commit 09a2aff7) is only in DataFusion 56. Vendor datafusion-physical-plan 55.2.0 under rust/third_party with that commit's src/spill/spill_pool.rs change and use it via [patch.crates-io]. With the patch the same configuration still spills in RepartitionExec and completed 20 of 20 runs; the crate's 20 spill_pool tests, including the upstream scenario fuzzer, pass. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I6f24b7436d4bcd3ee74bfd3cccaff5ae6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [Rust]: let DATAFUSION_EXECUTION_BATCH_SIZE set the session batch size create_lakesoul_session_config read the DataFusion environment with SessionConfig::from_env() and then forced .with_batch_size(8192), so DATAFUSION_EXECUTION_BATCH_SIZE was silently ignored and the batch size could only be changed per session with SET. 8192 is DataFusion's default, so dropping the override changes nothing when the variable is unset. The batch size governs memory held outside LAKESOUL_MEMORY_LIMIT: at a 1Gi budget, TPC-H SF10 q09 peaked at 2.2 GiB RSS with 8192 rows per batch and 0.76 GiB with 2048. Document the variable, and that workers do not read it (datafusion-distributed builds worker tasks from DataFusion's defaults plus the coordinator's distributed.shuffle_batch_size). Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I4c829aed5a8c706bf7cadbaab4c4fec06a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [IO]: pack MOR range partitions into one bounded merge per scan task A merge-on-read scan built one MergeParquetExec per range partition. Its parent pulls every input partition at once, so a table with 84 range partitions opened 84 merges together, each holding a non-spillable LakeSoulMerge reservation, whatever target_partitions was: pkmonth q01 exhausted a 1 GiB pool at target_partitions = 2. The scan now packs whole range partitions into one merge per task the scan stage may get (scan_task_budget: the distributed per-stage cap, or the listed workers; 1 for a local plan or the local fallback), each with at most target_partitions lanes. A lane runs its bucket work units one at a time, so a local scan, or one distributed task, holds at most target_partitions merges of the scan open. Work units keep their own files and range constants, so MOR version order, partial updates and range defaults are unchanged, and a range never spans two tasks. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I408225af368ecbe11b680ae84615d2236a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [IO]: do not sample open merges after the first batch in the early-drop test CoalescePartitionsExec producers keep polling after sending a batch, so a merge may already be closed when the consumer receives the first batch. Assert that a merge executed instead; the lakesoul-io lane test pins the release of a merge that is open mid-unit. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I8b91213ab7d12b1e0724d99273afed5b6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [Rust]: test where the session batch size comes from DataFusion default 8192 without an override, DATAFUSION_EXECUTION_BATCH_SIZE when set, a session SET over both, and LakeSoul-fixed options such as prefer_hash_join over the environment. A separate test binary with one test, so changing the process environment races nothing else. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: Ibd0fa4abe6e75932f37620b250a70b456a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: accept any order of rows with equal sort keys in lsbench answers TPC-H q11 orders by `value` alone, and its SF10 answer has two pairs of parts with equal values. Every q11 "mismatch" of the SF10 console runs swaps one of these pairs, and the ordered check stopped at the first difference, so the rest of the result was never checked. When every ORDER BY key is a bare output column, rows with equal keys now match as a multiset at their positions; other keys keep exact row order. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: Iac26b16e21c686e17b450273eb5c00616a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [Rust]: make jemalloc the console binaries' allocator; drop hard-coded profiling lakesoul-io no longer installs a global allocator. Its `jemalloc` feature installed jemalloc with heap profiling forced on via `malloc_conf` and a developer-specific `prof_prefix` path. The feature is now stats-only (`resident_memory()` for LoggedMemoryPool); the unused flamegraph and stats-print helpers and the jemalloc_pprof dependency are gone. postgres-lakesoul and lakesoul-console now own `#[global_allocator]`: jemalloc 5.3 with built-in defaults (no profiling compiled in, no decay tuning) behind a default `jemalloc` feature; `--no-default-features` gives the system allocator. `unprefixed_malloc_on_supported_platforms` routes C `malloc` through jemalloc too. Each binary logs the active allocator at startup (version, config.prof, decay options, stats.allocated). lakesoul-worker, lakesoul-io-c (JNI), lakesoul-metadata-c, Python and the libraries do not link jemalloc and keep the system allocator. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I18781042bb92889610b11c1cd17290c16a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: validate jemalloc console allocator on SF10 pk (7100c918) Interleaved q09 A/B against the glibc control image e04e8205703e on the THP=always node (2 CPU, 3 GiB, 1 GiB fair pool, batch 8192), plus the full pk suite against the 20261024 glibc baseline. jemalloc lowers peak anon RSS from 2.63 to 1.81 GiB on q09 and from 2.87 to 1.84 GiB over the suite, with identical results, spill volume and total CPU, at ~16x (q09) to ~41x (suite) more minor faults. Document the allocator, its startup log line and rollback in the k8s and benchmark READMEs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I8b2b289b138b33281aa7972c697190a96a6a6964 Co-Authored-By: ChatGPT <[email protected]> Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [Meta]: preserve secondary workspace after history cleanup Retain the existing empty workspace state on the rewritten benchmark history without changing its files. Co-Authored-By: ChatGPT <[email protected]> Change-Id: I160644a5aa0de15e35107080b1c887566a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: compatiable test stage 1 Change-Id: I1a2a0de511ae2a12d29b96f5eb856ec06a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: compatibility stage 2 Change-Id: I9a3d72a98723593953ea84e0f71c31d66a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: compatibility stage 3 and 4 Stage 3 contracts with physical proofs: - storage: append duplicates, merge superseded values, partial columns, partition-local merge, CDC tombstone removal - catalog: namespace -> database mapping, public schema, no cross-database read via search_path - vector: codec, exact fallback, ANN route and recall, probes isolation Stage 4 strict TPC-H gate: - lsbench strict driver with SF 0.1/1/10 answers and negative exits - tpch-compat workflow; PK single coordinator, nopk/pkmonth dual worker Session and protocol fixes: - connection-local SET/RESET/DISCARD TIME ZONE with startup defaults and plan invalidation on settings generation - SHOW Describe reports the column it executes: hooks claim SHOW at Parse, responses carry text (OID 25) like PostgreSQL Change-Id: Ia029cbf7b6799a40c97170ba3b90152e6a6a6964 Co-Authored-By: DeepSeek V4 Flash <[email protected]> Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: borrow pg-catalog from the fork instead of vendoring it The RemoveSubqueryFromProjection removal now lives on the datafusion-postgres fork (mag1c1an1/datafusion-postgres, lakesoul-shared-static-tables, commit 03441f8) instead of in a local copy of the crate. datafusion-pg-catalog is a path dependency inside that fork workspace, so pinning a newer rev is enough. - bump the datafusion-postgres rev to 03441f8 - drop the [patch."https://github.com/mag1c1an1/datafusion-postgres.git"] entry - drop the workspace exclude for rust/third_party/datafusion-pg-catalog - delete rust/third_party/datafusion-pg-catalog (96 files, -9k lines) Verified after the switch: --bins 121 passed / 1 ignored and the six targets 16 passed / 1 ignored in both merge configurations on PostgreSQL 17.10; clippy --bin postgres-lakesoul --tests --no-deps -- -D warnings clean; metadata, reference schemas and connections back to zero. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I0a2387bb227ba54c1feb3b75c529132c6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: add CREATE TABLE and DROP TABLE to the front end Add src/ddl.rs: a statement hook, installed per connection ahead of the read-only guard, that answers PostgreSQL `CREATE TABLE` and `DROP TABLE [IF EXISTS]` instead of DataFusion's own DDL planning. A table is built with `LakeSoulTableProvider::new_from_create_external_table` and published through `LakeSoulNamespace::register_table`, so its metadata, primary-key partition layout and `WITH (hashbucketnum, file_format)` properties are exactly what the LakeSoul management SQL produces, while the location stays `{warehouse}/{database}/{table}` and cannot be chosen by the client. Supported: the PG column subset smallint/int/bigint/real/double/boolean/ varchar/char/text/date/timestamp/timestamptz/decimal(p,s)/bytea, one primary key (column or table constraint), `IF NOT EXISTS`, `DROP TABLE IF EXISTS` with several names, and extended-protocol Parse/Execute. Refused with an explicit SQLSTATE: CREATE TABLE AS (needs the INSERT slice), TEMPORARY/EXTERNAL/OR REPLACE, LIKE/CLONE, STORED AS/LOCATION, UNIQUE/CHECK/FOREIGN KEY, column defaults, other types and options (0A000), out-of-range hashbucketnum (22023), unknown schema (3F000), and any DDL inside a transaction block, because LakeSoul metadata commits immediately and would silently ignore ROLLBACK. Duplicate tables are 42P07; dropping an absent table is 42P01. tests/ddl_contract.rs proves each step through the wire protocol against the real child process and the metadata database: table_info namespace, path and properties, pg_class, column OIDs, duplicate/IF NOT EXISTS identity, cascaded drop counts, the refusal matrix (nothing published, connection usable), extended protocol, and the transaction boundary. The harness cleanup now enumerates table_info per namespace before deleting it, so tables a suite created through the server are reclaimed too. Verified on PostgreSQL 17.10 in both merge configurations: seven targets 18 passed/1 ignored each, --bins 122 passed/1 ignored, clippy -D warnings and rustfmt clean, metadata residue zero. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Ic4040e641174f8e41d299ae1820b93dc6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: tighten the DDL contract after review Address the reviewer findings on the CREATE/DROP TABLE slice. Identifiers are folded the way PostgreSQL folds them (unquoted to lower case), and a name that folding would change, or that would alter the data path or the partition description (`/`, `\`, `?`, `#`, `%`, `,`, `;`, control characters, `.`/`..`), is refused instead of producing a location outside `{warehouse}/{database}` or a `partitions` string that decodes to different keys. This also makes the metadata identity consistent: the provider folds column and table names itself, so a quoted mixed-case name used to publish `id` as the field and `;ID` as the key. CREATE TABLE now refuses every parsed clause LakeSoul cannot honour (DEFERRABLE and other constraint characteristics, USING/INCLUDE on the key, INHERITS, PARTITION BY/PARTITION OF, ON COMMIT, ON CLUSTER, Hive distribution/row format, WITHOUT ROWID, STRICT, VERSION, COMMENT, ...) instead of silently ignoring it, rejects duplicate columns (42701), repeated key columns (42P16), a key naming a missing column (42703), conflicting NULL/NOT NULL declarations (42601), FLOAT(0) and non-default timestamp precision, and reports `hashbucketnum = -1` as 22023. DROP TABLE resolves and validates every name before the first deletion, so a statement that PostgreSQL would reject leaves all named tables and their metadata in place. A simple-query message that mixes DDL with other statements is refused with 0A000 before its first statement runs: PostgreSQL would execute the batch in an implicit transaction and roll the CREATE back when a later statement fails, while LakeSoul table metadata is committed immediately. Tests: the drop cascade is now counted by the table_id captured before the DROP, pg_class queries are parameterized, and the new cases cover the identifier rules, all-or-nothing multi-name DROP, the batch refusal, IF NOT EXISTS keeping the existing schema, extended-protocol refusals, and the tightened error matrix. Verified on PostgreSQL 17.10 in both merge configurations: seven targets 18 passed/1 ignored each, --bins 122 passed/1 ignored, clippy -D warnings and rustfmt clean, metadata residue zero. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I29b3887605aea551c3ef411943edf7a36a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: support INSERT with merge-on-read conflict semantics Add the PostgreSQL front end's INSERT hook and the write-side changes it relies on: - accept INSERT in the read-only guard; the hook resolves the target namespace from the parsed DML target, maps ON CONFLICT onto LakeSoul write modes, refuses unsupported clauses with PostgreSQL SQLSTATEs and reports INSERT 0 n. - add a per-statement conflict mode to the session and typed WriteViolation/NotNullViolation errors mapped to 23505/23502. - publish every range partition of one statement in a single metadata transaction instead of one commit per partition. - do not publish when the result stream was cancelled before the commit started; remove the attempted files instead. Follow-ups from review: - delete a statement's attempted files when metadata publication fails, but only after checking by commit id that no partition became visible: a dropped connection during COMMIT leaves the outcome unknown. - map InsertConflict/CommitConflict to SQLSTATE 40001 (retryable) instead of XX000; the loser of a write race cannot be classified as 23505 until it retries. - publish multi-partition commits in partition_desc order so concurrent statements cannot take opposite partition orders. - document that index maintenance after a successful commit still reports an error while the rows are committed and visible. - add a multi-partition INSERT case over a non-PK range table (three partitions, one statement). - rustfmt two pre-existing files in lakesoul-datafusion. Covered by tests/insert_contract.rs against a real PostgreSQL metadata database, including authoritative snapshot checks around every refusal. Verified on PostgreSQL 17.10 in both merge configurations: eight targets 20 passed/1 ignored each, --bins 123 passed/1 ignored, rustfmt clean, metadata residue zero. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Icaae5770900ea866036e8bec268b04196a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * fix(pg): encode zero NUMERIC with the column scale Zero DECIMAL values lost their scale on the extended-protocol binary path: rust_decimal serializes zero with dscale 0, while PostgreSQL keeps the column scale (0.00 decodes as scale 2). Re-encode top-level Decimal128 zero cells in the query response stream using the statement's Arrow schema before they reach the client; nonzero cells stay byte-identical to the upstream encoder. Decimal256 results and negative DECIMAL scales are rejected with 0A000 instead of being truncated, and a scale beyond PostgreSQL's 14-bit dscale is 22003. The strict reproduction becomes a normal gate and now diffs 0, 0.00, 1e-20 and -0.00 against PostgreSQL; sql_compat no longer has an ignored case. Verified on PostgreSQL 17.10 in both merge configurations: sql_compat 5 passed/0 ignored, --bins 127 passed/1 ignored (the remaining ignored case is the unrelated routing timezone test), rustfmt and scoped clippy clean. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: If0bd4670e9b387ce6a529aad57fcd5626a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * fix(pg): report 25P02 for statements in a failed transaction PostgreSQL ignores statements in a failed transaction block with SQLSTATE 25P02 (in_failed_sql_transaction); the upstream transaction hook reports 25P01 instead. Add an in-repo guard immediately in front of that hook: it reads the same per-connection transaction status and answers first for simple queries, extended Parse and Execute of statements prepared while the block was healthy. COMMIT and ROLLBACK still pass through, so a failed block ends as PostgreSQL ends it (COMMIT is accepted and tagged ROLLBACK). The two failed-transaction contracts become differential against PostgreSQL, asserting the same 25P02 for the same SQL on both sides and covering COMMIT-in-failed-block; the state is read from the connection, never matched on error text. Verified on PostgreSQL 17.10 in both merge configurations: session_contract 3 passed/0 ignored, sql_compat 5 passed/0 ignored, --bins 127 passed/1 ignored (unrelated routing timezone case), rustfmt and scoped clippy clean. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I5b5e85e3f61ca40dc1aa882e1500ca226a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * fix(pg): invalidate prepared plans after own DDL; document catalog binding A prepared statement binds its TableProvider at Parse. A successful CREATE TABLE or DROP TABLE on the same connection now advances the connection's settings generation, so the next Execute replans: after the connection drops a table the bound scan reports 42P01 like PostgreSQL instead of the generic catalog-resolution XX000, and DROP + same-name CREATE binds the new table. DDL performed by another connection does not invalidate this connection's plans: the old binding keeps executing (zero rows for a dropped table, and a prepared INSERT into a dropped table fails with 42P01 and leaves no metadata or file residue). That is a documented LakeSoul catalog-binding boundary, not PostgreSQL equivalence. Committed data still needs no re-Prepare: the physical scan re-lists files on every Execute, as read-committed-first-select already proves. Tests: same-connection DDL replans a prepared statement, cross-connection drop and drop+recreate keep the bound plan, prepared INSERT into a dropped table leaves no state. Verified on PostgreSQL 17.10 in both merge configurations: ddl_contract 2 passed, insert_contract 2 passed, session_contract 3 passed, sql_compat 5 passed/0 ignored, --bins 127 passed/1 ignored, rustfmt and scoped clippy clean. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I22e66f0de7857d6241c64029dbee00446a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * fix(pg): analyze INSERT like PostgreSQL for 42601/42804/22P02 An INSERT is analysed before planning, the way PostgreSQL analyses it. The source width must match the exported columns: too many expressions is 42601, and an explicit column list with too few expressions is 42601; a no-list source shorter than the table is completed with NULLs for the unlisted columns, in VALUES rows and in SELECT projections, which is PostgreSQL's default semantics for those columns. An untyped string literal is converted against the target column (22P02 when it does not convert), while every other expression must be assignment-castable (42804) under a matrix derived from PostgreSQL 17.10: numeric families interchange, text accepts every type, temporal types interchange, and boolean and binary stay within their own family. `SELECT` expression casts keep their documented XX000 boundary, and a wildcard projection is checked through the planned source type because positions cannot be mapped from the AST. Tests: differential cases against the reference PostgreSQL for 22P02, 42804 and 42601 in both protocols, the NULL-fill success paths for VALUES and SELECT, a mixed literal/column SELECT, and a control insert; every rejection keeps the authoritative snapshot unchanged. Verified on PostgreSQL 17.10 in both merge configurations: insert_contract 2 passed, sql_compat 5 passed/0 ignored, ddl_contract 2 passed, session_contract 3 passed, --bins 127 passed/1 ignored, rustfmt and scoped clippy clean. Signed-off-by: mag1cian <[email protected]> Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I8d05f208eef2cedbd147ac08bff8d6706a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * fix(pg): reject duplicate upsert keys with 21000 An `INSERT ... ON CONFLICT DO UPDATE` whose source produces the same primary key twice is now rejected with PostgreSQL's cardinality violation (21000, "cannot affect row a second time"). The statement's source is grouped by key before the write; a duplicated key fails the statement before any file is written or metadata published, so a rejected upsert leaves no state behind. The check runs at Execute in both protocols, because parameters must be bound before the source can be evaluated. `ON CONFLICT DO NOTHING` still drops the duplicate instead of failing, a plain `INSERT` still reports 23505 from the sink, and distinct keys still replace as before. Tests: duplicate source in both protocols against the reference PostgreSQL (21000 on both), authoritative snapshot unchanged, DO NOTHING accepted with one surviving row, and a distinct-key control that updates an existing row and inserts a new one. Verified on PostgreSQL 17.10 in both merge configurations: insert_contract 2 passed, sql_compat 5 passed/0 ignored, ddl_contract 2 passed, session_contract 3 passed, --bins 127 passed/1 ignored, rustfmt and scoped clippy clean. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Ibde0ea1390534716531a5990eeddcf576a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: rename hash_bucket_num table option `CREATE TABLE ... WITH (hashbucketnum = N)` stored the bucket count but the spelling was inconsistent with the other `WITH` options (`file_format`, `use_cdc`, `cdc_column`). Rename the PG-facing option to `hash_bucket_num`. - `ddl.rs`: match `hash_bucket_num` in `WITH` options and in the default injection key. The injection key must move in lockstep: leaving the old spelling as the default would overwrite a user-supplied `hash_bucket_num` with the provider default (`table_provider.rs` reads `hash_bucket_num` before `format.hash_bucket_num`). - Old spellings `hashbucketnum` and `hashBucketNum` are rejected `0A000` (no alias shim); out-of-range values remain `22023`. - Metadata property stays `hashBucketNum` (Spark/Flink persisted key); no storage migration. Contract tests: `options_and_column_types` asserts a non-default bucket count is stored; the refusal matrix and extended-protocol rejection list cover the old spellings; zero/negative/bound tests use the new spelling. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Ie093110c9c0a1774f66b159c158fd64f6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: default file_format to vortex-compact `CREATE TABLE` without `WITH (file_format = ...)` previously defaulted to `parquet`; the B7 product decision is `vortex-compact`. The write path already supported it (`writer/mod.rs` dispatches `PhysicalFormat::VortexCompact` to `VortexSink::new(..., compact=true)` = ZSTD) and the read path auto-detects `.vortex`, so the default is a one-line injection change. - `ddl.rs`: default injection key now `vortex-compact`. - `ddl_contract::create_table_publishes_metadata`: asserts the default is `vortex-compact`; `options_and_column_types` keeps an explicit `file_format = 'parquet'` to prove the default does not override an explicit choice. - `insert_contract` wording no longer names parquet for the default-format in-test tables. - README stage-7 semantics reflect the new default. Verified: legacy and v2 merge `ddl_contract` + `insert_contract` pass; end-to-end smoke writes real `.vortex` files (metadata `vortex-compact`) with exact 96-row readback, and explicit `parquet`/`vortex`/`vortex-compact` each produce their own layout. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I956d3065f07ae6633657e43093528eb36a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: accept append_only table option `CREATE TABLE ... WITH (append_only = true|false)` now persists the Spark/Flink `appendOnly` table property. The value is validated at the PG boundary (case-insensitive `true`/`false`, normalised to lowercase, invalid `22023`) and also accepts a bare boolean literal. Persistence goes through the typed property struct: `LakeSoulTableProperty` gains an `append_only` field (`serde(rename = "appendOnly")`), and `table_provider` maps `format.append_only`/`append_only` into it. Without both, `serde_json::to_string` would silently drop the option. The PG layer does not enforce the property yet: `UPDATE`/`DELETE` remain `0A000`, and the Rust read/write paths do not consult `appendOnly`. The README states it is persisted but not yet enforced. Verified: legacy and v2 merge `ddl_contract` pass (explicit `append_only = true`, quoted `FALSE`, invalid `yes` -> 22023, extended protocol), and `insert_contract` + bins pass under legacy. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: I868037803deca51bc48764e2a39ff1856a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * [PG]: accept use_cdc and lakesoul_cdc_change_column options `CREATE TABLE ... WITH (use_cdc = true|false, lakesoul_cdc_change_column = 'col')` now persists the two LakeSoul CDC properties. Both are live in this front end, unlike `append_only`: - `use_cdc` enables `stable_sort` on write (input order decides the newest version of a repeated key). - `lakesoul_cdc_change_column` enables the read-side tombstone filter: `col != 'delete'` is applied after merge-on-read, so a delete marker is the newest version of a key, participates in the merge, and is then dropped. A later insert revives the key. At CREATE TABLE the change column is resolved against the declared schema: it must name a declared column (`42703` otherwise), and `use_cdc = true` without an explicit column falls back to `rowKinds`, which must also be declared (the front end folds names, so the property stores `rowkinds`). The PG layer never injects a hidden column into the schema. `UPDATE`/`DELETE` statements remain `0A000`: CDC here is data-driven, the user writes the change-column value. Verified: legacy and v2 merge `ddl_contract` + `insert_contract` pass, including the tombstone/revival case. Co-Authored-By: DeepSeek V4 Flash <[email protected]> Change-Id: Ia3b1f2e1977be54b64cd485bfbf8de226a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * benchmark: apply treefmt formatting Format the PostgreSQL benchmark driver, scripts, TPC-H answer generator, workload mixes, the TPC-H compatibility workflow, and the lakesoul-io manifest with the repository formatter. Python changes are AST-identical; this only satisfies the Format Check CI gate. Exclude the unrendered console-job.yaml template from prettier: its bare ${EXTRA_ENV} placeholder is substituted with an indented env block by scripts/bench.py, so the template is not valid YAML on its own. Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * fix: skip metadata publication when no files were written The multi-partition publish path resolves the table id up front so one transaction can publish every range partition. A statement that wrote no files (zero-row INSERT, or an empty sink) has nothing to publish, but it still resolved the table and failed with `table ... not found` when the name was not in the metadata store. Upstream main iterated the partitions and committed nothing in that case. Resolve and publish only when at least one partition has files, restoring the no-op behaviour; the commit metric still finalizes on the same path. Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> * fix: keep the merge memory consumer for v2 merges v2 merges run the merger on a spawned task, and the plan-level LakeSoulMerge consumer was dropped as soon as the stream was built, so a lane held no consumer while its merge was open: memory accounting missed the merge and the one-merge-per-lane guard saw nothing. Thread the reservation through create_merger! and have WindowSlidingMerger hold it for its task lifetime. A v2 unit can still finish and release before its first batch is read (the task buffers the unit), so the lane test now asserts the durable invariants: one merge opened, none for later units, and release after the lane is dropped (awaited, because the task is aborted asynchronously). Verified: physical_plan::merge tests 27/27 under both merge modes; lakesoul-io lib 190 passed / 1 ignored under both modes; clippy -D warnings on lakesoul-io. Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]> --------- Signed-off-by: mag1cian <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: DeepSeek V4 Flash <[email protected]>
1 parent 3fdef59 commit c593cdf

235 files changed

Lines changed: 93243 additions & 541 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/rust-ci.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,11 @@ jobs:
104104
echo "RustFS Configuration Complete!"
105105
'
106106
- uses: actions/checkout@v4
107+
- name: Strict TPC-H driver boundary regressions
108+
env:
109+
PYTHONPATH: rust/postgres-lakesoul/benchmark/driver
110+
PYTHONDONTWRITEBYTECODE: "1"
111+
run: python3 -m unittest discover -s rust/postgres-lakesoul/benchmark/driver/tests -v
107112
- name: Install Rust toolchain
108113
uses: dtolnay/rust-toolchain@master
109114
with:
@@ -140,13 +145,31 @@ jobs:
140145
RUST_BACKTRACE: full
141146
run: |
142147
cargo test --profile test-fast --lib --bins --tests --jobs 4
148+
- name: PostgreSQL SQL compatibility (legacy merge)
149+
env:
150+
LAKESOUL_IO_USE_V2_MERGE: "false"
151+
LAKESOUL_PG_URL: "jdbc:postgresql://127.0.0.1:5432/lakesoul_test?stringtype=unspecified"
152+
LAKESOUL_PG_USERNAME: lakesoul_test
153+
LAKESOUL_PG_PASSWORD: lakesoul_test
154+
TZ: UTC
155+
RUST_BACKTRACE: full
156+
run: cargo -q test --profile test-fast -p postgres-lakesoul --test sql_compat --test session_contract --test storage_contract --test catalog_contract --test vector_contract --test tpch_contract --jobs 4 -- --nocapture
143157
- name: Test v2 merge
144158
env:
145159
LAKESOUL_IO_USE_V2_MERGE: "true"
146160
RUST_LOG: info
147161
RUST_BACKTRACE: full
148162
run: |
149163
cargo test --profile test-fast --lib --bins --tests --jobs 4
164+
- name: PostgreSQL SQL compatibility (v2 merge)
165+
env:
166+
LAKESOUL_IO_USE_V2_MERGE: "true"
167+
LAKESOUL_PG_URL: "jdbc:postgresql://127.0.0.1:5432/lakesoul_test?stringtype=unspecified"
168+
LAKESOUL_PG_USERNAME: lakesoul_test
169+
LAKESOUL_PG_PASSWORD: lakesoul_test
170+
TZ: UTC
171+
RUST_BACKTRACE: full
172+
run: cargo -q test --profile test-fast -p postgres-lakesoul --test sql_compat --test session_contract --test storage_contract --test catalog_contract --test vector_contract --test tpch_contract --jobs 4 -- --nocapture
150173

151174
clippy_check:
152175
runs-on: ubuntu-24.04

‎.github/workflows/tpch-compat.yml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# SPDX-FileCopyrightText: 2026 LakeSoul Contributors
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
name: TPC-H SQL regression
6+
7+
on:
8+
workflow_dispatch:
9+
inputs:
10+
scale:
11+
description: "Checked-in tpchgen-rs answer set"
12+
type: choice
13+
options: ["0.1", "1", "10"]
14+
default: "1"
15+
schedule:
16+
- cron: "41 2 * * 0"
17+
18+
permissions:
19+
contents: read
20+
21+
concurrency:
22+
group: ${{ github.workflow }}-${{ github.ref }}
23+
cancel-in-progress: false
24+
25+
defaults:
26+
run:
27+
shell: bash
28+
29+
jobs:
30+
workload:
31+
runs-on: ubuntu-24.04
32+
timeout-minutes: 90
33+
strategy:
34+
fail-fast: false
35+
matrix:
36+
merge: ["false", "true"]
37+
shape:
38+
- { layout: pk, workers: "0" }
39+
- { layout: nopk, workers: "2" }
40+
- { layout: pkmonth, workers: "2" }
41+
env:
42+
RUSTFLAGS: "-Awarnings -C link-arg=-fuse-ld=lld"
43+
RUST_BACKTRACE: full
44+
LAKESOUL_PG_URL: "jdbc:postgresql://127.0.0.1:5432/lakesoul_test?stringtype=unspecified"
45+
LAKESOUL_PG_USERNAME: lakesoul_test
46+
LAKESOUL_PG_PASSWORD: lakesoul_test
47+
LAKESOUL_IO_USE_V2_MERGE: ${{ matrix.merge }}
48+
LAKESOUL_TPCH_SCALE: ${{ inputs.scale || '1' }}
49+
LAKESOUL_TPCH_LAYOUT: ${{ matrix.shape.layout }}
50+
LAKESOUL_TPCH_WORKERS: ${{ matrix.shape.workers }}
51+
TZ: UTC
52+
PYTHONDONTWRITEBYTECODE: "1"
53+
services:
54+
postgres:
55+
image: postgres:14.5
56+
env:
57+
POSTGRES_PASSWORD: lakesoul_test
58+
POSTGRES_USER: lakesoul_test
59+
POSTGRES_DB: lakesoul_test
60+
options: >-
61+
--health-cmd pg_isready
62+
--health-interval 10s
63+
--health-timeout 5s
64+
--health-retries 5
65+
ports:
66+
- 5432:5432
67+
steps:
68+
- uses: actions/checkout@v4
69+
- uses: dtolnay/rust-toolchain@master
70+
with:
71+
toolchain: stable
72+
- uses: ./.github/actions/setup-flatc
73+
- uses: Swatinem/rust-cache@v2
74+
with:
75+
workspaces: ". -> rust/target"
76+
shared-key: rust-test-fast
77+
save-if: ${{ github.ref == 'refs/heads/main' }}
78+
- name: Install native requirements
79+
run: |
80+
sudo apt-get update
81+
sudo apt-get install -y lld postgresql-client-16
82+
command -v ld.lld
83+
- uses: arduino/setup-protoc@v2
84+
with:
85+
version: "23.x"
86+
repo-token: ${{ secrets.GITHUB_TOKEN }}
87+
- name: Initialize metadata
88+
run: ./script/meta_init_for_local_test.sh -j 1
89+
- name: Strict q01-q22 over both wire protocols
90+
run: cargo -q test --profile test-fast -p postgres-lakesoul --test tpch_contract --jobs 4 -- --nocapture

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,3 +109,5 @@ agent_export/
109109
/rust/lakesoul-datafusion/lakesoul_cache_dir/
110110

111111
**/result
112+
113+

0 commit comments

Comments
 (0)