-
-
Notifications
You must be signed in to change notification settings - Fork 64
[bug][clickhouse] column name conflict after casting #417
Copy link
Copy link
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Issue Description
- Description of the issue:
I create a table in clickhouse
SHOW CREATE TABLE vv
Query id: 2fcd3457-fecc-4243-9dfd-93e78b4af3e8
┌─statement─────────────────────────────────────────────────────────────┐
1. │ CREATE TABLE default.vv
(
`id` Int32,
`Id` Int32
)
ENGINE = Memory │
└───────────────────────────────────────────────────────────────────────┘
1 row in set. Elapsed: 0.002 sec.
and wanna use sling to sync for another table, but find the column naming conflict.
./sling run --src-conn CLICKHOUSE --src-stream 'vv' --tgt-conn CLICKHOUSE --tgt-object "vv2" --mode full-refresh -d
-
Sling version (
sling --version):
wget https://github.com/slingdata-io/sling-cli/releases/download/v1.2.21/sling_linux_amd64.tar.gz -
Operating System (
linux,mac,windows):
linux
(base) ➜ ck export CLICKHOUSE='clickhouse://default@localhost:9000/default'
(base) ➜ ck ./sling run --src-conn CLICKHOUSE --src-stream 'vv' --tgt-conn CLICKHOUSE --tgt-object "vv2" --mode full-refresh -d
2024-10-23 07:18:21 DBG Sling version: 1.2.21 (linux amd64)
2024-10-23 07:18:21 DBG type is db-db
2024-10-23 07:18:21 DBG using: {"columns":null,"mode":"full-refresh","transforms":null}
2024-10-23 07:18:21 DBG using source options: {"empty_as_null":false,"null_if":"NULL","datetime_format":"AUTO","max_decimals":11}
2024-10-23 07:18:21 DBG using target options: {"batch_limit":100000,"datetime_format":"auto","file_max_rows":0,"max_decimals":11,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"}
2024-10-23 07:18:21 DBG opened "clickhouse" connection (conn-clickhouse-mlD)
2024-10-23 07:18:21 DBG opened "clickhouse" connection (conn-clickhouse-dFr)
2024-10-23 07:18:21 INF connecting to source database (clickhouse)
2024-10-23 07:18:21 INF connecting to target database (clickhouse)
2024-10-23 07:18:21 INF reading from source database
2024-10-23 07:18:21 DBG select * from `default`.`vv`
2024-10-23 07:18:21 INF writing to target database [mode: full-refresh]
2024-10-23 07:18:21 DBG drop table if exists `default`.`vv2_tmp`
2024-10-23 07:18:21 DBG table `default`.`vv2_tmp` dropped
2024-10-23 07:18:21 DBG create table `default`.`vv2_tmp` (`id` Nullable(Int64),
`Id` Nullable(Int64)) engine=MergeTree ORDER BY tuple()
2024-10-23 07:18:21 INF created table `default`.`vv2_tmp`
2024-10-23 07:18:21 INF streaming data
2024-10-23 07:18:21 DBG use `default`
2024-10-23 07:18:21 DBG drop table if exists `default`.`vv2_tmp`
2024-10-23 07:18:21 DBG table `default`.`vv2_tmp` dropped
2024-10-23 07:18:21 DBG closed "clickhouse" connection (conn-clickhouse-dFr)
2024-10-23 07:18:21 DBG closed "clickhouse" connection (conn-clickhouse-mlD)
2024-10-23 07:18:21 INF execution failed
fatal:
--- proc.go:271 main ---
--- sling_cli.go:458 main ---
--- sling_cli.go:494 cliInit ---
--- cli.go:286 CliProcess ---
--- sling_run.go:225 processRun ---
~ failure running task (see docs @ https://docs.slingdata.io/sling-cli)
--- sling_run.go:396 runTask ---
--- task_run.go:155 Execute ---
--- task_run.go:116 func2 ---
--- task_run.go:559 runDbToDb ---
--- task_run_write.go:234 WriteToDb ---
--- database.go:2313 BulkImportFlow ---
~ could not bulk import
--- database.go:2300 func1 ---
~ could not copy data
--- database_clickhouse.go:261 BulkImportStream ---
~ could not prepare statement
--- database_clickhouse.go:171 func2 ---
~ could not prepare statement
--- database.go:1062 Prepare ---
~ could not prepare Tx: insert into `default`.`vv2_tmp` (`Id`, `Id`) values ($1, $2)
--- transaction.go:95 Prepare ---
code: 15, message: Column Id in table default.vv2_tmp (44a2af7b-757a-4ec1-a543-ff3bd745ce0a) specified more than once
--- task_run.go:116 func2 ---
~ Could not WriteToDb
--- task_run.go:559 runDbToDb ---
~ could not insert into `default`.`vv2_tmp`
--- task_run_write.go:240 WriteToDb ---
(base) ➜ ck ./clickhouse client -q "show create vv"
CREATE TABLE default.vv\n(\n `id` Int32,\n `Id` Int32\n)\nENGINE = Memory
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request