Skip to content

[bug][clickhouse] column name conflict after casting #417

@yokofly

Description

@yokofly

Issue Description

  • Description of the issue:
    I create a table in clickhouse
SHOW CREATE TABLE vv

Query id: 2fcd3457-fecc-4243-9dfd-93e78b4af3e8

   ┌─statement─────────────────────────────────────────────────────────────┐
1. │ CREATE TABLE default.vv
(
    `id` Int32,
    `Id` Int32
)
ENGINE = Memory │
   └───────────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.002 sec. 


and wanna use sling to sync for another table, but find the column naming conflict.

./sling run --src-conn CLICKHOUSE  --src-stream 'vv' --tgt-conn CLICKHOUSE  --tgt-object "vv2"  --mode full-refresh -d
(base) ➜  ck            export CLICKHOUSE='clickhouse://default@localhost:9000/default'      
(base) ➜  ck ./sling run --src-conn CLICKHOUSE  --src-stream 'vv' --tgt-conn CLICKHOUSE  --tgt-object "vv2"  --mode full-refresh -d
2024-10-23 07:18:21 DBG Sling version: 1.2.21 (linux amd64)
2024-10-23 07:18:21 DBG type is db-db
2024-10-23 07:18:21 DBG using: {"columns":null,"mode":"full-refresh","transforms":null}
2024-10-23 07:18:21 DBG using source options: {"empty_as_null":false,"null_if":"NULL","datetime_format":"AUTO","max_decimals":11}
2024-10-23 07:18:21 DBG using target options: {"batch_limit":100000,"datetime_format":"auto","file_max_rows":0,"max_decimals":11,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"}
2024-10-23 07:18:21 DBG opened "clickhouse" connection (conn-clickhouse-mlD)
2024-10-23 07:18:21 DBG opened "clickhouse" connection (conn-clickhouse-dFr)
2024-10-23 07:18:21 INF connecting to source database (clickhouse)
2024-10-23 07:18:21 INF connecting to target database (clickhouse)
2024-10-23 07:18:21 INF reading from source database
2024-10-23 07:18:21 DBG select * from `default`.`vv`
2024-10-23 07:18:21 INF writing to target database [mode: full-refresh]
2024-10-23 07:18:21 DBG drop table if exists `default`.`vv2_tmp`
2024-10-23 07:18:21 DBG table `default`.`vv2_tmp` dropped
2024-10-23 07:18:21 DBG create table `default`.`vv2_tmp` (`id` Nullable(Int64),
`Id` Nullable(Int64)) engine=MergeTree   ORDER BY tuple()
2024-10-23 07:18:21 INF created table `default`.`vv2_tmp`
2024-10-23 07:18:21 INF streaming data
2024-10-23 07:18:21 DBG use `default`
2024-10-23 07:18:21 DBG drop table if exists `default`.`vv2_tmp`
2024-10-23 07:18:21 DBG table `default`.`vv2_tmp` dropped
2024-10-23 07:18:21 DBG closed "clickhouse" connection (conn-clickhouse-dFr)
2024-10-23 07:18:21 DBG closed "clickhouse" connection (conn-clickhouse-mlD)
2024-10-23 07:18:21 INF execution failed
fatal:
--- proc.go:271 main ---
--- sling_cli.go:458 main ---
--- sling_cli.go:494 cliInit ---
--- cli.go:286 CliProcess ---
--- sling_run.go:225 processRun ---
~ failure running task (see docs @ https://docs.slingdata.io/sling-cli)
--- sling_run.go:396 runTask ---
--- task_run.go:155 Execute ---


--- task_run.go:116 func2 ---
--- task_run.go:559 runDbToDb ---
--- task_run_write.go:234 WriteToDb ---
--- database.go:2313 BulkImportFlow ---
~ could not bulk import
--- database.go:2300 func1 ---
~ could not copy data
--- database_clickhouse.go:261 BulkImportStream ---
~ could not prepare statement
--- database_clickhouse.go:171 func2 ---
~ could not prepare statement
--- database.go:1062 Prepare ---
~ could not prepare Tx: insert into `default`.`vv2_tmp` (`Id`, `Id`) values  ($1, $2)
--- transaction.go:95 Prepare ---
code: 15, message: Column Id in table default.vv2_tmp (44a2af7b-757a-4ec1-a543-ff3bd745ce0a) specified more than once

--- task_run.go:116 func2 ---
~ Could not WriteToDb
--- task_run.go:559 runDbToDb ---
~ could not insert into `default`.`vv2_tmp`
--- task_run_write.go:240 WriteToDb ---
(base) ➜  ck ./clickhouse client -q "show create vv"
CREATE TABLE default.vv\n(\n    `id` Int32,\n    `Id` Int32\n)\nENGINE = Memory

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions