Skip to content

bug: Error copying table structure from Postgres -> DuckDB #12440

New issue
New issue
Closed
#12518
Closed
bug: Error copying table structure from Postgres -> DuckDB#12440
#12518
Assignees
erezrokah
@nicosuave

Description

@nicosuave
nicosuave
opened on Jul 21, 2023

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

On first run from this particular app's database, I encounter

terminate called after throwing an instance of 'duckdb::CatalogException'                                                                                                                                               │
 what():  Catalog Error: Column with name created_at already exists!

Have tried updating to the most recent versions of each plugin (Postgres source, DuckDB destination) without any luck.

Expected Behavior

For the structure to be replicated in the destination

CloudQuery (redacted) config

kind: source
spec:
  name: abc
  registry: "github"
  path: "cloudquery/postgresql"
  version: "v2.0.2"
  destinations: ["xyz"]
  tables: ["*"]
  spec:
    connection_string: ...
---
kind: destination
spec:
  name: xyz
  path: cloudquery/duckdb
  version: "v4.1.2"
  write_mode: overwrite
  spec:
    connection_string: ...

Steps To Reproduce

No response

CloudQuery (redacted) logs

... Loading spec(s) from ....yml
... Downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-source-postgresql-v2.0.2/postgresql_linux_amd64.zip
... 
Downloading   0% |                                      | ( 0 B/7.5 MB) [0s:0s]
                                                                               

Downloading  15% |████                          | (1.2/7.5 MB, 18 MB/s) [0s:0s]
                                                                               

Downloading 100% |██████████████████████████████| (7.5/7.5 MB, 75 MB/s)
... Downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-destination-duckdb-v4.1.2/duckdb_linux_amd64.zip
... 
Downloading   0% |                                       | ( 0 B/17 MB) [0s:0s]
                                                                               

Downloading  73% |██████████████████████         | (13/17 MB, 193 MB/s) [0s:0s]
                                                                               

Downloading 100% |███████████████████████████████| (17/17 MB, 200 MB/s)
... Starting sync for: ... ([email protected]) -> [motherduck ([email protected])]
... terminate called after throwing an instance of 'duckdb::CatalogException'
...   what():  Catalog Error: Column with name id already exists!
... SIGABRT: abort
... PC=0x7fd51196accc m=7 sigcode=18446744073709551610
... signal arrived during cgo execution
... 
... goroutine 25 [syscall]:
... runtime.cgocall(0x165dcb0, 0xc0000ef3d8)
...     /__t/go/1.19.10/x64/src/runtime/cgocall.go:158 +0x5c fp=0xc0000ef3b0 sp=0xc0000ef378 pc=0x8ca49c
... github.com/marcboeker/go-duckdb._Cfunc_duckdb_extract_statements(0x4405340, 0x7fd4d02a2fb0, 0xc000015060)
...     _cgo_gotypes.go:1171 +0x4d fp=0xc0000ef3d8 sp=0xc0000ef3b0 pc=0xe220ed
... github.com/marcboeker/go-duckdb.(*conn).extractStmts.func2(0x7fd4d02a2fb0?, 0x24a?, 0x8?)
...     /go/pkg/mod/github.com/marcboeker/[email protected]/connection.go:172 +0x87 fp=0xc0000ef420 sp=0xc0000ef3d8 pc=0xe24f67
... github.com/marcboeker/go-duckdb.(*conn).extractStmts(0x0?, {0xc0006d2000?, 0xe24349?})
...     /go/pkg/mod/github.com/marcboeker/[email protected]/connection.go:172 +0x92 fp=0xc0000ef4a0 sp=0xc0000ef420 pc=0xe24c12
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000067fe8 sp=0xc000067fe0 pc=0x92d341
... created by runtime.gcenable
...     /__t/go/1.19.10/x64/src/runtime/mgc.go:179 +0xaa
... 
... goroutine 18 [finalizer wait]:
... runtime.gopark(0x37e7ec0?, 0xc0001061a0?, 0x0?, 0x0?, 0xc000066770?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000066628 sp=0xc000066608 pc=0x8ff0b6
... runtime.goparkunlock(...)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:369
... runtime.runfinq()
...     /__t/go/1.19.10/x64/src/runtime/mfinal.go:180 +0x10f fp=0xc0000667e0 sp=0xc000066628 pc=0x8df34f
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x92d341
... created by runtime.createfing
...     /__t/go/1.19.10/x64/src/runtime/mfinal.go:157 +0x45
... 
... goroutine 19 [GC worker (idle)]:
... runtime.gopark(0xd7f977a067db?, 0x0?, 0x0?, 0x0?, 0x0?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000062750 sp=0xc000062730 pc=0x8ff0b6
... runtime.gcBgMarkWorker()
...     /__t/go/1.19.10/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000627e0 sp=0xc000062750 pc=0x8e2391
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000627e8 sp=0xc0000627e0 pc=0x92d341
... created by runtime.gcBgMarkStartWorkers
...     /__t/go/1.19.10/x64/src/runtime/mgc.go:1159 +0x25
... 
... goroutine 20 [GC worker (idle)]:
... runtime.gopark(0xd7fa2cfd4e84?, 0x0?, 0x0?, 0x0?, 0x0?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000062f50 sp=0xc000062f30 pc=0x8ff0b6
... runtime.gcBgMarkWorker()
...     /__t/go/1.19.10/x64/src/runtime/mgc.go:1235 +0xf1 fp=0xc000062fe0 sp=0xc000062f50 pc=0x8e2391
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000062fe8 sp=0xc000062fe0 pc=0x92d341
... created by runtime.gcBgMarkStartWorkers
...     /__t/go/1.19.10/x64/src/runtime/mgc.go:1159 +0x25
... 
... goroutine 21 [select, locked to thread]:
... runtime.gopark(0xc0000637a8?, 0x2?, 0x2e?, 0xf6?, 0xc0000637a4?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000063618 sp=0xc0000635f8 pc=0x8ff0b6
... runtime.selectgo(0xc0000637a8, 0xc0000637a0, 0x0?, 0x0, 0x0?, 0x1)
...     /__t/go/1.19.10/x64/src/runtime/select.go:328 +0x7bc fp=0xc000063758 sp=0xc000063618 pc=0x90e8bc
... runtime.ensureSigM.func1()
...     /__t/go/1.19.10/x64/src/runtime/signal_unix.go:995 +0x1b0 fp=0xc0000637e0 sp=0xc000063758 pc=0x912b30
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000637e8 sp=0xc0000637e0 pc=0x92d341
... created by runtime.ensureSigM
...     /__t/go/1.19.10/x64/src/runtime/signal_unix.go:978 +0xbd
... 
... goroutine 5 [syscall]:
... runtime.notetsleepg(0x0?, 0x0?)
...     /__t/go/1.19.10/x64/src/runtime/lock_futex.go:236 +0x34 fp=0xc0000687a0 sp=0xc000068768 pc=0x8d2894
... os/signal.signal_recv()
...     /__t/go/1.19.10/x64/src/runtime/sigqueue.go:152 +0x2f fp=0xc0000687c0 sp=0xc0000687a0 pc=0x929daf
... os/signal.loop()
...     /__t/go/1.19.10/x64/src/os/signal/signal_unix.go:23 +0x19 fp=0xc0000687e0 sp=0xc0000687c0 pc=0x1556839
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000687e8 sp=0xc0000687e0 pc=0x92d341
... created by os/signal.Notify.func1.1
...     /__t/go/1.19.10/x64/src/os/signal/signal.go:151 +0x2a
... 
... goroutine 6 [select]:
... runtime.gopark(0xc000068fb0?, 0x2?, 0x0?, 0x0?, 0xc000068f5c?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000068dd8 sp=0xc000068db8 pc=0x8ff0b6
... runtime.selectgo(0xc000068fb0, 0xc000068f58, 0x0?, 0x0, 0x0?, 0x1)
...     /__t/go/1.19.10/x64/src/runtime/select.go:328 +0x7bc fp=0xc000068f18 sp=0xc000068dd8 pc=0x90e8bc
... github.com/cloudquery/plugin-sdk/v4/serve.(*PluginServe).newCmdPluginServe.func1.3()
...     /go/pkg/mod/github.com/cloudquery/plugin-sdk/[email protected]/serve/plugin.go:222 +0xc6 fp=0xc000068fe0 sp=0xc000068f18 pc=0x165cfe6
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x92d341
... created by github.com/cloudquery/plugin-sdk/v4/serve.(*PluginServe).newCmdPluginServe.func1
...     /go/pkg/mod/github.com/cloudquery/plugin-sdk/[email protected]/serve/plugin.go:221 +0x1085
... 
... goroutine 8 [select]:
... runtime.gopark(0xc0004dff00?, 0x2?, 0x65?, 0x75?, 0xc0004dfed4?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc0004dfd60 sp=0xc0004dfd40 pc=0x8ff0b6
... runtime.selectgo(0xc0004dff00, 0xc0004dfed0, 0x7070e0910100402?, 0x0, 0xc0003fe870?, 0x1)
...     /__t/go/1.19.10/x64/src/runtime/select.go:328 +0x7bc fp=0xc0004dfea0 sp=0xc0004dfd60 pc=0x90e8bc
... google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0003fe690, 0x1)
...     /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418 +0x115 fp=0xc0004dff30 sp=0xc0004dfea0 pc=0x149b455
... google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0004d20e0)
...     /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552 +0x91 fp=0xc0004dff90 sp=0xc0004dff30 pc=0x149bbd1
... google.golang.org/grpc/internal/transport.NewServerTransport.func2()
...     /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:341 +0xda fp=0xc0004dffe0 sp=0xc0004dff90 pc=0x14b3f3a
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0004dffe8 sp=0xc0004dffe0 pc=0x92d341
... created by google.golang.org/grpc/internal/transport.NewServerTransport
...     /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:338 +0x1ad3
... 
... goroutine 9 [select]:
... runtime.gopark(0xc000063f70?, 0x4?, 0x0?, 0x30?, 0xc000063ec0?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc000079d08 sp=0xc000079ce8 pc=0x8ff0b6
... runtime.selectgo(0xc000079f70, 0xc000063eb8, 0x0?, 0x0, 0x0?, 0x1)
...     /__t/go/1.19.10/x64/src/runtime/select.go:328 +0x7bc fp=0xc000079e48 sp=0xc000079d08 pc=0x90e8bc
... google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0004b6680)
...     /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1155 +0x233 fp=0xc000079fc8 sp=0xc000079e48 pc=0x14bb873
... google.golang.org/grpc/internal/transport.NewServerTransport.func4()
...     /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:344 +0x26 fp=0xc000079fe0 sp=0xc000079fc8 pc=0x14b3e26
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x92d341
... created by google.golang.org/grpc/internal/transport.NewServerTransport
...     /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:344 +0x1b18
... 
... goroutine 10 [IO wait]:
... runtime.gopark(0x3620?, 0xb?, 0x0?, 0x0?, 0x7?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc0000edad8 sp=0xc0000edab8 pc=0x8ff0b6
... runtime.netpollblock(0x9405a5?, 0x2000?, 0x0?)
...     /__t/go/1.19.10/x64/src/runtime/netpoll.go:526 +0xf7 fp=0xc0000edb10 sp=0xc0000edad8 pc=0x8f7a77
... internal/poll.runtime_pollWait(0x7fd4e862e328, 0x72)
...     /__t/go/1.19.10/x64/src/runtime/netpoll.go:305 +0x89 fp=0xc0000edb30 sp=0xc0000edb10 pc=0x927d89
... internal/poll.(*pollDesc).wait(0xc00014e080?, 0xc0000d0000?, 0x0)
...     /__t/go/1.19.10/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0000edb58 sp=0xc0000edb30 pc=0x997812
... internal/poll.(*pollDesc).waitRead(...)
...     /__t/go/1.19.10/x64/src/internal/poll/fd_poll_runtime.go:89
... internal/poll.(*FD).Read(0xc00014e080, {0xc0000d0000, 0x8000, 0x8000})
...     /__t/go/1.19.10/x64/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0000edbd8 sp=0xc0000edb58 pc=0x998b7a
... net.(*netFD).Read(0xc00014e080, {0xc0000d0000?, 0x7fd51189b5b8?, 0x180?})
...     /__t/go/1.19.10/x64/src/net/fd_posix.go:55 +0x29 fp=0xc0000edc20 sp=0xc0000edbd8 pc=0xc097a9
... net.(*conn).Read(0xc000124018, {0xc0000d0000?, 0x0?, 0xc0000edc98?})
...     /__t/go/1.19.10/x64/src/net/net.go:183 +0x45 fp=0xc0000edc68 sp=0xc0000edc20 pc=0xc1b9a5
... bufio.(*Reader).Read(0xc0001263c0, {0xc0003e8040, 0x9, 0x0?})
...     /__t/go/1.19.10/x64/src/bufio/bufio.go:237 +0x1bb fp=0xc0000edca0 sp=0xc0000edc68 pc=0xb26d3b
... io.ReadAtLeast({0x2a550e0, 0xc0001263c0}, {0xc0003e8040, 0x9, 0x9}, 0x9)
...     /__t/go/1.19.10/x64/src/io/io.go:332 +0x9a fp=0xc0000edce8 sp=0xc0000edca0 pc=0x98fd1a
... io.ReadFull(...)
...     /__t/go/1.19.10/x64/src/io/io.go:351
... golang.org/x/net/http2.readFrameHeader({0xc0003e8040?, 0x9?, 0xc0006a95f0?}, {0x2a550e0?, 0xc0001263c0?})
...     /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc0000edd38 sp=0xc0000edce8 pc=0x148b66e
... golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003e8000)
...     /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc0000edde8 sp=0xc0000edd38 pc=0x148bed5
... google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0004b6680, 0x0?, 0x0?)
...     /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:642 +0x174 fp=0xc0000edf10 sp=0xc0000edde8 pc=0x14b7374
... google.golang.org/grpc.(*Server).serveStreams(0xc0001143c0, {0x2a75740?, 0xc0004b6680})
...     /go/pkg/mod/google.golang.org/[email protected]/server.go:946 +0x162 fp=0xc0000edf80 sp=0xc0000edf10 pc=0x1529682
... google.golang.org/grpc.(*Server).handleRawConn.func1()
...     /go/pkg/mod/google.golang.org/[email protected]/server.go:889 +0x46 fp=0xc0000edfe0 sp=0xc0000edf80 pc=0x1528e06
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000edfe8 sp=0xc0000edfe0 pc=0x92d341
... created by google.golang.org/grpc.(*Server).handleRawConn
...     /go/pkg/mod/google.golang.org/[email protected]/server.go:888 +0x185
... 
... goroutine 24 [select]:
... runtime.gopark(0xc0004f9928?, 0x2?, 0xa0?, 0xf9?, 0xc0004f988c?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc0004f9708 sp=0xc0004f96e8 pc=0x8ff0b6
... runtime.selectgo(0xc0004f9928, 0xc0004f9888, 0x1800?, 0x1, 0x8?, 0x1)
...     /__t/go/1.19.10/x64/src/runtime/select.go:328 +0x7bc fp=0xc0004f9848 sp=0xc0004f9708 pc=0x90e8bc
... github.com/cloudquery/plugin-sdk/v4/internal/servers/plugin/v3.(*Server).Write(0xc00014fb00, {0x2a742b8, 0xc000052990})
...     /go/pkg/mod/github.com/cloudquery/plugin-sdk/[email protected]/internal/servers/plugin/v3/plugin.go:241 +0x5be fp=0xc0004f9978 sp=0xc0004f9848 pc=0x162d0be
... github.com/cloudquery/plugin-pb-go/pb/plugin/v3._Plugin_Write_Handler({0x27e6040?, 0xc00014fb00}, {0x2a6e0b0?, 0xc0000161e0})
...     /go/pkg/mod/github.com/cloudquery/[email protected]/pb/plugin/v3/plugin_grpc.pb.go:375 +0x9f fp=0xc0004f99b0 sp=0xc0004f9978 pc=0x15ef55f
... github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1({0x27e6040, 0xc00014fb00}, {0x2a6e188, 0xc0004ee870}, 0xc000012618, 0x28f2a20)
...     /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:35 +0x31e fp=0xc0004f9b20 sp=0xc0004f99b0 pc=0x1653ebe
... google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001143c0, {0x2a75740, 0xc0004b6680}, 0xc0001626c0, 0xc00044fb90, 0x3703ca0, 0x0)
...     /go/pkg/mod/google.golang.org/[email protected]/server.go:1638 +0x1303 fp=0xc0004f9e48 sp=0xc0004f9b20 pc=0x152f763
... google.golang.org/grpc.(*Server).handleStream(0xc0001143c0, {0x2a75740, 0xc0004b6680}, 0xc0001626c0, 0x0)
...     /go/pkg/mod/google.golang.org/[email protected]/server.go:1718 +0x9ea fp=0xc0004f9f68 sp=0xc0004f9e48 pc=0x153102a
... google.golang.org/grpc.(*Server).serveStreams.func1.1()
...     /go/pkg/mod/google.golang.org/[email protected]/server.go:959 +0x98 fp=0xc0004f9fe0 sp=0xc0004f9f68 pc=0x1529938
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0004f9fe8 sp=0xc0004f9fe0 pc=0x92d341
... created by google.golang.org/grpc.(*Server).serveStreams.func1
...     /go/pkg/mod/google.golang.org/[email protected]/server.go:957 +0x18c
... 
... goroutine 23 [select]:
... runtime.gopark(0xc0000cff88?, 0x2?, 0xb8?, 0x4a?, 0xc0000cff84?)
...     /__t/go/1.19.10/x64/src/runtime/proc.go:363 +0xd6 fp=0xc0000cfe10 sp=0xc0000cfdf0 pc=0x8ff0b6
... runtime.selectgo(0xc0000cff88, 0xc0000cff80, 0x1529938?, 0x0, 0x2a75740?, 0x1)
...     /__t/go/1.19.10/x64/src/runtime/select.go:328 +0x7bc fp=0xc0000cff50 sp=0xc0000cfe10 pc=0x90e8bc
... database/sql.(*DB).connectionOpener(0xc0000c6340, {0x2a63008, 0xc0003f72c0})
...     /__t/go/1.19.10/x64/src/database/sql/sql.go:1224 +0x8d fp=0xc0000cffb8 sp=0xc0000cff50 pc=0x9c41cd
... database/sql.OpenDB.func1()
...     /__t/go/1.19.10/x64/src/database/sql/sql.go:792 +0x2e fp=0xc0000cffe0 sp=0xc0000cffb8 pc=0x9c27ee
... runtime.goexit()
...     /__t/go/1.19.10/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x92d341
... created by database/sql.OpenDB
...     /__t/go/1.19.10/x64/src/database/sql/sql.go:792 +0x18d
... 
... rax    0x0
... rbx    0x1b
... rcx    0x7fd51196accc
... rdx    0x6
... rdi    0x15
... rsi    0x1b
... rbp    0x7fd4dabfd6c0
... rsp    0x7fd4dabfa3a0
... r8     0x0
... r9     0x7fd4dabf9eb0
... r10    0x8
... r11    0x246
... r12    0x6
... r13    0x7fd4dabfa5a0
... r14    0x7fd4dabfa580
... r15    0x7fd4d0160dd0
... rip    0x7fd51196accc
... rflags 0x246
... cs     0x33
... fs     0x0
... gs     0x0
... 
/ Syncing resources... (25/-, 249 resources/s) [0s] source plugin process failed with exit status 2
... Error: failed to sync v3 source ...: write client returned error (insert): error reading from server: EOF

CloudQuery version

cloudquery version 3.9.0

Additional Context

More details can be found in this discord thread

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions