I see that these breaking changes are due to SDKv3 using arrow.DataType names in doc generation directly [1][2].
Is this expected? Or should these be compatible with previous versions, namely v2?

[1] https://github.com/cloudquery/plugin-sdk/blob/e890385102e2668a16e35cff75fe2ffea32f2937/plugins/source/docs.go#L231
[2] https://github.com/apache/arrow/blob/07f142df646c9070fc9c627004dfdc96914fe487/go/arrow/datatype_binary.go#L28

@brucehsu the breaking change in sources is expected when performing the upgrade to plugin-sdk/v3.
I'll mark the PR as breaking accordingly.

Nice one, thank you @brucehsu!

We will have to wait for all destinations to be migrated to v3 (and released) before merging this, because sources using SDK v3 will require a destination that also supports SDK v3.

The other thing is that I think we should/will try and update the "Breaking Change" detection script that commented above to accept Arrow type names as aliases of the old names, so it looks a bit less scary and we can verify that there are no real breaking changes. Not saying you need to do this for this PR, I can take a look at this soon :)

@hermanschaaf Gotcha! I was about to ask how do we cope with the change in types when say writing to an existing PostgreSQL database since that'd probably also change the types of columns.

I was about to ask how do we cope with the change in types when say writing to an existing PostgreSQL database since that'd probably also change the types of columns.

@brucehsu I can probably shed some light on this! Previously we had our own type system, which we refer to as CQTypes. As you know, we're now migrating to use Arrow types. We've created Arrow extensions for some of the unique CQTypes, so there is a fairly straight-forward mapping between CQTypes and Arrow types. For example, Int -> Int64, String -> String, JSON -> JSON, etc.

We're first migrating all the destination plugins to support plugin-sdk V3 right now. It's mostly easy at this point, as most of them already support Arrow from the v2 migration. Now when a destination plugin receives an Arrow type of Int64, it will translate this to the same column type that it was using before when it received an Int CQType. Destinations have already been doing this since V2. Because it's translating to the same column type as before, it doesn't break the schema for users.

The next step will be to have all the sources use and send Arrow types directly. We'll start out by translating CQTypes to their direct Arrow equivalents in the source (as you've done here by upgrading to v3 😄 ). Then in the next phase, we will start allowing more Arrow types to be used, which may result in new, or more specific, column types in the destinations. For example, we could convert nested Go structs to their equivalent Arrow struct types, and then send that over the wire instead of a JSON column. The destination can then decide how to handle this: BigQuery might write it as a RECORD column, while postgres might still write it as a JSON column. We will also introduce a setting that allows users to maintain the previous behavior, so that this again does not need to be breaking.