-
Notifications
You must be signed in to change notification settings - Fork 547
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
CQ has a strict naming convention according to which "column names must contain only lower-case letters, numbers and underscores, and must start with a lower-case letter or underscore".
Sometimes structs that come with pre-existing APIs are already annotated with json tags that have non-compliant naming, such as json:"OS-SRV-USG:launched_at".
Under such conditions CloudQuery panics.
Expected Behavior
Not to panic; return a meaningful and informative error instead (e.g. error: non compliant column name "OS-SRV-USG:launched_at" at plugin "XYZ": column names must contain only lower-case letters, numbers and underscores, and must start with a lower-case letter or underscore.
CloudQuery (redacted) config
kind: source
spec:
name: "openstack_test_env"
registry: "local"
path: "/path/to/cq-source-openstack/dist/cq-source-openstack_linux_amd64_v1/cq-source-openstack"
version: "v1.0.0"
tables:
["*"]
destinations:
- "sqlite"
spec:
endpoint_url: https://<my-endpoint>:<my-port>
username: <my-username>
password: <my-password>
project_id: <my-project-id>
project_name: <my-project-name>
domain_name: <my-domain>
---
kind: destination
spec:
name: sqlite
path: cloudquery/sqlite
version: "v1.3.6"
spec:
connection_string: ./test.dbSteps To Reproduce
Write a custom plugin using transformers.TransformWithStruct() with a struct that is json-annotated with a funny column name.
CloudQuery (redacted) logs
With a struct with the following field:
type MyStruct struct {
// ...
AFunnyLookingField string `json:"OS-EXT:a-funny-looking-field"`
}I got this:
Loading spec(s) from _test/my-test-env.yaml
panic: found column with invalid name in source plugin: github.com/dihedron-openstack: column name "o_s-_ex_t:a-funny-looking-field" on table "openstack_instances" is not valid: column names must contain only lower-case letters, numbers and underscores, and must start with a lower-case letter or underscore
goroutine 1 [running]:
github.com/cloudquery/plugin-sdk/plugins/source.NewPlugin({0xb26928, 0x1d}, {0xb19c74, 0xb}, {0xc000012310, 0x1, 0x1}, 0xb6bca8, {0x0, 0x0, ...})
/data/workspaces/go/pkg/mod/github.com/cloudquery/[email protected]/plugins/source/plugin.go:160 +0x35c
github.com/dihedron/cq-source-openstack/plugin.Plugin()
/data/workspaces/gomods/cq-source-openstack/plugin/plugin.go:15 +0x8f
main.main()
/data/workspaces/gomods/cq-source-openstack/main.go:9 +0x19
Nothing is written to log.
CloudQuery version
2.5.3
Additional Context
Opened as per @erezrokah 's request; related to enhancement request #10045.
Pull request (optional)
- I can submit a pull request