Skip to content

bug: Panic with transformers.TransformWithStruct() on non compliant column names. #10274

@dihedron

Description

@dihedron

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

CQ has a strict naming convention according to which "column names must contain only lower-case letters, numbers and underscores, and must start with a lower-case letter or underscore".
Sometimes structs that come with pre-existing APIs are already annotated with json tags that have non-compliant naming, such as json:"OS-SRV-USG:launched_at".
Under such conditions CloudQuery panics.

Expected Behavior

Not to panic; return a meaningful and informative error instead (e.g. error: non compliant column name "OS-SRV-USG:launched_at" at plugin "XYZ": column names must contain only lower-case letters, numbers and underscores, and must start with a lower-case letter or underscore.

CloudQuery (redacted) config

kind: source
spec:
  name: "openstack_test_env"
  registry: "local"
  path: "/path/to/cq-source-openstack/dist/cq-source-openstack_linux_amd64_v1/cq-source-openstack"
  version: "v1.0.0"
  tables: 
    ["*"]
  destinations:
    - "sqlite"
  spec:
    endpoint_url: https://<my-endpoint>:<my-port>
    username: <my-username>
    password: <my-password>
    project_id: <my-project-id>
    project_name: <my-project-name>
    domain_name: <my-domain>
---
kind: destination
spec:
  name: sqlite
  path: cloudquery/sqlite
  version: "v1.3.6"
  spec:
    connection_string: ./test.db

Steps To Reproduce

Write a custom plugin using transformers.TransformWithStruct() with a struct that is json-annotated with a funny column name.

CloudQuery (redacted) logs

With a struct with the following field:

type MyStruct struct {
    // ...
   AFunnyLookingField string `json:"OS-EXT:a-funny-looking-field"`
}

I got this:

Loading spec(s) from _test/my-test-env.yaml
panic: found column with invalid name in source plugin: github.com/dihedron-openstack: column name "o_s-_ex_t:a-funny-looking-field" on table "openstack_instances" is not valid: column names must contain only lower-case letters, numbers and underscores, and must start with a lower-case letter or underscore

goroutine 1 [running]:
github.com/cloudquery/plugin-sdk/plugins/source.NewPlugin({0xb26928, 0x1d}, {0xb19c74, 0xb}, {0xc000012310, 0x1, 0x1}, 0xb6bca8, {0x0, 0x0, ...})
        /data/workspaces/go/pkg/mod/github.com/cloudquery/[email protected]/plugins/source/plugin.go:160 +0x35c
github.com/dihedron/cq-source-openstack/plugin.Plugin()
        /data/workspaces/gomods/cq-source-openstack/plugin/plugin.go:15 +0x8f
main.main()
        /data/workspaces/gomods/cq-source-openstack/main.go:9 +0x19

Nothing is written to log.

CloudQuery version

2.5.3

Additional Context

Opened as per @erezrokah 's request; related to enhancement request #10045.

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions