Skip to content

bug: char() bug for mysql source > mysql destination #11325

@pandaijoeyong

Description

@pandaijoeyong

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

when setting up a source of MariaDB to MySQL, (using the mysql connector), i get an char() error.

ec2 AMI: ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20230516

Error:
Error: failed to sync v1 source mysql: rpc error: code = Unknown desc = got unknown MySQL type "char(32)" for column "USER" of table "users" while trying to convert it to CloudQuery internal schema type. Supported MySQL types are ["bigint" "bigint unsigned" "bigint(20)" "bigint(20) unsigned" "binary(16)" "blob" "datetime" "double" "float" "int" "int unsigned" "int(10) unsigned" "int(11)" "json" "smallint" "smallint unsigned" "smallint(5)" "smallint(5) unsigned" "smallint(6)" "text" "tinyint" "tinyint unsigned" "tinyint(1)" "tinyint(3) unsigned" "tinyint(4)"]

Expected Behavior

no errors.

CloudQuery (redacted) config

<cloudquery-config/mysql_source.yaml>

kind: source
spec:
  name: "mysql"
  path: "cloudquery/mysql"
  version: "v1.2.0"
  tables: ["*"]
  destinations: ["mysql"]
  spec:
    connection_string: "user:password@tcp(dns:port)/dbname"

<cloudquery-config/mysql_destination.yaml>

kind: destination
spec:
  name: "mysql"
  registry: "github"
  path: "cloudquery/mysql"
  version: "v2.1.1"
  spec:
    connection_string: "user:password@tcp(dns:port)/dbname"

Steps To Reproduce

  1. In ubuntu-jammy-22.04, sudo nano the following cloudquery config files
  2. run ./cloudquery sync cloudquery-config/mysql_source.yaml cloudquery-config/mysql_destination.yaml

CloudQuery (redacted) logs

023-06-07T07:43:43Z INF Loading spec(s) args=["./cloudquery-config/mysql_source.yaml","cloudquery-config/mysql_destination.yaml"] module=cli
2023-06-07T07:43:44Z INF Source plugin server listening address=/tmp/cq-kOBmPLmBrRLPvAec.sock module=cli
2023-06-07T07:43:47Z INF Destination plugin server listening address=/tmp/cq-DsZoSMKgqDgBBPdd.sock module=cli
2023-06-07T07:43:48Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-06-07T07:43>
2023-06-07T07:43:48Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-06-07T07:4>
2023-06-07T07:43:48Z INF Start sync destinations=["mysql"] module=cli source="mysql (v1.2.0)" sync_time=2023-06-07T07:43:48Z
2023-06-07T07:43:48Z INF started call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:43:48Z grpc.tim>
2023-06-07T07:43:48Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:43:48Z grpc.ti>
2023-06-07T07:43:48Z INF started call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:43:>
2023-06-07T07:43:48Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:43>
2023-06-07T07:43:48Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:43:48Z grpc.time_ms=>
2023-06-07T07:43:48Z INF finished call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:43:48Z grpc.time_ms>
2023-06-07T07:43:48Z INF started call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:43:48Z grpc.time_ms=0.>
2023-06-07T07:43:48Z INF End sync destinations=["mysql"] module=cli source="mysql (v1.2.0)" sync_time=2023-06-07T07:43:48Z
2023-06-07T07:43:48Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-06-07T07:43:48Z ERR finished call grpc.code=Unknown grpc.component=server grpc.error="Error 1045 (28000): Access denied for user 'pandai'@'172.31.18.25' (using password: YES)" grpc.method=Migrat>
2023-06-07T07:43:48Z INF waiting for destination plugin to terminate module=cli
2023-06-07T07:43:48Z INF Got stop signal. Destination plugin server shutting down address=/tmp/cq-DsZoSMKgqDgBBPdd.sock module=cli signal=interrupt
2023-06-07T07:43:48Z INF waiting for source plugin to terminate module=cli
2023-06-07T07:43:48Z INF Got stop signal. Source plugin server shutting down address=/tmp/cq-kOBmPLmBrRLPvAec.sock module=cli signal=interrupt
2023-06-07T07:43:48Z ERR exiting with error error="failed to sync v1 source mysql: rpc error: code = Unknown desc = Error 1045 (28000): Access denied for user 'pandai'@'172.31.18.25' (using password:>
2023-06-07T07:44:32Z INF Loading spec(s) args=["./cloudquery-config/mysql_source.yaml","cloudquery-config/mysql_destination.yaml"] module=cli
2023-06-07T07:44:32Z INF Source plugin server listening address=/tmp/cq-tBYhtZFoLHLCECUn.sock module=cli
2023-06-07T07:44:33Z INF Destination plugin server listening address=/tmp/cq-WHUkEeZrPawTomVX.sock module=cli
2023-06-07T07:44:34Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-06-07T07:44>
2023-06-07T07:44:34Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-06-07T07:4>
2023-06-07T07:44:34Z INF Start sync destinations=["mysql"] module=cli source="mysql (v1.2.0)" sync_time=2023-06-07T07:44:34Z
2023-06-07T07:44:34Z INF started call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:44:34Z grpc.tim>
2023-06-07T07:44:34Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:44:34Z grpc.ti>
2023-06-07T07:44:34Z INF started call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:44:>
2023-06-07T07:44:34Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:44>
2023-06-07T07:44:34Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:44:34Z grpc.time_ms=>
2023-06-07T07:44:34Z INF finished call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:44:34Z grpc.time_ms>
2023-06-07T07:44:34Z INF started call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:44:34Z grpc.time_ms=0.>
2023-06-07T07:44:34Z ERR finished call grpc.code=Unknown grpc.component=server grpc.error="got unknown MySQL type "char(32)" for column "USER" of table "users" while trying to convert it to Clo>
2023-06-07T07:44:34Z INF End sync destinations=["mysql"] module=cli source="mysql (v1.2.0)" sync_time=2023-06-07T07:44:34Z
2023-06-07T07:44:34Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-06-07T07:44:35Z INF waiting for destination plugin to terminate module=cli
2023-06-07T07:44:35Z INF Got stop signal. Source plugin server shutting down address=/tmp/cq-tBYhtZFoLHLCECUn.sock module=cli signal=interrupt
2023-06-07T07:44:35Z ERR exiting with error error="failed to sync v1 source mysql: rpc error: code = Unknown desc = got unknown MySQL type "char(32)" for column "USER" of table "users" while tr>
2023-06-07T07:48:27Z INF Loading spec(s) args=["./cloudquery-config/mysql_source.yaml","cloudquery-config/mysql_destination.yaml"] module=cli
2023-06-07T07:48:27Z INF Source plugin server listening address=/tmp/cq-eNBnZaYPOjpFcLmN.sock module=cli
2023-06-07T07:48:28Z INF Destination plugin server listening address=/tmp/cq-FEpEaiyxbmMVSsrj.sock module=cli
2023-06-07T07:48:29Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-06-07T07:48>
2023-06-07T07:48:29Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-06-07T07:4>
2023-06-07T07:48:29Z INF Start sync destinations=["mysql"] module=cli source="mysql (v1.2.0)" sync_time=2023-06-07T07:48:29Z
2023-06-07T07:48:29Z INF started call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:48:29Z grpc.tim>
2023-06-07T07:48:29Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:48:29Z grpc.ti>
2023-06-07T07:48:29Z INF started call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:48:>
2023-06-07T07:48:29Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-06-07T07:48>
2023-06-07T07:48:29Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:48:29Z grpc.time_ms=>
2023-06-07T07:48:29Z INF finished call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:48:29Z grpc.time_ms>
2023-06-07T07:48:29Z INF started call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-06-07T07:48:29Z grpc.time_ms=0.>
2023-06-07T07:48:30Z ERR finished call grpc.code=Unknown grpc.component=server grpc.error="got unknown MySQL type "varchar(255)" for column "log_name" of table "activity_log" while trying to co>
2023-06-07T07:48:30Z INF End sync destinations=["mysql"] module=cli source="mysql (v1.2.0)" sync_time=2023-06-07T07:48:29Z
2023-06-07T07:48:30Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-06-07T07:48:30Z INF waiting for destination plugin to terminate module=cli
2023-06-07T07:48:30Z INF Got stop signal. Destination plugin server shutting down address=/tmp/cq-FEpEaiyxbmMVSsrj.sock module=cli signal=interrupt
2023-06-07T07:48:30Z INF waiting for source plugin to terminate module=cli
2023-06-07T07:48:30Z INF Got stop signal. Source plugin server shutting down address=/tmp/cq-eNBnZaYPOjpFcLmN.sock module=cli signal=interrupt
2023-06-07T07:48:30Z ERR exiting with error error="failed to sync v1 source mysql: rpc error: code = Unknown desc = got unknown MySQL type "varchar(255)" for column "log_name" of table "activity>

CloudQuery version

cloudquery command not found?

Additional Context

No response

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions