bug (mysql): missing records for several tables #11962
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Using source plugin GCP and writing data simultaneously to 2 destination plugins: Postgres and MySQL.
The number of records differs between the 2 databases!
The Postgres number is correct, the MySQL number is incorrect (too low).
Postgres:
select count(*) from gcp_compute_instances;
count|704
select count(*) from gcp_container_clusters
count|72
MySQL/MariaDB:
select count(*) from gcp_compute_instances;
+----------+
| count(*) |
+----------+
| 70 |
+----------+
select count(*) from gcp_container_clusters;
+----------+
| count(*) |
+----------+
| 38 |
+----------+
Same problem for gcp_compute_addresses.
The row counts for gcp_storage_buckets and gcp_iam_service_accounts are identical, so correct.
Expected Behavior
The number of records should be the same in both databases, Postgres and MySQL/MariaDB.
CloudQuery (redacted) config
ind: destination
spec:
name: "assetdb_gcp_org1"
registry: "github"
path: "cloudquery/mysql"
version: "v2.2.2"
write_mode: "overwrite-delete-stale"
migrate_mode: "forced"
spec:
connection_string: "user:password@(asset-db-hostname)/gcp_org1"
---
kind: destination
spec:
name: "assetdb_pq_gcp_org1"
registry: "github"
path: "cloudquery/postgresql"
version: "v4.2.2"
write_mode: "overwrite-delete-stale"
migrate_mode: "forced"
spec:
connection_string: "postgresql://user:password@asset-db-pq-hostname/gcp_org1?sslmode=prefer"
pgx_log_level: info
---
kind: source
spec:
name: "gcp_org1"
path: "cloudquery/gcp"
version: "v9.2.1"
concurrency: 1500
tables: ["gcp_apigateway_apis", "gcp_artifactregistry_locations", "gcp_artifactregistry_repositories", "gcp_compute_vpn_gateways", "gcp_compute_addresses", "gcp_compute_instances", "gcp_container_clusters", "gcp_functions_functions", "gcp_iam_service_accounts", "gcp_iam_service_account_keys", "gcp_resourcemanager_projects", "gcp_storage_buckets"]
skip_dependent_tables: true
destinations: ["assetdb_gcp_org1", "assetdb_pq_gcp_org1"]
spec:
backoff_retries: 3Steps To Reproduce
Using bitnami/mariadb:10.11.4-debian-11-r0 as MySQL/MariaDB server.
Using bitnami/postgresql:15.3.0-debian-11-r17 as Postgres server
Running ./cloudquery sync.
CloudQuery (redacted) logs
2023-07-05T07:09:43Z INF Loading spec(s) args=["./"] module=cli
2023-07-05T07:09:44Z INF Source plugin server listening address=/tmp/cq-PRnlUnUepKNhgvQM.sock module=cli
2023-07-05T07:09:45Z INF Destination plugin server listening address=/tmp/cq-EDHrTaheupGGEiSf.sock module=cli
2023-07-05T07:09:47Z INF Destination plugin server listening address=/tmp/cq-GpnoHdPtDOtoppqW.sock module=cli
2023-07-05T07:09:48Z INF Destination plugin server listening address=/tmp/cq-erTVxuuBdtdhBmWd.sock module=cli
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.027 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.053 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.063 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.09 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.025 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.052 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.019 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.038 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF Start sync destinations=["assetdb_gcp_org1 ([email protected])","assetdb_pq_gcp_org1 ([email protected])","file_json_myorg ([email protected])"] module=cli source="gcp_org1 ([email protected])" sync_time=2023-07-05T07:09:49Z
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.011 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.497 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.018 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=1.443 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.014 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.221 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.013 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF Initializing postgresql destination module=pg-dest pgx_log_level=info
2023-07-05T07:09:49Z INF Connect database=gcp_org1 host=asset-db-pq-hostname module=pgx pid=6341 port=5432
2023-07-05T07:09:49Z INF Prepare alreadyPrepared=false module=pgx name=stmtcache_1 pid=6341 sql="select current_database()"
2023-07-05T07:09:49Z INF Query args=[] commandTag="SELECT 1" module=pgx pid=6341 sql="select current_database()"
2023-07-05T07:09:49Z INF Prepare alreadyPrepared=false module=pgx name=stmtcache_2 pid=6341 sql="select current_schema()"
2023-07-05T07:09:49Z INF Query args=[] commandTag="SELECT 1" module=pgx pid=6341 sql="select current_schema()"
2023-07-05T07:09:49Z INF Prepare alreadyPrepared=false module=pgx name=stmtcache_3 pid=6341 sql="select version()"
2023-07-05T07:09:49Z INF Query args=[] commandTag="SELECT 1" module=pgx pid=6341 sql="select version()"
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=69.902 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.02 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF finished call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.299 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF started call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=0.017 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:49Z INF Migrating table module=mysql table=gcp_apigateway_apis
2023-07-05T07:09:49Z INF Table exists, force migration required module=mysql table=gcp_apigateway_apis
2023-07-05T07:09:49Z INF Migrating table module=mysql table=gcp_artifactregistry_locations
2023-07-05T07:09:49Z INF Table exists, force migration required module=mysql table=gcp_artifactregistry_locations
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_artifactregistry_repositories
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_artifactregistry_repositories
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_compute_addresses
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_compute_addresses
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_compute_instances
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_compute_instances
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_compute_vpn_gateways
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_compute_vpn_gateways
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_container_clusters
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_container_clusters
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_functions_functions
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_functions_functions
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_iam_service_accounts
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_iam_service_accounts
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_iam_service_account_keys
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_iam_service_account_keys
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_resourcemanager_projects
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_resourcemanager_projects
2023-07-05T07:09:50Z INF Migrating table module=mysql table=gcp_storage_buckets
2023-07-05T07:09:50Z INF Table exists, force migration required module=mysql table=gcp_storage_buckets
2023-07-05T07:09:50Z INF finished call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:49Z grpc.time_ms=486.185 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF started call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=0.015 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF Prepare alreadyPrepared=false module=pgx name=stmtcache_4 pid=6341 sql="\nSELECT\n\tcolumns.ordinal_position AS ordinal_position,\n\tpg_class.relname AS table_name,\n\tpg_attribute.attname AS column_name,\n\tpg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) AS data_type,\n\tCASE \n\t\tWHEN conkey IS NOT NULL AND array_position(conkey, pg_attribute.attnum) > 0 THEN true\n\t\tELSE false\n\tEND AS is_primary_key,\n\tCASE \n\t\tWHEN pg_attribute.attnotnull THEN true\n\t\tELSE false\n\tEND AS not_null,\n\tCOALESCE(pg_constraint.conname, '') AS primary_key_constraint_name\nFROM\n\tpg_catalog.pg_attribute\n\tINNER JOIN\n\tpg_catalog.pg_class ON pg_class.oid = pg_attribute.attrelid\n\tINNER JOIN\n\tpg_catalog.pg_namespace ON pg_namespace.oid = pg_class.relnamespace\n\tLEFT JOIN\n\tpg_catalog.pg_constraint ON pg_constraint.conrelid = pg_attribute.attrelid\n\tAND conkey IS NOT NULL AND array_position(conkey, pg_attribute.attnum) > 0\n\tAND contype = 'p'\n\tINNER JOIN\n\tinformation_schema.columns ON columns.table_name = pg_class.relname AND columns.column_name = pg_attribute.attname AND columns.table_schema = pg_catalog.pg_namespace.nspname\nWHERE\n\tpg_attribute.attnum > 0\n\tAND NOT pg_attribute.attisdropped\n\tAND pg_catalog.pg_namespace.nspname = 'public'\nORDER BY\n\ttable_name ASC , ordinal_position ASC;\n"
2023-07-05T07:09:50Z INF Query args=[] commandTag="SELECT 299" module=pgx pid=6341 sql="\nSELECT\n\tcolumns.ordinal_position AS ordinal_position,\n\tpg_class.relname AS table_name,\n\tpg_attribute.attname AS column_name,\n\tpg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) AS data_type,\n\tCASE \n\t\tWHEN conkey IS NOT NULL AND array_position(conkey, pg_attribute.attnum) > 0 THEN true\n\t\tELSE false\n\tEND AS is_primary_key,\n\tCASE \n\t\tWHEN pg_attribute.attnotnull THEN true\n\t\tELSE false\n\tEND AS not_null,\n\tCOALESCE(pg_constraint.conname, '') AS primary_key_constraint_name\nFROM\n\tpg_catalog.pg_attribute\n\tINNER JOIN\n\tpg_catalog.pg_class ON pg_class.oid = pg_attribute.attrelid\n\tINNER JOIN\n\tpg_catalog.pg_namespace ON pg_namespace.oid = pg_class.relnamespace\n\tLEFT JOIN\n\tpg_catalog.pg_constraint ON pg_constraint.conrelid = pg_attribute.attrelid\n\tAND conkey IS NOT NULL AND array_position(conkey, pg_attribute.attnum) > 0\n\tAND contype = 'p'\n\tINNER JOIN\n\tinformation_schema.columns ON columns.table_name = pg_class.relname AND columns.column_name = pg_attribute.attname AND columns.table_schema = pg_catalog.pg_namespace.nspname\nWHERE\n\tpg_attribute.attnum > 0\n\tAND NOT pg_attribute.attisdropped\n\tAND pg_catalog.pg_namespace.nspname = 'public'\nORDER BY\n\ttable_name ASC , ordinal_position ASC;\n"
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_apigateway_apis
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_apigateway_apis
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_artifactregistry_locations
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_artifactregistry_locations
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_artifactregistry_repositories
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_artifactregistry_repositories
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_compute_addresses
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_compute_addresses
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_compute_instances
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_compute_instances
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_compute_vpn_gateways
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_compute_vpn_gateways
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_container_clusters
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_container_clusters
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_functions_functions
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_functions_functions
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_iam_service_accounts
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_iam_service_accounts
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_iam_service_account_keys
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_iam_service_account_keys
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_resourcemanager_projects
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_resourcemanager_projects
2023-07-05T07:09:50Z INF Migrating table module=pg-dest table=gcp_storage_buckets
2023-07-05T07:09:50Z INF Table exists, auto-migrating module=pg-dest table=gcp_storage_buckets
2023-07-05T07:09:50Z INF finished call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=23.029 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF started call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=0.017 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF finished call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=0.722 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF End migration destinations=["assetdb_gcp_org1 ([email protected])","assetdb_pq_gcp_org1 ([email protected])","file_json_myorg ([email protected])"] module=cli source="gcp_org1 ([email protected])" time_took=0.512819894
2023-07-05T07:09:50Z INF Start fetching resources destinations=["assetdb_gcp_org1 ([email protected])","assetdb_pq_gcp_org1 ([email protected])","file_json_myorg ([email protected])"] module=cli source="gcp_org1 ([email protected])"
2023-07-05T07:09:50Z INF started call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=0.092 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF started call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=0.19 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF started call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=0.252 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF started call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=0.231 module=cli peer.address=@ protocol=grpc
2023-07-05T07:09:50Z INF No project_ids, folder_ids, or project_filter specified - assuming all active projects module=gcp-src
2023-07-05T07:09:50Z INF Prepare alreadyPrepared=false module=pgx name=stmtcache_5 pid=6341 sql="\nSELECT\n\tcolumns.ordinal_position AS ordinal_position,\n\tpg_class.relname AS table_name,\n\tpg_attribute.attname AS column_name,\n\tpg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) AS data_type,\n\tCASE \n\t\tWHEN conkey IS NOT NULL AND array_position(conkey, pg_attribute.attnum) > 0 THEN true\n\t\tELSE false\n\tEND AS is_primary_key,\n\tCASE \n\t\tWHEN pg_attribute.attnotnull THEN true\n\t\tELSE false\n\tEND AS not_null,\n\tCOALESCE(pg_constraint.conname, '') AS primary_key_constraint_name\nFROM\n\tpg_catalog.pg_attribute\n\tINNER JOIN\n\tpg_catalog.pg_class ON pg_class.oid = pg_attribute.attrelid\n\tINNER JOIN\n\tpg_catalog.pg_namespace ON pg_namespace.oid = pg_class.relnamespace\n\tLEFT JOIN\n\tpg_catalog.pg_constraint ON pg_constraint.conrelid = pg_attribute.attrelid\n\tAND conkey IS NOT NULL AND array_position(conkey, pg_attribute.attnum) > 0\n\tAND contype = 'p'\n\tINNER JOIN\n\tinformation_schema.columns ON columns.table_name = pg_class.relname AND columns.column_name = pg_attribute.attname AND columns.table_schema = pg_catalog.pg_namespace.nspname\nWHERE\n\tpg_attribute.attnum > 0\n\tAND NOT pg_attribute.attisdropped\n\tAND pg_catalog.pg_namespace.nspname = 'public'\nORDER BY\n\ttable_name ASC , ordinal_position ASC;\n"
2023-07-05T07:09:50Z INF Query args=[] commandTag="SELECT 299" module=pgx pid=6341 sql="\nSELECT\n\tcolumns.ordinal_position AS ordinal_position,\n\tpg_class.relname AS table_name,\n\tpg_attribute.attname AS column_name,\n\tpg_catalog.format_type(pg_attribute.atttypid, pg_attribute.atttypmod) AS data_type,\n\tCASE \n\t\tWHEN conkey IS NOT NULL AND array_position(conkey, pg_attribute.attnum) > 0 THEN true\n\t\tELSE false\n\tEND AS is_primary_key,\n\tCASE \n\t\tWHEN pg_attribute.attnotnull THEN true\n\t\tELSE false\n\tEND AS not_null,\n\tCOALESCE(pg_constraint.conname, '') AS primary_key_constraint_name\nFROM\n\tpg_catalog.pg_attribute\n\tINNER JOIN\n\tpg_catalog.pg_class ON pg_class.oid = pg_attribute.attrelid\n\tINNER JOIN\n\tpg_catalog.pg_namespace ON pg_namespace.oid = pg_class.relnamespace\n\tLEFT JOIN\n\tpg_catalog.pg_constraint ON pg_constraint.conrelid = pg_attribute.attrelid\n\tAND conkey IS NOT NULL AND array_position(conkey, pg_attribute.attnum) > 0\n\tAND contype = 'p'\n\tINNER JOIN\n\tinformation_schema.columns ON columns.table_name = pg_class.relname AND columns.column_name = pg_attribute.attname AND columns.table_schema = pg_catalog.pg_namespace.nspname\nWHERE\n\tpg_attribute.attnum > 0\n\tAND NOT pg_attribute.attisdropped\n\tAND pg_catalog.pg_namespace.nspname = 'public'\nORDER BY\n\ttable_name ASC , ordinal_position ASC;\n"
2023-07-05T07:09:51Z INF No organization_ids or organization_filter specified - assuming all organizations module=gcp-src
2023-07-05T07:09:51Z INF Listing organizations... module=gcp-src
2023-07-05T07:09:51Z INF Found 2500 projects. First 100: module=gcp-src projects=["sys-65108888269647641311091401","sys-25106988965593817093516382","sys-58257826082457756148730758","mdb-org0rz5e","sys-45939093210557396330654212","sys-52153347491513480885763192","sys-92783023946556130905664646","sys-04416425968873458851630332","sys-68226732397885417100954061","sys-07494509635859185881560066","sys-82122387360203666775486495","sys-61179378987655434818041088","sys-84844441510465338751932340","sys-94847617811796173320282453","sys-18877672634283974771393151","sys-19438907523984818352750451","sys-44017926239301917165247489","sys-69590488312526568302708569","sys-56423313888734410065929784","sys-54361326571040643317862339","sys-25639305723846994721115790","sys-95411437683294256692111029","sys-43697511555101498137244812","sys-79582841751592952090165155","sys-32882278741665160405635186","sys-48337099775869477449101289","mdb-org7ke95","mdb0005970-1670431721384","sys-28348302734249606933023072","sys-56917143001871208201639592","sys-66463313355743603724404711","sys-98619113708773161087204733","sys-29606110194597624546291045","sys-01987358896211864607460847","sys-31184401851333238730737798","sys-03974414809859498487932177","sys-11892244823099737045775194","sys-47902115567719231533272543","sys-56184814804128162591570182","sys-61188960796969516708458337","sys-66269153663380525153086155","sys-36407207487454935816999977","sys-58654808258317940182363956","mdb-orgfg37r","sys-00521796609106483217263504","sys-45166966310542201947133451","sys-17560717681339892247488053","sys-27347995862280691194981145","mdb-orghs2ap","mdb-org9hvvr","sys-10217162237778674986306749","sys-08675365922309563148818119","sys-78927354763424372403619456","sys-32910974845125376857893496","sys-87331264204510220703564995","sys-88042863459271282050830266","sys-16410513361614253940284841","sys-63001670845150993142607542","sys-05273555506621538022100473","sys-17543646154582564195662598","sys-73036921300794907451793601","sys-52631980420771213442042454","sys-47008758079531663574419543","sys-18917708700806117457940205","sys-54298312069541821948272439","sys-06092150234550867974799855","sys-01325448630408381973387755","mdb-org6mks8","sys-06242971889300604173509109","sys-62120592862848045924196986","sys-12160589557880681330952758","sys-08474120987340335990872200","sys-95665278901404803094234443","sys-43255010379755352789576026","sys-33230072325042879871699243","sys-30374736917860448090843111","sys-99471043229298175643942684","sys-73298353250170937256744297","mdb-orgk710g","sys-77469456685313334323361569","sys-14567353273628798930918192","sys-32356967807935357607862496","sys-26302152256713813521702882","sys-37196886529151753271665927","sys-51800252138638713186354847","sys-85915014767460480246373261","sys-86515891392327856444933193","sys-73404196386364612838275436","sys-10479355041426313824313577","sys-13074961466617676840277621","sys-55157373610613865769356003","sys-94509482037564687800223931","sys-52851360190767367564609967","sys-71265307056639055063736079","sys-02169129761755670033381601","sys-96964659805957048132024301","sys-08368805212036643686788966","sys-58604809744500801584002517","sys-06802212065366862441287787","sys-09603924409887721826104063"]
2023-07-05T07:09:51Z INF Found 1 organizations in folders module=gcp-src organizations=["organizations/123456789012"]
2023-07-05T07:09:51Z INF Retrieved organizations module=gcp-src orgs=[{"creationTime":"2017-03-03T18:04:43.943Z","displayName":"myorg.com","lifecycleState":"ACTIVE","name":"organizations/123456789012","owner":{"directoryCustomerId":"1234567890"}}]
2023-07-05T07:09:51Z INF top level table resolver started client=project:sys-18877672634283974771393151 module=gcp-src table=gcp_apigateway_apis
[...]
2023-07-05T07:10:21Z INF Sync in progress in_progress_tables=["gcp_compute_addresses"] module=gcp-src num_in_progress_tables=1 num_queued_tables=9 queued_tables=["gcp_compute_addresses","gcp_compute_instances","gcp_compute_vpn_gateways","gcp_container_clusters","gcp_functions_functions","gcp_iam_service_account_keys","gcp_iam_service_accounts","gcp_resourcemanager_projects","gcp_storage_buckets"] total_errors=6884 total_panics=0 total_resources=725
[...]
2023-07-05T07:11:03Z INF batch written successfully duration=287.010078 len=57 module=cli table=gcp_compute_instances
[...]
2023-07-05T07:14:37Z INF BatchQuery args=["gcp_org1","2023-07-05T07:09:49.653077763Z"] commandTag="DELETE 0" module=pgx pid=6341 sql="delete from \"gcp_compute_instances\" where _cq_source_name = $1 and _cq_sync_time < $2"
[...]
2023-07-05T07:14:37Z INF BatchClose module=pgx pid=6341
2023-07-05T07:14:37Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=287027.6 module=cli peer.address=@ protocol=grpc
2023-07-05T07:14:37Z INF started call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:14:37Z grpc.time_ms=0.017 module=cli peer.address=@ protocol=grpc
2023-07-05T07:14:37Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:14:37Z grpc.time_ms=0.158 module=cli peer.address=@ protocol=grpc
2023-07-05T07:14:37Z INF batch written successfully duration=168.208841 len=2645 module=cli table=gcp_iam_service_accounts
2023-07-05T07:14:37Z INF batch written successfully duration=399.107276 len=5426 module=cli table=gcp_iam_service_account_keys
2023-07-05T07:14:37Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:09:50Z grpc.time_ms=287517.3 module=cli peer.address=@ protocol=grpc
2023-07-05T07:14:37Z INF started call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:14:37Z grpc.time_ms=0.011 module=cli peer.address=@ protocol=grpc
2023-07-05T07:14:37Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-05T07:14:37Z grpc.time_ms=0.03 module=cli peer.address=@ protocol=grpc
2023-07-05T07:14:37Z INF started call grpc.code=OK grpc.component=server grpc.method=GetMetrics grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-05T07:14:37Z grpc.time_ms=0.021 module=cli peer.address=@ protocol=grpc
2023-07-05T07:14:37Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetMetrics grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-05T07:14:37Z grpc.time_ms=2.586 module=cli peer.address=@ protocol=grpc
2023-07-05T07:14:37Z INF End sync destinations=["assetdb_gcp_org1 ([email protected])","assetdb_pq_gcp_org1 ([email protected])","file_json_myorg ([email protected])"] module=cli source="gcp_org1 ([email protected])" sync_time=2023-07-05T07:09:49Z
2023-07-05T07:14:37Z INF waiting for source plugin to terminate module=cli
2023-07-05T07:14:37Z INF Got stop signal. Destination plugin server shutting down address=/tmp/cq-EDHrTaheupGGEiSf.sock module=cli signal=interrupt
2023-07-05T07:14:37Z INF waiting for source plugin to terminate module=cli
2023-07-05T07:14:37Z INF Got stop signal. Destination plugin server shutting down address=/tmp/cq-GpnoHdPtDOtoppqW.sock module=cli signal=interrupt
2023-07-05T07:14:37Z INF waiting for source plugin to terminate module=cli
2023-07-05T07:14:37Z INF Got stop signal. Destination plugin server shutting down address=/tmp/cq-erTVxuuBdtdhBmWd.sock module=cli signal=interrupt
2023-07-05T07:14:37Z INF waiting for source plugin to terminate module=cli
2023-07-05T07:14:37Z INF Got stop signal. Source plugin server shutting down address=/tmp/cq-PRnlUnUepKNhgvQM.sock module=cli signal=interrupt
CloudQuery version
v3.5.4 / v3.8.0
Additional Context
Changing write_mode from overwrite-delete-stale to overwrite does not increase the number of records for MySQL destination server.
This is collecting data from an entire GCP organization, consisting of hundred of GCP projects.
Pull request (optional)
- I can submit a pull request
Metadata
Metadata
Assignees
Labels
No labels