-
Notifications
You must be signed in to change notification settings - Fork 544
Description
Describe the Bug
Observations after testing new MSSQL destination plugin:
- Primary Key Violations
- Unicode characters
I am using the Azure source plugin. In the tags column I am seeing issues with unicode characters
{"BillTo":"P\u0026T"}instead of{"BillTo":"P&T"}. This works properly with Postgres destination.
Expected Behavior
No primary key violations, and data values with the actual special characters instead of the escaped unicode representations.
CloudQuery Version
cloudquery version 2.2.0
azure (v3.4.0) -> mssql (v1.0.0)
Debug Output
{"level":"error","module":"cli","class":14,"line_no":24,"message":"Violation of PRIMARY KEY constraint 'azure_network_azure_firewall_fqdn_tags_cqpk'. Cannot insert duplicate key in object 'dbo.azure_network_azure_firewall_fqdn_tags'. The duplicate key value is (/subscriptions//resourceGroups//providers/Microsoft.Network/azureFirewallFqdnTags/).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_network_azure_firewall_fqdn_tags","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:24:08Z"} {"level":"error","module":"cli","duration":1765.3814,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_network_azure_firewall_fqdn_tags_cqpk'. Cannot insert duplicate key in object 'dbo.azure_network_azure_firewall_fqdn_tags'. The duplicate key value is (/subscriptions//resourceGroups//providers/Microsoft.Network/azureFirewallFqdnTags/).","len":3,"message":"failed to write batch","table":"azure_network_azure_firewall_fqdn_tags","time":"2023-01-20T21:24:08Z"} {"level":"error","module":"cli","class":14,"line_no":23,"message":"Violation of PRIMARY KEY constraint 'azure_network_express_route_service_providers_cqpk'. Cannot insert duplicate key in object 'dbo.azure_network_express_route_service_providers'. The duplicate key value is (/subscriptions//resourceGroups//providers/Microsoft.Network/expressRouteServiceProviders/).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_network_express_route_service_providers","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:24:21Z"} {"level":"error","module":"cli","duration":44.6381,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_network_express_route_service_providers_cqpk'. Cannot insert duplicate key in object 'dbo.azure_network_express_route_service_providers'. The duplicate key value is (/subscriptions//resourceGroups//providers/Microsoft.Network/expressRouteServiceProviders/).","len":3,"message":"failed to write batch","table":"azure_network_express_route_service_providers","time":"2023-01-20T21:24:21Z"} {"level":"error","module":"cli","class":14,"line_no":23,"message":"Violation of PRIMARY KEY constraint 'azure_authorization_provider_operations_metadata_cqpk'. Cannot insert duplicate key in object 'dbo.azure_authorization_provider_operations_metadata'. The duplicate key value is (/providers/Microsoft.Authorization/providerOperations/Qumulo.QaaS).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_authorization_provider_operations_metadata","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:29:57Z"} {"level":"error","module":"cli","duration":141.0392,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_authorization_provider_operations_metadata_cqpk'. Cannot insert duplicate key in object 'dbo.azure_authorization_provider_operations_metadata'. The duplicate key value is (/providers/Microsoft.Authorization/providerOperations/Qumulo.QaaS).","len":1000,"message":"failed to write batch","table":"azure_authorization_provider_operations_metadata","time":"2023-01-20T21:29:57Z"} {"level":"error","module":"cli","class":14,"line_no":31,"message":"Violation of PRIMARY KEY constraint 'azure_compute_skus_cqpk'. Cannot insert duplicate key in object 'dbo.azure_compute_skus'. The duplicate key value is (Aligned).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_compute_skus","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:29:59Z"} {"level":"error","module":"cli","duration":97.4163,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_compute_skus_cqpk'. Cannot insert duplicate key in object 'dbo.azure_compute_skus'. The duplicate key value is (Aligned).","len":1000,"message":"failed to write batch","table":"azure_compute_skus","time":"2023-01-20T21:29:59Z"} {"level":"error","module":"cli","class":14,"line_no":22,"message":"Violation of PRIMARY KEY constraint 'azure_security_topology_cqpk'. Cannot insert duplicate key in object 'dbo.azure_security_topology'. The duplicate key value is (/subscriptions/<AZ_SUBSCRIPTION>/resourceGroups/<AZ_RG>/providers/Microsoft.Security/locations/westcentralus/topologies/virtualMachines).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_security_topology","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:36:57Z"} {"level":"error","module":"cli","duration":93.7219,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_security_topology_cqpk'. Cannot insert duplicate key in object 'dbo.azure_security_topology'. The duplicate key value is (/subscriptions/<AZ_SUBSCRIPTION>/resourceGroups/<AZ_RG>/providers/Microsoft.Security/locations/westcentralus/topologies/virtualMachines).","len":564,"message":"failed to write batch","table":"azure_security_topology","time":"2023-01-20T21:36:57Z"}
Steps to Reproduce
cloudquery.exe sync .\config --log-level warn --log-format json
kind: source
spec:
name: azure
path: cloudquery/azure
version: "v3.4.0"
destinations: ["mssql"]
tables: ["*"]
concurrency: 150000
scheduler: "round-robin"
skip_tables:
[
"azure_monitor_tenant_activity_logs",
"azure_portal_list_tenant_configuration_violations",
"azure_appservice_web_app_auth_settings",
"azure_policy_assignments",
"azure_security_contacts",
"azure_reservations_reservation_order",
"azure_security_regulatory_compliance_standards",
"azure_storage_tables",
"azure_storage_blob_services",
"azure_storage_containers",
]
spec:
subscriptions:
[
...
]
kind: destination
spec:
name: "mssql"
registry: "github"
path: "cloudquery/mssql"
version: "v1.0.0"
spec:
connection_string: ${MSSQL_CONNECTION_STRING}
auth_mode: ms
schema: dbo
Additional Context
N/A