Skip to content

MSSQL Destination PK violations & unicode characters #6984

@rafaelrodrigues3092

Description

@rafaelrodrigues3092

Describe the Bug

Observations after testing new MSSQL destination plugin:

  1. Primary Key Violations
  2. Unicode characters
    I am using the Azure source plugin. In the tags column I am seeing issues with unicode characters
    {"BillTo":"P\u0026T"} instead of {"BillTo":"P&T"}. This works properly with Postgres destination.

Expected Behavior

No primary key violations, and data values with the actual special characters instead of the escaped unicode representations.

CloudQuery Version

cloudquery version 2.2.0
azure (v3.4.0) -> mssql (v1.0.0)

Debug Output

{"level":"error","module":"cli","class":14,"line_no":24,"message":"Violation of PRIMARY KEY constraint 'azure_network_azure_firewall_fqdn_tags_cqpk'. Cannot insert duplicate key in object 'dbo.azure_network_azure_firewall_fqdn_tags'. The duplicate key value is (/subscriptions//resourceGroups//providers/Microsoft.Network/azureFirewallFqdnTags/).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_network_azure_firewall_fqdn_tags","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:24:08Z"} {"level":"error","module":"cli","duration":1765.3814,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_network_azure_firewall_fqdn_tags_cqpk'. Cannot insert duplicate key in object 'dbo.azure_network_azure_firewall_fqdn_tags'. The duplicate key value is (/subscriptions//resourceGroups//providers/Microsoft.Network/azureFirewallFqdnTags/).","len":3,"message":"failed to write batch","table":"azure_network_azure_firewall_fqdn_tags","time":"2023-01-20T21:24:08Z"} {"level":"error","module":"cli","class":14,"line_no":23,"message":"Violation of PRIMARY KEY constraint 'azure_network_express_route_service_providers_cqpk'. Cannot insert duplicate key in object 'dbo.azure_network_express_route_service_providers'. The duplicate key value is (/subscriptions//resourceGroups//providers/Microsoft.Network/expressRouteServiceProviders/).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_network_express_route_service_providers","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:24:21Z"} {"level":"error","module":"cli","duration":44.6381,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_network_express_route_service_providers_cqpk'. Cannot insert duplicate key in object 'dbo.azure_network_express_route_service_providers'. The duplicate key value is (/subscriptions//resourceGroups//providers/Microsoft.Network/expressRouteServiceProviders/).","len":3,"message":"failed to write batch","table":"azure_network_express_route_service_providers","time":"2023-01-20T21:24:21Z"} {"level":"error","module":"cli","class":14,"line_no":23,"message":"Violation of PRIMARY KEY constraint 'azure_authorization_provider_operations_metadata_cqpk'. Cannot insert duplicate key in object 'dbo.azure_authorization_provider_operations_metadata'. The duplicate key value is (/providers/Microsoft.Authorization/providerOperations/Qumulo.QaaS).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_authorization_provider_operations_metadata","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:29:57Z"} {"level":"error","module":"cli","duration":141.0392,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_authorization_provider_operations_metadata_cqpk'. Cannot insert duplicate key in object 'dbo.azure_authorization_provider_operations_metadata'. The duplicate key value is (/providers/Microsoft.Authorization/providerOperations/Qumulo.QaaS).","len":1000,"message":"failed to write batch","table":"azure_authorization_provider_operations_metadata","time":"2023-01-20T21:29:57Z"} {"level":"error","module":"cli","class":14,"line_no":31,"message":"Violation of PRIMARY KEY constraint 'azure_compute_skus_cqpk'. Cannot insert duplicate key in object 'dbo.azure_compute_skus'. The duplicate key value is (Aligned).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_compute_skus","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:29:59Z"} {"level":"error","module":"cli","duration":97.4163,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_compute_skus_cqpk'. Cannot insert duplicate key in object 'dbo.azure_compute_skus'. The duplicate key value is (Aligned).","len":1000,"message":"failed to write batch","table":"azure_compute_skus","time":"2023-01-20T21:29:59Z"} {"level":"error","module":"cli","class":14,"line_no":22,"message":"Violation of PRIMARY KEY constraint 'azure_security_topology_cqpk'. Cannot insert duplicate key in object 'dbo.azure_security_topology'. The duplicate key value is (/subscriptions/<AZ_SUBSCRIPTION>/resourceGroups/<AZ_RG>/providers/Microsoft.Security/locations/westcentralus/topologies/virtualMachines).","module":"dest-mssql","number":2627,"proc_name":"dbo.cq_proc_azure_security_topology","server_name":"<SQL_SERVER_INSTANCE>","state":1,"time":"2023-01-20T21:36:57Z"} {"level":"error","module":"cli","duration":93.7219,"error":"mssql: Violation of PRIMARY KEY constraint 'azure_security_topology_cqpk'. Cannot insert duplicate key in object 'dbo.azure_security_topology'. The duplicate key value is (/subscriptions/<AZ_SUBSCRIPTION>/resourceGroups/<AZ_RG>/providers/Microsoft.Security/locations/westcentralus/topologies/virtualMachines).","len":564,"message":"failed to write batch","table":"azure_security_topology","time":"2023-01-20T21:36:57Z"}

Steps to Reproduce

cloudquery.exe sync .\config --log-level warn --log-format json
kind: source
spec:
 name: azure
 path: cloudquery/azure
 version: "v3.4.0"
 destinations: ["mssql"]
 tables: ["*"]
 concurrency: 150000
 scheduler: "round-robin"
 skip_tables:
  [
   "azure_monitor_tenant_activity_logs",
   "azure_portal_list_tenant_configuration_violations",
   "azure_appservice_web_app_auth_settings",
   "azure_policy_assignments",
   "azure_security_contacts",
   "azure_reservations_reservation_order",
   "azure_security_regulatory_compliance_standards",
   "azure_storage_tables",
   "azure_storage_blob_services",
   "azure_storage_containers",
  ]
 spec:
  subscriptions:
   [
    ...
   ]

kind: destination
spec:
 name: "mssql"
 registry: "github"
 path: "cloudquery/mssql"
 version: "v1.0.0"
 spec:
  connection_string: ${MSSQL_CONNECTION_STRING}
  auth_mode: ms
  schema: dbo

Additional Context

N/A

References

#6145 (comment)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions