bug: Okta source plugin fails to unmarshall data #11149
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Some okta resources are failing to be retrieved. The output data is smaller than expected, and errors are seen in the logs.
Expected Behavior
All data retrieved from Okta, with no errors
CloudQuery (redacted) config
kind: source
spec:
# Source spec section
name: okta
path: cloudquery/okta
version: "v3.0.0"
tables: ["*"]
destinations: ["local-csv"]
concurrency: 1
spec:
# Required. Your Okta domain name
domain: "https://${OKTA_DOMAIN}/"
# Optional. Okta Token to access API, you can set this with OKTA_API_TOKEN environment variable
# ⚠️ Warning - Your token should be kept secret and not committed to source control
token: ${OKTA_TOKEN}
---
kind: destination
spec:
name: "local-csv"
path: "cloudquery/file"
version: "v3.2.0"
write_mode: "append"
spec:
path: "/tmp/csv_output/{{TABLE}}.{{FORMAT}}"
format: csvSteps To Reproduce
- Run
cloudquery sync ./cq.cfg --log-level debug - Observe errors were reported:
Sync completed successfully. Resources: 21600, Errors: 109, Panics: 0, Time: 11m6s - Observe the resulting CSV files are missing data
CloudQuery (redacted) logs
2023-05-31T16:47:36Z INF Loading spec(s) args=["./cq.cfg"] module=cli
2023-05-31T16:47:36Z INF Source plugin server listening address=/tmp/cq-aSqYBsBdgODBBRBP.sock module=cli
2023-05-31T16:47:37Z INF Destination plugin server listening address=/tmp/cq-DRYIeIhZpQcoDnMw.sock module=cli
2023-05-31T16:47:38Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.012 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.026 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.023 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v0.Discovery grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.037 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF Start sync destinations=["local-csv"] module=cli source="okta (v3.0.0)" sync_time=2023-05-31T16:47:38Z
2023-05-31T16:47:38Z INF started call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.005 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.094 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF started call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.008 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetDynamicTables grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=1.256 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF started call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.015 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF finished call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.16 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF started call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.006 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF finished call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.209 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF End migration destinations=["local-csv"] module=cli source="okta (v3.0.0)" time_took=0.000496474
2023-05-31T16:47:38Z INF Start fetching resources destinations=["local-csv"] module=cli source="okta (v3.0.0)"
2023-05-31T16:47:38Z INF started call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.037 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF started call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=0.127 module=cli peer.address=@ protocol=grpc
2023-05-31T16:47:38Z INF top level table resolver started client=okta module=okta-src table=okta_users
2023-05-31T16:47:40Z INF table sync finished client=okta errors=0 module=okta-src resources=674 table=okta_users
2023-05-31T16:47:40Z INF top level table resolver started client=okta module=okta-src table=okta_groups
2023-05-31T16:48:00Z INF batch written successfully duration=22.687368 len=674 module=cli table=okta_users
2023-05-31T16:48:08Z INF Sync in progress module=okta-src total_errors=0 total_panics=0 total_resources=2790
2023-05-31T16:48:38Z INF Sync in progress module=okta-src total_errors=0 total_panics=0 total_resources=4548
2023-05-31T16:49:06Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_group_users
2023-05-31T16:49:08Z INF Sync in progress module=okta-src total_errors=1 total_panics=0 total_resources=5578
2023-05-31T16:49:14Z INF table sync finished client=okta errors=0 module=okta-src resources=442 table=okta_groups
2023-05-31T16:49:14Z INF table sync finished client=okta errors=1 module=okta-src resources=4516 table=okta_group_users
2023-05-31T16:49:14Z INF top level table resolver started client=okta module=okta-src table=okta_applications
2023-05-31T16:49:28Z ERR table resolver finished with error error="json: cannot unmarshal array into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:49:34Z INF batch written successfully duration=62.258013 len=4516 module=cli table=okta_group_users
2023-05-31T16:49:34Z INF batch written successfully duration=11.596622 len=442 module=cli table=okta_groups
2023-05-31T16:49:38Z INF Sync in progress module=okta-src total_errors=2 total_panics=0 total_resources=7512
2023-05-31T16:49:41Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:49:50Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:49:51Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:49:51Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:49:54Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:49:58Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:50:00Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:50:01Z ERR table resolver finished with error error="json: cannot unmarshal array into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:50:08Z INF Sync in progress module=okta-src total_errors=10 total_panics=0 total_resources=8346
2023-05-31T16:50:16Z INF batch written successfully duration=75.381862 len=2473 module=cli table=okta_application_users
2023-05-31T16:50:32Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:50:38Z INF Sync in progress module=okta-src total_errors=11 total_panics=0 total_resources=10723
2023-05-31T16:50:41Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:50:45Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:50:51Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:07Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:51:08Z INF Sync in progress module=okta-src total_errors=15 total_panics=0 total_resources=11071
2023-05-31T16:51:11Z INF batch written successfully duration=2.672914 len=70 module=cli table=okta_application_group_assignments
2023-05-31T16:51:11Z INF batch written successfully duration=4.559051 len=42 module=cli table=okta_applications
2023-05-31T16:51:11Z INF batch written successfully duration=86.420244 len=2697 module=cli table=okta_application_users
2023-05-31T16:51:19Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:23Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:23Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:51:26Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:27Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:51:27Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:36Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:36Z ERR table resolver finished with error error="json: cannot unmarshal bool into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:51:38Z INF Sync in progress module=okta-src total_errors=23 total_panics=0 total_resources=11641
2023-05-31T16:51:40Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:43Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:45Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:51:45Z ERR table resolver finished with error error="json: cannot unmarshal array into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:51:48Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:52:05Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_users
2023-05-31T16:52:08Z INF Sync in progress module=okta-src total_errors=29 total_panics=0 total_resources=12165
2023-05-31T16:52:09Z INF batch written successfully duration=31.255065 len=1045 module=cli table=okta_application_users
2023-05-31T16:52:09Z INF batch written successfully duration=2.487061 len=49 module=cli table=okta_application_group_assignments
2023-05-31T16:52:19Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:52:19Z ERR table resolver finished with error error="json: cannot unmarshal array into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:52:21Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:52:31Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:52:38Z INF Sync in progress module=okta-src total_errors=33 total_panics=0 total_resources=13549
2023-05-31T16:52:41Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:52:44Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:52:46Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:03Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:06Z INF batch written successfully duration=1.200815 len=63 module=cli table=okta_application_group_assignments
2023-05-31T16:53:07Z INF batch written successfully duration=54.49704 len=1530 module=cli table=okta_application_users
2023-05-31T16:53:08Z INF Sync in progress module=okta-src total_errors=37 total_panics=0 total_resources=13758
2023-05-31T16:53:19Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:21Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:21Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:24Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:24Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:53:29Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:29Z ERR table resolver finished with error error="json: cannot unmarshal array into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:53:32Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:33Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:53:36Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:36Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:53:38Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:38Z INF Sync in progress module=okta-src total_errors=49 total_panics=0 total_resources=14299
2023-05-31T16:53:41Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:53:42Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:53:42Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:01Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:05Z INF batch written successfully duration=24.968301 len=922 module=cli table=okta_application_users
2023-05-31T16:54:05Z INF batch written successfully duration=1.842293 len=30 module=cli table=okta_application_group_assignments
2023-05-31T16:54:08Z INF Sync in progress module=okta-src total_errors=53 total_panics=0 total_resources=14710
2023-05-31T16:54:17Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:54:19Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:22Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:22Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:54:23Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:25Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:26Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:54:28Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:31Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:33Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:38Z INF Sync in progress module=okta-src total_errors=63 total_panics=0 total_resources=15039
2023-05-31T16:54:38Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:39Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:54:40Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:40Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:40Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:43Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:43Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:54:47Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:47Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:54:52Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:54:55Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:55:08Z INF Sync in progress module=okta-src total_errors=74 total_panics=0 total_resources=15117
2023-05-31T16:55:09Z INF batch written successfully duration=10.928364 len=331 module=cli table=okta_application_users
2023-05-31T16:55:11Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_users
2023-05-31T16:55:16Z INF batch written successfully duration=3.546323 len=76 module=cli table=okta_application_group_assignments
2023-05-31T16:55:24Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:55:25Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:55:25Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:55:29Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:55:29Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:55:35Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:55:36Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:55:38Z INF Sync in progress module=okta-src total_errors=82 total_panics=0 total_resources=15529
2023-05-31T16:55:45Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:55:45Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:55:47Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:55:47Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:56:02Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_users
2023-05-31T16:56:03Z INF batch written successfully duration=29.857594 len=972 module=cli table=okta_application_users
2023-05-31T16:56:03Z INF batch written successfully duration=1.711176 len=40 module=cli table=okta_application_group_assignments
2023-05-31T16:56:08Z INF Sync in progress module=okta-src total_errors=87 total_panics=0 total_resources=16129
2023-05-31T16:56:18Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:56:38Z INF Sync in progress module=okta-src total_errors=88 total_panics=0 total_resources=17701
2023-05-31T16:56:42Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:56:46Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:56:46Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:57:05Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:57:08Z INF Sync in progress module=okta-src total_errors=92 total_panics=0 total_resources=18619
2023-05-31T16:57:09Z INF batch written successfully duration=1.160819 len=31 module=cli table=okta_application_group_assignments
2023-05-31T16:57:10Z INF batch written successfully duration=10.391386 len=147 module=cli table=okta_applications
2023-05-31T16:57:10Z INF batch written successfully duration=139.760584 len=2459 module=cli table=okta_application_users
2023-05-31T16:57:21Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_users
2023-05-31T16:57:22Z ERR table resolver finished with error error="json: cannot unmarshal array into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:57:34Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:57:38Z INF Sync in progress module=okta-src total_errors=95 total_panics=0 total_resources=20301
2023-05-31T16:57:43Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:57:43Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:57:45Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:57:48Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:57:51Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:58:07Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:58:08Z INF Sync in progress module=okta-src total_errors=101 total_panics=0 total_resources=20675
2023-05-31T16:58:10Z INF batch written successfully duration=68.172554 len=1985 module=cli table=okta_application_users
2023-05-31T16:58:10Z INF batch written successfully duration=1.135512 len=38 module=cli table=okta_application_group_assignments
2023-05-31T16:58:11Z INF batch written successfully duration=1.342109 len=23 module=cli table=okta_applications
2023-05-31T16:58:25Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:58:25Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _ApplicationGroupAssignment.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_group_assignments
2023-05-31T16:58:25Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:58:26Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:58:28Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:58:29Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:58:33Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:58:36Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
2023-05-31T16:58:38Z INF Sync in progress module=okta-src total_errors=109 total_panics=0 total_resources=21009
2023-05-31T16:58:44Z INF table sync finished client=okta errors=0 module=okta-src resources=232 table=okta_applications
2023-05-31T16:58:44Z INF table sync finished client=okta errors=78 module=okta-src resources=15327 table=okta_application_users
2023-05-31T16:58:44Z INF table sync finished client=okta errors=30 module=okta-src resources=409 table=okta_application_group_assignments
2023-05-31T16:58:44Z INF sync finished duration=666526.58945 errors=109 module=okta-src panics=0 resources=21600
2023-05-31T16:58:44Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=666527.2 module=cli peer.address=@ protocol=grpc
2023-05-31T16:58:44Z INF started call grpc.code=OK grpc.component=server grpc.method=GetMetrics grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-05-31T12:58:44-04:00 grpc.time_ms=0.021 module=cli peer.address=@ protocol=grpc
2023-05-31T16:58:44Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetMetrics grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-05-31T12:58:44-04:00 grpc.time_ms=0.2 module=cli peer.address=@ protocol=grpc
2023-05-31T16:58:44Z INF batch written successfully duration=0.615673 len=12 module=cli table=okta_application_group_assignments
2023-05-31T16:58:44Z INF batch written successfully duration=1.151973 len=20 module=cli table=okta_applications
2023-05-31T16:58:44Z INF batch written successfully duration=34.564145 len=913 module=cli table=okta_application_users
2023-05-31T16:58:44Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-05-31T12:47:38-04:00 grpc.time_ms=666568.56 module=cli peer.address=@ protocol=grpc
2023-05-31T16:58:44Z INF End sync destinations=["local-csv"] module=cli source="okta (v3.0.0)" sync_time=2023-05-31T16:47:38Z
2023-05-31T16:58:44Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-05-31T16:58:45Z DBG sending interrupt signal to destination plugin module=cli
2023-05-31T16:58:45Z INF waiting for destination plugin to terminate module=cli
2023-05-31T16:58:45Z INF Got stop signal. Destination plugin server shutting down address=/tmp/cq-DRYIeIhZpQcoDnMw.sock module=cli signal=interrupt
2023-05-31T16:58:45Z DBG sending interrupt signal to source plugin module=cli
2023-05-31T16:58:45Z INF waiting for source plugin to terminate module=cli
2023-05-31T16:58:45Z INF Got stop signal. Source plugin server shutting down address=/tmp/cq-aSqYBsBdgODBBRBP.sock module=cli signal=interrupt
CloudQuery version
cloudquery version 3.5.0
Additional Context
I see 2023-05-31T16:49:06Z ERR table resolver finished with error error="too many requests" client=okta module=okta-src table=okta_group_users which might indicate the root cause is related to #10570.
I wonder if something could also be done to make these errors more helpful?
2023-05-31T16:49:41Z ERR table resolver finished with error error="json: cannot unmarshal string into Go struct field _AppUser.profile of type map[string]interface {}" client=okta module=okta-src table=okta_application_users
Pull request (optional)
- I can submit a pull request
Metadata
Metadata
Assignees
Labels
No labels