Skip to content

bug: GetBucketPolicyStatus 404 Error Causing other S3 Bucket Attributes to not Resolve  #12163

New issue
New issue
Closed
#12165
Closed
bug: GetBucketPolicyStatus 404 Error Causing other S3 Bucket Attributes to not Resolve #12163
#12165
Assignees
bbernays
@RyPeck

Description

@RyPeck
RyPeck
opened on Jul 13, 2023

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

When an S3 Bucket has no BucketPolicy - we see the following error -

2023-07-13T15:21:45Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 2SZSQVGGBBH0ES5V, HostID: g8sM7P3WdfLtoGU7GqFeYEdLD8VVeoTj+CaMK+X0HZiPvVyqHPs51Ag+t0OFfJvt5WPuCosyF0Q=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account> module=aws-src region=us-east-1

For the S3 Bucket that meets this condition - we see that other attributes aren't populated.

% grep -h "ryan-test-bucket1" aws_s3_buckets/* | jq .
{
  "_cq_id": "4af2eea4-c9d6-4458-8ae7-c6552621ace9",
  "_cq_parent_id": null,
  "_cq_source_name": "aws-local-test",
  "_cq_sync_time": "2023-07-13 15:20:39.264268",
  "account_id": "257613929019",
  "arn": "arn:aws:s3:::ryan-test-bucket1",
  "block_public_acls": false,
  "block_public_policy": false,
  "creation_date": "2022-07-21 15:13:59",
  "ignore_public_acls": false,
  "logging_target_bucket": null,
  "logging_target_prefix": null,
  "name": "ryan-test-bucket1",
  "ownership_controls": null,
  "policy": {},
  "policy_status": null,
  "region": "us-east-1",
  "replication_role": null,
  "replication_rules": [],
  "restrict_public_buckets": false,
  "tags": {},
  "versioning_mfa_delete": "",
  "versioning_status": ""
}

Expected Behavior

Other attributes should be populated successfully even with the error. Specifically looking for bucket versioning enabled.

aws s3api get-bucket-versioning  --bucket ryan-test-bucket1
{
    "Status": "Enabled"
}

CloudQuery (redacted) config

kind: source
spec:
  name: "aws-local-test"
  
  version: "v19.1.0"
  path: "cloudquery/aws"
  registry: "github"

  destinations: ["file"]

  spec:
    regions:
      - us-east-1

  skip_dependent_tables: true

  tables:
    - aws_s3_buckets

Steps To Reproduce

  1. Create new Bucket (bucket policy will be empty)
  2. Enable Object Versioning
  3. Run CQ Sync

CloudQuery (redacted) logs

2023-07-13T15:20:39Z INF started call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-13T11:20:39-04:00 grpc.time_ms=0.388 module=cli peer.address= protocol=grpc
2023-07-13T15:20:39Z INF started call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-13T11:20:39-04:00 grpc.time_ms=0.093 module=cli peer.address= protocol=grpc
2023-07-13T15:20:39Z INF top level table resolver started client=<account_id>:us-east-1 module=aws-src table=aws_s3_buckets
2023-07-13T15:20:43Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: TZ1A0Y684WN9D480, HostID: 5kJvKrIS/szu3FULLscpUHz6FBU24cINL7WkMu7x2Xe789ybM1wK+nUGFRWci7QIBcSbtTGOCK0=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:20:44Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: THK9QS8FPTNHKDEQ, HostID: T8yusy7jV1bB9PCF8zo1VfXvfp3CBdcDhLELMgHs/CMMGUpOUH2ecfzWCUBvCnRD1C6Ddcy5jlU=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:20:45Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 0QF5XJR31JMFMG5H, HostID: cTFUqjNAiDR8WkUddxZOaGBztGvTKK4fEyAW3SWq+It3A/HJ9C/JARVWxdJ0bijMvxM2frCZCis=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:20:45Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 0QFBB9VYTRM8NE1J, HostID: A6R3+yxSi90hS5amu7lFmx1bGs/c8ECR+Nwv22kV3IyVM0bYpzI3rsrPBzALP16RAI+VsIgE/34=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:20:46Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: BFVYVWR9SFXZKHEK, HostID: KuirqnxF4+CLd1Cp/LqhBxCndiqItaDniZ2fdCZKTIpMap8GY+xS3QOBnGqfuxqPVYZk8n9zz8U=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:20:47Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetPublicAccessBlock, https response error StatusCode: 404, RequestID: 87PA3RV1RT2616F3, HostID: qfvud6bo5wMNxiYxBdlqZyO5yiIvIaap+QgpV+CMTNSdgp68UH7KSPGbwIKLcPYaytsoWbsSwL8=, api error NoSuchPublicAccessBlockConfiguration: The public access block configuration was not found" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:20:49Z ERR pre resource resolver failed error="operation error S3: GetBucketLocation, https response error StatusCode: 403, RequestID: EPY1B1D0JKQ047GD, HostID: sjpRBqm4xp3Ke78ssCs9WqmeMhTrYW1yA8kpAX1w6i4/wr9JxVfBwXAmwGfSnYDR6W1uvT0RqaI=, api error AccessDenied: Access Denied" client=<account_id>:us-east-1 module=aws-src table=aws_s3_buckets
2023-07-13T15:20:50Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 84EEX3JC0SSZTS2S, HostID: 7VSFZdjfaOsWMt5As3q8jjMj9nUl/SOXQrx0yCPrhnaZ7RkrCEZOkv8GZEyzPDUF9ECsJOv9/Nwr57s6Bw/Vww==, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:20:52Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: ST55TT79CCH665S3, HostID: 7WGu1YC1Nao9NMhfdx21ZKXv4R5PxK2XW7yagYrr3Sg/FVrI+1ne9lJ2PwnX7iVDG6ukjul6rMw=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:09Z INF Sync in progress in_progress_tables=["aws_s3_buckets"] module=aws-src num_in_progress_tables=1 num_queued_tables=0 queued_tables=[] total_errors=1 total_panics=0 total_resources=13
2023-07-13T15:21:37Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 7FF859SB4YRS63K7, HostID: Pz01+OTJ5ZbENW3iWXPQn6oXefQ4pEyAini+59mItnpx7bkaYG/HW26mJHSj2M4REmijNUHoZIU=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:38Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: ACBXMNSGT65BDB5M, HostID: RfPjX4tjz3eGY4Uv5TgpIDwRuCNmPI7WDSTRlTT+mrhQD1qCXZR8xlh16BYWGk8anKsBdOLcaSs=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:39Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 4NDHGFTMC6QM5REX, HostID: 8Ih+LoY7pLkLfsdDkfJGjLux8fyh5KuPV66LfCf6ewJjMzL/3uagXyz2tBujBWXaWJQbbzhmtrQ=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:39Z INF Sync in progress in_progress_tables=["aws_s3_buckets"] module=aws-src num_in_progress_tables=1 num_queued_tables=0 queued_tables=[] total_errors=1 total_panics=0 total_resources=17
2023-07-13T15:21:42Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: QTKC0P4SQ2NANE6E, HostID: 07UdvzZ9yBdWvVTi8jYi2w3KSUcMb66g5215cQumOhqrTZGABrBE3TzmK3CkT6DgltRKBodprzo=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:43Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetPublicAccessBlock, https response error StatusCode: 404, RequestID: 1N7QSTSBD3FWQYZT, HostID: /DfkEl3GayaXYCSKRFJu94yWxerLm1jzcJZ96DlIA4gbSG57/8E74ettpeT2aGby2sMo+asAOqw=, api error NoSuchPublicAccessBlockConfiguration: The public access block configuration was not found" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:44Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 28BGG0Z4P0RQQJT7, HostID: +UTa88XxqY+5aZu3YDDzbLGtbAZx5X1ziXyK6yGE95f+R2tSwQlfp1kW8ngrScVt8dMq/L1dNo8=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:44Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 28BV0N5MMC2G1654, HostID: 0PTuqdRXiWK8U4aP0DWDhAU8CQtMx+OWDKTCM4vFgzIlIR+qf25er8Mjbhc6D6AbCO0sCmjh/TU=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:45Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 2SZNHV3DVWWQZ1GA, HostID: XYhlYR3Uc6QuiU84VmP/RmXw1vZoknPj2Z3o10+vA/o8IGrzKCIRa03Rb/w0yfAcbVaF1kutB/8=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:45Z WRN API returned "NotFound" error ignoring it... error="operation error S3: GetBucketPolicyStatus, https response error StatusCode: 404, RequestID: 2SZSQVGGBBH0ES5V, HostID: g8sM7P3WdfLtoGU7GqFeYEdLD8VVeoTj+CaMK+X0HZiPvVyqHPs51Ag+t0OFfJvt5WPuCosyF0Q=, api error NoSuchBucketPolicy: The bucket policy does not exist" account_id=<account_id> module=aws-src region=us-east-1
2023-07-13T15:21:45Z INF table sync finished client=<account_id>:us-east-1 errors=1 module=aws-src resources=25 table=aws_s3_buckets
2023-07-13T15:21:45Z INF sync finished duration=65822.523167 errors=1 module=aws-src panics=0 resources=25
2023-07-13T15:21:45Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-13T11:20:39-04:00 grpc.time_ms=66546.36 module=cli peer.address= protocol=grpc
2023-07-13T15:21:45Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.destination.v1.Destination grpc.start_time=2023-07-13T11:20:39-04:00 grpc.time_ms=66550.52 module=cli peer.address= protocol=grpc
2023-07-13T15:21:45Z INF started call grpc.code=OK grpc.component=server grpc.method=GetMetrics grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-13T11:21:45-04:00 grpc.time_ms=0.037 module=cli peer.address= protocol=grpc
2023-07-13T15:21:45Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetMetrics grpc.method_type=unary grpc.service=cloudquery.source.v2.Source grpc.start_time=2023-07-13T11:21:45-04:00 grpc.time_ms=1.408 module=cli peer.address= protocol=grpc
2023-07-13T15:21:45Z INF End sync destinations=["file (v3.3.0)"] module=cli source="aws-local-test ([email protected])" sync_time=2023-07-13T15:20:39Z
2023-07-13T15:21:45Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-07-13T15:21:45Z INF waiting for source plugin to terminate module=cli
2023-07-13T15:21:45Z INF Got stop signal. Plugin server shutting down address=/var/folders/1g/2gl5j3m13csbs63z10f32byh0000gn/T/cq-RvScyMVuWSrPweGH.sock module=cli signal=interrupt
2023-07-13T15:21:45Z INF waiting for source plugin to terminate module=cli
2023-07-13T15:21:45Z INF Got stop signal. Source plugin server shutting down address=/var/folders/1g/2gl5j3m13csbs63z10f32byh0000gn/T/cq-hzAhZqWMToNMVRZK.sock module=cli signal=interrupt

CloudQuery version

cloudquery version 3.5.4

Additional Context

No response

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions