Skip to content

Important Update - Amazon CloudWatch protocol changes in Public Preview will affect LocalStack #13028

@yasmewad

Description

@yasmewad

LocalStack team,

We’re from the AWS SDKs team and are reaching out about an important upcoming change to Amazon CloudWatch that may impact this library and its users.

What's changing

Amazon CloudWatch is migrating from the AWS Query protocol to the more efficient AWS JSON 1.0 and Smithy RPC v2 CBOR protocols. This change is currently available in public preview and will become the default behavior via a release in the coming months.

Based on our experience with similar protocol migrations, LocalStack appears to be affected by these protocol changes.

Testing Results

We tested LocalStack for CloudWatch with the boto3 preview SDK and encountered issues in the existing CloudWatch tests located at [tests/aws/services/cloudwatch](https://github.com/localstack/localstack/tree/main/tests/aws/services/cloudwatch) path. The tests fail with the following error:

ERROR: Operation detection failed. Missing Action in request for query-protocol service ServiceModel(cloudwatch)
HTTP Status: 500 Internal Server Error

Example error response:

<ErrorResponse xmlns="http://monitoring.amazonaws.com/doc/2010-08-01/">
  <Error>
    <Code>InternalError</Code>
    <Message>exception while calling cloudwatch with unknown operation: Operation detection failed. Missing Action in request for query-protocol service ServiceModel(cloudwatch).</Message>
  </Error>
  <RequestId>38ee9b95-671c-44eb-a123-634b16108cca</RequestId>
</ErrorResponse>

LocalStack container logs:

2025-08-19T01:20:47.557 ERROR --- [et.reactor-0] l.aws.handlers.logging     : exception during call chain: Operation detection failed. Missing Action in request for query-protocol service ServiceModel(cloudwatch).
2025-08-19T01:20:47.557  INFO --- [et.reactor-0] localstack.request.http    : POST / => 500

Based on our analysis, the issue is related to detection of the protocol as CloudWatch previously used query value from the protocol field and the LocalStack library looks up the protocol metadata field. With the new changes, there is a protocols field that contains a list of protocols that the service supports, where a client will choose its most performant option. For boto3, this will resolve json with a jsonVersion of 1.0 (this client priority order can be seen here in botocore). For the AWS SDK for Java v2, this will resolve smithy-rpc-v2-cbor (this client priority order can be seen here).

Steps to reproduce:

  1. Create a virtual environment with the preview version of boto3 using the instructions and link in the CloudWatch Protocols FAQ page. When testing with the preview boto3 artifact, the version of botocore is v1.39.7. We used LocalStack version 4.7.1 for testing.
  2. Start LocalStack in docker mode with localstack start -d .
  3. Run any CloudWatch operation using the preview botocore against LocalStack endpoint.
  4. You will noticed the HTTP 500 errors with "Missing Action" messages.

Here are some of the references to fixes applied in LocalStack during SQS’ migration to JSON format to help address the gaps:

  • PR #9710 - Fix sqs multi-protocol handling and upgrade botocore ( also provides timelines for SQS changes)
  • PR #11726 - Add AWS json protocol support for SQS developer endpoints
  • PR #8268 - Fix SQS JSON protocol support in ASF
  • PR #10741 - Update ASF APIs, switch SQS to JSON again
  • PR #9634 - Fix SQS json requests sent to query route
  • PR #9607 - Fix SQS JSON leftovers

Thank you for maintaining tools that help AWS customers build and test their applications. We appreciate your partnership and look forward to working together to ensure a smooth migration. If you encounter issues or have questions during testing, please don't hesitate to reach out.

Resources

Metadata

Metadata

Labels

aws:cloudwatchAmazon CloudWatchstatus: in progressCurrently being worked ontype: featureNew feature, or improvement to an existing feature

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions