ASF: implement RPC V2 CBOR parser and serializer #13125

bentsku · 2025-09-10T21:45:51Z

Motivation

This PR is part of a bigger initiative to implement support for the new Smithy Protocol smithy-rpc-v2-cbor, as well as the new Botocore multi-protocol support for single services (before, a service could only define one protocol).

This PR is stacked on top of #13103.

This PR properly implements the smithy-rpc-v2-cbor protocol and adds support for it.
Currently, no service that LocalStack supports implements this protocol, but we can still our unit tests to validate the serialization/deserialization works.

The logic is based on Botocore's own serializer and parser, with some adaptations due to the client/server difference.

One of the main feature of RPC v2-based protocol is its routing, following the /service/<service-name>/operation/<operation-name> format. This is implemented in our parser.

In the next steps, while implementing proper multi-protocol support, we will run integration tests with this protocol against AWS, to make sure our deserialization/serialization works end-to-end.

Because we do not support any service with smithy-rpc-v2-cbor yet, I have not implemented the Service detection yet, because we cannot test it. This will come in a follow-up when implementing the multi-protocol support (as it will also implement protocol detection, we will add rpc-v2-cbor service name parsing).

Changes

implements the BaseRpcV2RequestParser and BaseRpcV2ResponseSerializer
implement the child RpcV2CBORRequestParser and RpcV2CBORResponseSerializer which supports smithy-rpc-v2-cbor
add new unit tests to verify round trip parsing and serialization of requests/responses of service using this protocol
add unit test to verify operation detection (service detection of RPC v2 will come in a follow up)

github-actions · 2025-09-10T21:54:58Z

S3 Image Test Results (AMD64 / ARM64)

2 files ±0 2 suites ±0 7m 52s ⏱️ -22s
532 tests ±0 480 ✅ ±0 52 💤 ±0 0 ❌ ±0
1 064 runs ±0 960 ✅ ±0 104 💤 ±0 0 ❌ ±0

Results for commit 646b04c. ± Comparison against base commit 396f80e.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-10T21:57:43Z

Test Results - Preflight, Unit

22 160 tests +11 20 422 ✅ +11 6m 29s ⏱️ +5s
1 suites ± 0 1 738 💤 ± 0
1 files ± 0 0 ❌ ± 0

Results for commit 646b04c. ± Comparison against base commit 396f80e.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-10T22:06:43Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 23s ⏱️ +7s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 646b04c. ± Comparison against base commit 396f80e.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-10T22:50:28Z

Test Results (amd64) - Integration, Bootstrap

5 files 5 suites 2h 38m 38s ⏱️
5 046 tests 4 555 ✅ 491 💤 0 ❌
5 052 runs 4 555 ✅ 497 💤 0 ❌

Results for commit 646b04c.

♻️ This comment has been updated with latest results.

github-actions · 2025-09-10T23:04:46Z

LocalStack Community integration with Pro

2 files 2 suites 1h 58m 41s ⏱️
4 672 tests 4 341 ✅ 331 💤 0 ❌
4 674 runs 4 341 ✅ 333 💤 0 ❌

Results for commit 646b04c.

♻️ This comment has been updated with latest results.

alexrashed

Absolutely great! Another great step towards #13028 / to support CBOR for CloudWatch before the SDKs perform the switch! 💯

The next step / only step left would be to adjust the service router and the serializer / parser calls to support first-class multi-protocol support on handler-chain level! 🤩

I really want to stress how great this nice, clean, iterative approach here is! 🦸🏽

alexrashed · 2025-09-16T09:25:13Z

localstack-core/localstack/aws/spec.py


 ServiceName = str
-ProtocolName = Literal["query", "json", "rest-json", "rest-xml", "ec2"]
+ProtocolName = Literal["query", "json", "rest-json", "rest-xml", "ec2", "smithy-rpc-v2-cbor"]


praise: 🥳

alexrashed · 2025-09-16T09:28:07Z

localstack-core/localstack/aws/protocol/serializer.py

+    # TODO: do we want to add a check if the user-defined protocol is part of the available ones in the ServiceModel?
+    #  or should it be checked once
+    service_protocol = protocol or service.protocol


thought: I guess this here depends a bit on the question how strict or loose we want to be here. When thinking about it, I think the serializer should be loose (we want to avoid failing with the serialization, therefore using maybe another layer of fallback).
For the parser, I think this is different. We want to fail fast / be strict / only take the protocol determined in the service router, otherwise this could lead to weird parsing issues which are hard to debug.
What do you think?

Yes, I agree. I believe the protocol determination in the Service Name Parser will only be able to select the one defined in the service model, with a fallback to the default one of the service.

In this comment, I was wondering if we would need to add an additional check if the protocol is indeed again available in the service model. Technically, I don't think we could use a protocol not defined in the service model here, because the Service Name Parser would have properly detected it and can only return a protocol that is defined.

I suppose this check would only be enforced when we are manually using the parser and serializer, which can maybe still happen somewhere in the code. Having clear exceptions might be better here. It's just that this check is going to be "useful" in our normal usage I suppose.

As I need to address the unix timestamp issue in this PR, I'll update this comment to be clearer and will leave it to be addressed in the multi-protocol PRs, as this where things are coming come into place 👍

alexrashed · 2025-09-16T09:53:57Z

localstack-core/localstack/aws/protocol/serializer.py

+            #  think about a better way, this is very hacky
+            # Error responses in the rpcv2Cbor protocol MUST be serialized identically to standard responses with one
+            # additional component to distinguish which error is contained: a body field named __type.
+            shape_copy = copy.deepcopy(shape)


thought: Similar to this comment in #13103, I hope we might find a less expensive solution for this... 😅

alexrashed · 2025-09-16T09:57:44Z

localstack-core/localstack/aws/protocol/serializer.py

+
+    # the Smithy spec defines that only `application/cbor` is supported for RPC v2 CBOR
+    SUPPORTED_MIME_TYPES = [APPLICATION_CBOR]
+    # TODO: check the timestamp format for RpcV2CBOR, which might be different than Kinesis CBOR


thought: Based on this code comment in botocore, I think unixtimestamp should be just right.

Right, thanks for linking it! the comment right under is talking about milli seconds: https://github.com/boto/botocore/blob/28a90a45d7de077770665462ef8e1b53a9dd57ad/botocore/parsers.py#L898C10-L898C62

and the specs is also talking about milli seconds precision, and that we need to disregard the timestampFormat trait: https://smithy.io/2.0/additional-specs/protocols/smithy-rpc-v2.html#timestamp-type-serialization

As support for half-precision floating-point values is inconsistent across implementations, timestamp types SHOULD NOT serialize into a half-precision (16 bit) numeric CBOR value for the tagged value.

This protocol uses epoch-seconds, also known as Unix timestamps, with millisecond (1/1000th of a second) resolution. The timestampFormat MUST NOT be respected to customize timestamp serialization.

So they talk about Unix timestamps but with millisecond resolution, so this is probably wrong now that I see it 😅 I'll try to manually test it against AWS and fix that for this PR + unit test it

circling back: this is going be a little of a challenge, because ArcRegionSwitch does not define any InputShape with a date time... and is the only service supporting smithy-rpc-v2-cbor for now. We can get back to this once we add CloudWatch, I'll still check on the serializer side because it does return some timestamp data and define it in its output shapes 👍

alexrashed · 2025-09-16T09:59:16Z

localstack-core/localstack/aws/protocol/parser.py

+    request body.
+    """
+
+    # TODO: investigate datetime format for RpcV2CBOR protocol, which might be different than Kinesis CBOR


nit: Seems like a duplicate to a comment in the serializer, and I think unixtimestamp should just be fine for now?

bentsku

Thanks a lot for the review! I'll investigate the timestamp resolution and will update the PR 👍 I'll try to open the multi-protocol one later today 🚀

bentsku · 2025-09-16T12:27:36Z

localstack-core/localstack/aws/protocol/serializer.py

+
+    # the Smithy spec defines that only `application/cbor` is supported for RPC v2 CBOR
+    SUPPORTED_MIME_TYPES = [APPLICATION_CBOR]
+    # TODO: check the timestamp format for RpcV2CBOR, which might be different than Kinesis CBOR


Right, thanks for linking it! the comment right under is talking about milli seconds: https://github.com/boto/botocore/blob/28a90a45d7de077770665462ef8e1b53a9dd57ad/botocore/parsers.py#L898C10-L898C62

and the specs is also talking about milli seconds precision, and that we need to disregard the timestampFormat trait: https://smithy.io/2.0/additional-specs/protocols/smithy-rpc-v2.html#timestamp-type-serialization

As support for half-precision floating-point values is inconsistent across implementations, timestamp types SHOULD NOT serialize into a half-precision (16 bit) numeric CBOR value for the tagged value.

This protocol uses epoch-seconds, also known as Unix timestamps, with millisecond (1/1000th of a second) resolution. The timestampFormat MUST NOT be respected to customize timestamp serialization.

So they talk about Unix timestamps but with millisecond resolution, so this is probably wrong now that I see it 😅 I'll try to manually test it against AWS and fix that for this PR + unit test it

bentsku · 2025-09-16T12:33:36Z

localstack-core/localstack/aws/protocol/serializer.py

+    # TODO: do we want to add a check if the user-defined protocol is part of the available ones in the ServiceModel?
+    #  or should it be checked once
+    service_protocol = protocol or service.protocol


Yes, I agree. I believe the protocol determination in the Service Name Parser will only be able to select the one defined in the service model, with a fallback to the default one of the service.

In this comment, I was wondering if we would need to add an additional check if the protocol is indeed again available in the service model. Technically, I don't think we could use a protocol not defined in the service model here, because the Service Name Parser would have properly detected it and can only return a protocol that is defined.

I suppose this check would only be enforced when we are manually using the parser and serializer, which can maybe still happen somewhere in the code. Having clear exceptions might be better here. It's just that this check is going to be "useful" in our normal usage I suppose.

bentsku · 2025-09-16T12:34:39Z

localstack-core/localstack/aws/protocol/serializer.py

+    # TODO: do we want to add a check if the user-defined protocol is part of the available ones in the ServiceModel?
+    #  or should it be checked once
+    service_protocol = protocol or service.protocol


As I need to address the unix timestamp issue in this PR, I'll update this comment to be clearer and will leave it to be addressed in the multi-protocol PRs, as this where things are coming come into place 👍

bentsku · 2025-09-17T10:15:25Z

@alexrashed re-requested your review following the latest changes fixing timestamp handling and adding a different serialization format for data structures (finite vs indefinite maps & list).

Had to force-push after the merge, but the changes starts at 2ba257d

I think this would the proper diff of the changes: https://github.com/localstack/localstack/pull/13125/files/a02653974f9af7f5d9b178c4d204eafd278d7bab..646b04c8544e25894ad0837be801a7586d2e813e

Thanks!

alexrashed

Awesome! Huge kudos for this really great deep dive into the data structure length handling! The implementation is looking good, I'm curious if the fixed length mode will become necessary 😅

bentsku · 2025-09-17T10:41:32Z

I'm curious if the fixed length mode will become necessary 😅

Hopefully not 😄 it will be easy to remove once we're certain, as the work was already done, I kept it there. It feels like different service might implement different serializers on their side, but hopefully for us it won't be the case. 🤞

bentsku · 2025-09-24T19:37:37Z

localstack-core/localstack/aws/protocol/serializer.py

+        timestamp = float(self._convert_timestamp_to_str(value))
+        initial_byte = self._get_initial_byte(self.FLOAT_AND_SIMPLE_MAJOR_TYPE, 27)
+        serialized.extend(initial_byte + struct.pack(">d", timestamp))


we might want to revisit this for compatibility reasons: even if it looks like most clients should be able to deserialize this timestamp, we know Botocore handles it, and the Go SDK deserializer is able to deserialize float32, float64, cbor uint (within int64 bounds) and cbor -int (within int64 bounds) as mentioned here:

https://github.com/aws/smithy-go/blob/04e063a48301226f81d645e0b899f32a6455ef63/encoding/cbor/coerce.go#L145-L180

We'll need to check the AWS raw response to see what kind of resolution they return, if it depends on operation/service, and if we can have any kind of compatibility issue.

We might need to serialize to an int instead, maybe depending on the resolution of the serialized datetime. Hopefully, it is the same for every datetime.

bentsku added this to the 4.9 milestone Sep 10, 2025

bentsku self-assigned this Sep 10, 2025

bentsku added area: asf semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases docs: skip Pull request does not require documentation changes labels Sep 10, 2025

bentsku force-pushed the rpcv2-cbor-parser-serializer branch from 281b641 to b5db830 Compare September 10, 2025 21:47

bentsku force-pushed the cbor-parser-serializer branch from 9809182 to a2853d1 Compare September 15, 2025 21:36

bentsku force-pushed the rpcv2-cbor-parser-serializer branch from b5db830 to 8806c2c Compare September 15, 2025 21:36

bentsku mentioned this pull request Sep 15, 2025

ASF: implement CBOR parser and serializer #13103

Merged

alexrashed approved these changes Sep 16, 2025

View reviewed changes

bentsku commented Sep 16, 2025

View reviewed changes

bentsku requested a review from alexrashed September 16, 2025 16:08

bentsku mentioned this pull request Sep 16, 2025

ASF: implement multi-protocol support #13151

Merged

Base automatically changed from cbor-parser-serializer to main September 17, 2025 10:11

bentsku added 10 commits September 17, 2025 12:13

update the tests and parser/serializer to have protocol selection

3aa799e

wip add test

e006bee

implement RpcV2CBORRequestParser

cd98bf9

implement RpcV2CBOR parser and serializer + tests

22758aa

add protocol selection test

2125af1

improve diagrams and naming

e6fb9e0

add operation detection test

a026539

improve timestamp and indefinite length structure behavior

2ba257d

improve comment

dac2fe7

fix test to use fixed timezone

646b04c

bentsku force-pushed the rpcv2-cbor-parser-serializer branch from aef8ffa to 646b04c Compare September 17, 2025 10:13

alexrashed approved these changes Sep 17, 2025

View reviewed changes

bentsku marked this pull request as ready for review September 17, 2025 10:43

bentsku requested a review from thrau as a code owner September 17, 2025 10:43

bentsku merged commit c8f43c9 into main Sep 17, 2025
48 checks passed

bentsku deleted the rpcv2-cbor-parser-serializer branch September 17, 2025 11:52

alexrashed added the notes: skip Pull request does not have to be mentioned in the release notes label Sep 24, 2025

bentsku commented Sep 24, 2025

View reviewed changes

Uh oh!

ASF: implement RPC V2 CBOR parser and serializer #13125

ASF: implement RPC V2 CBOR parser and serializer #13125

Conversation

bentsku commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

github-actions bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

github-actions bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

github-actions bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bentsku left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bentsku commented Sep 17, 2025

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

bentsku commented Sep 17, 2025

Uh oh!

Uh oh!

bentsku Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bentsku commented Sep 10, 2025 •

edited

Loading

github-actions bot commented Sep 10, 2025 •

edited

Loading

github-actions bot commented Sep 10, 2025 •

edited

Loading

github-actions bot commented Sep 10, 2025 •

edited

Loading

github-actions bot commented Sep 10, 2025 •

edited

Loading

github-actions bot commented Sep 10, 2025 •

edited

Loading

bentsku Sep 24, 2025 •

edited

Loading