Update deprecated type annotations produced by AWS scaffold tool #13347

purcell · 2025-11-06T11:45:54Z

Motivation

As of Python 3.9, common type annotations have changed, and the old ones (e.g. List, Optional, Union) generate deprecation warnings in some tools (pyright in my case). Our minimum Python version is 3.11, so it seems worthwhile to avoid those warnings, and I'm filing this PR as part of familiarising myself with the codebase.

Changes

This change updates the generation of scaffolded types to use the new names, and also adds default values in some cases, where further warnings were previously reported.

As a first step, the "logs" API has been regenerated, so that concrete changes can be inspected. It's not clear to me whether it is desirable or safe to actually update any generated modules yet, though.

Tests

Open question: would it be best to allow later gradual regeneration of scaffolds?

S3 Image Test Results (AMD64 / ARM64)

2 files 2 suites 7m 48s ⏱️
533 tests 481 ✅ 52 💤 0 ❌
1 066 runs 962 ✅ 104 💤 0 ❌

Results for commit df9a078.

♻️ This comment has been updated with latest results.

github-actions · 2025-11-06T12:05:17Z

Test Results - Preflight, Unit

22 259 tests - 23 20 507 ✅ - 23 16m 12s ⏱️ -23s
1 suites ± 0 1 752 💤 ± 0
1 files ± 0 0 ❌ ± 0

Results for commit 47cf9af. ± Comparison against base commit 447dd8e.

This pull request removes 23 tests.

tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-copy_volumes]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-create_capacity_manager_data_export]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-create_ipam_prefix_list_resolver]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-create_ipam_prefix_list_resolver_target]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-delete_capacity_manager_data_export]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-delete_ipam_prefix_list_resolver]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-delete_ipam_prefix_list_resolver_target]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-describe_capacity_manager_data_exports]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-describe_capacity_reservation_topology]
tests.unit.aws.api.test_asf_providers ‑ test_provider_signatures[Ec2Provider-Ec2Api-describe_ipam_prefix_list_resolver_targets]
…

♻️ This comment has been updated with latest results.

purcell · 2025-11-06T12:35:47Z

But would it also make sense to run through all the APIs and regenerate them? Or is there a risk I'm not seeing? It doesn't seem like things are often regenerated.

github-actions · 2025-11-06T13:01:37Z

LocalStack Community integration with Pro

2 files ±0 2 suites ±0 2h 1m 3s ⏱️ +2s
4 894 tests ±0 4 525 ✅ ±0 369 💤 ±0 0 ❌ ±0
4 896 runs ±0 4 525 ✅ ±0 371 💤 ±0 0 ❌ ±0

Results for commit 47cf9af. ± Comparison against base commit 447dd8e.

♻️ This comment has been updated with latest results.

alexrashed · 2025-11-06T13:10:59Z

But would it also make sense to run through all the APIs and regenerate them? Or is there a risk I'm not seeing? It doesn't seem like things are often regenerated.

That would definitely make sense! And in fact, they are regenerated every week by our "ASF Update Action", which results in PRs like this: #13283
You don't see a lot of updates though, because compared to the amount of operations there aren't many updates usually within a week and we really don't often update the actual code generation.
So from my perspective this change would be great! For the specific implementation right now however, it seems like there's still an issue as a unit test for the scaffold itself is failing on a compile error of some of the generated APIs.
Let me know if we should take a look at this together, happy to help!

purcell · 2025-11-06T16:49:59Z

Thand @alexrashed. Yeah, I'm looking at the compile errors.

purcell · 2025-11-06T17:56:44Z

Fixed now, and I also regenerated the module for S3, which was involved in the failure above.

Looking at the failures, there are interesting cases in the generated code that were possibly not correct even before this change. For example, in the s3 API:

Location = str

...

class PostResponse(TypedDict, total=False):
    StatusCode: None | GetObjectResponseStatusCode
    Location: None | Location
    LocationHeader: None | Location     # Here pyright sees a reference to PostResponse.Location, not the top-level Location type alias
    Bucket: None | BucketName
    Key: None | ObjectKey
    Expiration: None | Expiration
    ETag: None | ETag
    ETagHeader: None | ETag
    ChecksumCRC32: None | ChecksumCRC32
    ChecksumCRC32C: None | ChecksumCRC32C
    ChecksumCRC64NVME: None | ChecksumCRC64NVME
    ChecksumSHA1: None | ChecksumSHA1
    ChecksumSHA256: None | ChecksumSHA256
    ChecksumType: None | ChecksumType

purcell · 2025-11-06T18:09:43Z

If this looks acceptable, I expect the best plan would be to take out the generated changes from this commit, and merge only the scaffold generator changes, so that the next auto-update would cleanly update everything together.

github-actions · 2025-11-06T18:19:27Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 21s ⏱️ -1s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 47cf9af. ± Comparison against base commit 447dd8e.

♻️ This comment has been updated with latest results.

github-actions · 2025-11-06T18:55:35Z

Test Results (amd64) - Integration, Bootstrap

5 files ±0 5 suites ±0 2h 36m 30s ⏱️ - 2m 53s
5 268 tests ±0 4 739 ✅ ±0 529 💤 ±0 0 ❌ ±0
5 274 runs ±0 4 739 ✅ ±0 535 💤 ±0 0 ❌ ±0

Results for commit 47cf9af. ± Comparison against base commit 447dd8e.

♻️ This comment has been updated with latest results.

alexrashed

This is looking great! I guess the failing tests are not associated to the changes here, and will most likely be fixed by a rebase.
Concerning the approach, I don't have a strong opinion, but basically we can (a) either combine the update with the changes here maybe in separate commits (+ create a separate PR in our downstream repo to update the generated code there as well), or (b) we just merge the changes to the code generation and let the automation update all the code.
In guess (a) is a bit cleaner because then the API changes aren't mixed with the updated code generation and we have an isolated CI execution for the changes in the generated code, but (b) is a bit simpler to execute on.
I'd just leave that to you! 😃

localstack-core/localstack/aws/scaffold.py

As of Python 3.9, common type annotations have changed, and the old ones (e.g. List, Optional, Union) generate deprecation warnings in some tools (pyright in my case). This change updates the generation of scaffolded types to use the new names, and also adds default values in some cases, where further warnings were previously reported.

purcell · 2025-11-07T10:09:25Z

I've fixed that nit, and separated the change into two commits: one for the generator change, and the other for the API regeneration. If the tests run through fine, I'll drop the latter and we can just merge the former, waiting for the weekly job to cleanly regenerate everything.

purcell · 2025-11-07T11:25:25Z

Tests are good. I'll remove the generated API update commit, and mark this ready for review.

alexrashed

😛

giograno · 2025-11-07T17:45:59Z

Awesome 🎉 We even might simplify some ruff settings we added with py-upgrade.

purcell · 2025-11-07T18:21:55Z

Yes, it looks like all those exclusion codes relate to annotations, so they shouldn't be necessary now.

As of #13347, the type annotations in those modules are modern and will not upset ruff.

purcell · 2025-11-07T18:39:20Z

Submitted a fix in #13355.

As of #13347, the type annotations in those modules are modern and will not upset ruff.

purcell added semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes labels Nov 6, 2025

purcell force-pushed the updated-scaffold-types branch from d1e6ce7 to 3b7a3a8 Compare November 6, 2025 14:19

purcell force-pushed the updated-scaffold-types branch from 3b7a3a8 to c41dd71 Compare November 6, 2025 17:50

alexrashed approved these changes Nov 7, 2025

View reviewed changes

localstack-core/localstack/aws/scaffold.py Outdated Show resolved Hide resolved

purcell force-pushed the updated-scaffold-types branch from c41dd71 to 47cf9af Compare November 7, 2025 10:06

purcell force-pushed the updated-scaffold-types branch from 47cf9af to df9a078 Compare November 7, 2025 11:25

purcell marked this pull request as ready for review November 7, 2025 11:26

purcell requested a review from thrau as a code owner November 7, 2025 11:26

alexrashed approved these changes Nov 7, 2025

View reviewed changes

purcell merged commit 37b23d0 into main Nov 7, 2025
22 checks passed

purcell deleted the updated-scaffold-types branch November 7, 2025 11:40

purcell added a commit that referenced this pull request Nov 7, 2025

Remove redundant ruff ignores for generated ASF modules

bc85328

As of #13347, the type annotations in those modules are modern and will not upset ruff.

purcell mentioned this pull request Nov 7, 2025

Remove redundant ruff ignores for generated ASF modules #13355

Merged

alexrashed mentioned this pull request Nov 10, 2025

Update ASF APIs #13359

Merged

purcell added a commit that referenced this pull request Nov 10, 2025

Remove redundant ruff ignores for generated ASF modules

d0fcd74

As of #13347, the type annotations in those modules are modern and will not upset ruff.

Uh oh!

Update deprecated type annotations produced by AWS scaffold tool #13347

Update deprecated type annotations produced by AWS scaffold tool #13347

Uh oh!

Conversation

purcell commented Nov 6, 2025

Motivation

Changes

Tests

Related

Uh oh!

purcell commented Nov 6, 2025

Uh oh!

thrau commented Nov 6, 2025

Uh oh!

github-actions bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

github-actions bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

purcell commented Nov 6, 2025

Uh oh!

github-actions bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

alexrashed commented Nov 6, 2025

Uh oh!

purcell commented Nov 6, 2025

Uh oh!

purcell commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

purcell commented Nov 6, 2025

Uh oh!

github-actions bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

purcell commented Nov 7, 2025

Uh oh!

purcell commented Nov 7, 2025

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

giograno commented Nov 7, 2025

Uh oh!

purcell commented Nov 7, 2025

Uh oh!

purcell commented Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions bot commented Nov 6, 2025 •

edited

Loading

github-actions bot commented Nov 6, 2025 •

edited

Loading

github-actions bot commented Nov 6, 2025 •

edited

Loading

purcell commented Nov 6, 2025 •

edited

Loading

github-actions bot commented Nov 6, 2025 •

edited

Loading

github-actions bot commented Nov 6, 2025 •

edited

Loading