merge CFn V1 and CFnV2 tests #12944

pinzon · 2025-07-31T19:00:17Z

Motivation

This PR merges the new tests created exclusively for the new engine into the existing pre-existing folder. Also provides new test skip utility to use.

Co-Authored by @simonrw.

Changes

New confest.py to provide new utilitys.
Merging of ported_from_v1 tests with the originals

Testing

Green Pipeline

github-actions · 2025-07-31T22:07:49Z

Test Results (MA/MR) - Preflight, Unit

21 984 tests +4 20 250 ✅ +4 6m 19s ⏱️ -1s
1 suites ±0 1 734 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit d50c3d3. ± Comparison against base commit 4648b23.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-31T22:27:08Z

Test Results (amd64, MA/MR) - Acceptance

7 tests ±0 5 ✅ ±0 3m 12s ⏱️ +4s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit d50c3d3. ± Comparison against base commit 4648b23.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-31T22:27:14Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 7s ⏱️ -13s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit d50c3d3. ± Comparison against base commit 4648b23.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-31T22:57:48Z

Test Results (amd64, MA/MR) - Integration, Bootstrap

5 files ± 0 5 suites ±0 2h 20m 59s ⏱️ +17s
4 945 tests +94 4 383 ✅ +13 562 💤 +81 0 ❌ ±0
4 951 runs +94 4 383 ✅ +13 568 💤 +81 0 ❌ ±0

Results for commit d50c3d3. ± Comparison against base commit 4648b23.

This pull request removes 9 and adds 103 tests. Note that renamed tests count towards both.

tests.aws.services.cloudformation.api.test_stacks.TestStacksApi ‑ test_list_stack_resources_for_removed_resource
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2 ‑ test_deploy_instance_with_key_pair
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2 ‑ test_deploy_prefix_list
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2 ‑ test_deploy_security_group_with_tags
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2 ‑ test_deploy_vpc_endpoint
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestBasicCRD ‑ test_autogenerated_values
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestBasicCRD ‑ test_black_box
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestBasicCRD ‑ test_getatt
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestUpdates ‑ test_update_without_replacement

tests.aws.services.cloudformation.api.test_resources ‑ test_describe_non_existent_resource
tests.aws.services.cloudformation.api.test_resources ‑ test_describe_non_existent_stack
tests.aws.services.cloudformation.api.test_resources ‑ test_invalid_logical_resource_id
tests.aws.services.cloudformation.api.test_stacks ‑ test_no_parameters_given
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2_resource_provider ‑ test_deploy_instance_with_key_pair
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2_resource_provider ‑ test_deploy_prefix_list
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2_resource_provider ‑ test_deploy_security_group_with_tags
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2_resource_provider ‑ test_deploy_vpc_endpoint
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic_user.TestBasicCRD ‑ test_autogenerated_values
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic_user.TestBasicCRD ‑ test_black_box
…

This pull request removes 1 skipped test and adds 85 skipped tests. Note that renamed tests count towards both.

tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestUpdates ‑ test_update_without_replacement

tests.aws.services.cloudformation.api.test_resources ‑ test_describe_non_existent_stack
tests.aws.services.cloudformation.api.test_resources ‑ test_invalid_logical_resource_id
tests.aws.services.cloudformation.api.test_stacks ‑ test_no_parameters_given
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic_user.TestUpdates ‑ test_update_without_replacement
tests.aws.services.cloudformation.resources.test_stack_sets ‑ test_delete_nonexistent_stack_set
tests.aws.services.cloudformation.resources.test_stack_sets ‑ test_fetch_non_existent_stack_set_instances
tests.aws.services.cloudformation.test_change_set_conditions.TestChangeSetConditions ‑ test_condition_add_new_negative_condition_to_existent_resource
tests.aws.services.cloudformation.test_change_set_conditions.TestChangeSetConditions ‑ test_condition_add_new_positive_condition_to_existent_resource
tests.aws.services.cloudformation.test_change_set_conditions.TestChangeSetConditions ‑ test_condition_update_adds_resource
tests.aws.services.cloudformation.test_change_set_conditions.TestChangeSetConditions ‑ test_condition_update_removes_resource
…

♻️ This comment has been updated with latest results.

github-actions · 2025-07-31T22:58:15Z

Test Results (amd64) - Integration, Bootstrap

5 files ± 0 5 suites ±0 2h 19m 43s ⏱️ -57s
4 945 tests +94 4 383 ✅ +13 562 💤 +81 0 ❌ ±0
4 951 runs +94 4 383 ✅ +13 568 💤 +81 0 ❌ ±0

Results for commit d50c3d3. ± Comparison against base commit 4648b23.

This pull request removes 9 and adds 103 tests. Note that renamed tests count towards both.

tests.aws.services.cloudformation.api.test_stacks.TestStacksApi ‑ test_list_stack_resources_for_removed_resource
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2 ‑ test_deploy_instance_with_key_pair
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2 ‑ test_deploy_prefix_list
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2 ‑ test_deploy_security_group_with_tags
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2 ‑ test_deploy_vpc_endpoint
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestBasicCRD ‑ test_autogenerated_values
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestBasicCRD ‑ test_black_box
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestBasicCRD ‑ test_getatt
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestUpdates ‑ test_update_without_replacement

tests.aws.services.cloudformation.api.test_resources ‑ test_describe_non_existent_resource
tests.aws.services.cloudformation.api.test_resources ‑ test_describe_non_existent_stack
tests.aws.services.cloudformation.api.test_resources ‑ test_invalid_logical_resource_id
tests.aws.services.cloudformation.api.test_stacks ‑ test_no_parameters_given
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2_resource_provider ‑ test_deploy_instance_with_key_pair
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2_resource_provider ‑ test_deploy_prefix_list
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2_resource_provider ‑ test_deploy_security_group_with_tags
tests.aws.services.cloudformation.resource_providers.ec2.test_ec2_resource_provider ‑ test_deploy_vpc_endpoint
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic_user.TestBasicCRD ‑ test_autogenerated_values
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic_user.TestBasicCRD ‑ test_black_box
…

This pull request removes 1 skipped test and adds 85 skipped tests. Note that renamed tests count towards both.

tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic.TestUpdates ‑ test_update_without_replacement

tests.aws.services.cloudformation.api.test_resources ‑ test_describe_non_existent_stack
tests.aws.services.cloudformation.api.test_resources ‑ test_invalid_logical_resource_id
tests.aws.services.cloudformation.api.test_stacks ‑ test_no_parameters_given
tests.aws.services.cloudformation.resource_providers.iam.aws_iam_user.test_basic_user.TestUpdates ‑ test_update_without_replacement
tests.aws.services.cloudformation.resources.test_stack_sets ‑ test_delete_nonexistent_stack_set
tests.aws.services.cloudformation.resources.test_stack_sets ‑ test_fetch_non_existent_stack_set_instances
tests.aws.services.cloudformation.test_change_set_conditions.TestChangeSetConditions ‑ test_condition_add_new_negative_condition_to_existent_resource
tests.aws.services.cloudformation.test_change_set_conditions.TestChangeSetConditions ‑ test_condition_add_new_positive_condition_to_existent_resource
tests.aws.services.cloudformation.test_change_set_conditions.TestChangeSetConditions ‑ test_condition_update_adds_resource
tests.aws.services.cloudformation.test_change_set_conditions.TestChangeSetConditions ‑ test_condition_update_removes_resource
…

♻️ This comment has been updated with latest results.

simonrw · 2025-08-01T11:03:33Z

.github/workflows/aws-tests.yml


      - name: Run CloudFormation Engine v2 Tests
-        timeout-minutes: 30
+        timeout-minutes: 60


Bumping this up because I have noticed the v2 test suite alone takes nearly 30 minutes already.

simonrw and others added 30 commits July 31, 2025 16:23

Add helpers to conftest.py

ae7019f

Move test_changesets.py

e3f97d8

move drift detection

0ed2bfc

Move test_nested_stacks

7b211b0

move test_extensions_api

d5bc5bd

move test_extensions_hooks

8403635

Move test_reference_resolving

b6f13f2

move test_extensions_modules

bec0a01

move test_resources

4f1b3ee

move test_stack_policies

4f46e04

move test_templates.py

7839b91

Move test_stacks.py

3a3dfcc

Move test_change_set_conditions.py

a6a331a

Move test_change_set_depends_on

4c8aff9

Move test_change_set_fn_base64

37246e9

Move test_change_set_fn_get_attr

0c0b261

move test_transformers

927764b

move test_extensions_resourcetypes

aa3cffb

move and update test_update_stack

846c9c3

move test_attributes.py

9bf5c9c

move test_mappings.py

b1508b6

move and update test_apigw

34cd1ff

move test_cloudformation

c4309e1

move test_cloudwatch

58f039f

move and updated test_events

08124fa

move test_integration

4a8bcc9

move test_kms

df73869

move test_opensearch

58f83f3

move test_s3

7ad1710

move test_redshift

d68ae8f

simonrw added 6 commits July 31, 2025 22:29

Remove test_ec2

915c364

Remove firehose

a0ba386

Remove v2 dir

4051f32

Minor tidying up

76bd03e

Deduplicate paths to allow test collection

84cf214

Update CI configuration

0b3ffa4

Stupid formatting errors

04915b9

simonrw added 7 commits August 1, 2025 10:04

Add v1 skips

1befd9d

Fix incorrect skips

d58e99c

Skip iam test

130c10a

Clean up failing v2 tests

1584631

Add unit test for optional_list

e49ad80

Reset formatting of test configuration

d88ecc1

fixup! Add unit test for optional_list

d50c3d3

simonrw marked this pull request as ready for review August 1, 2025 11:14

simonrw requested review from alexrashed, dfangl, dominikschubert, k-a-il, silv-io and simonrw as code owners August 1, 2025 11:14

simonrw reviewed Aug 1, 2025

View reviewed changes

simonrw approved these changes Aug 1, 2025

View reviewed changes

simonrw merged commit f5d850d into main Aug 1, 2025
61 checks passed

simonrw deleted the cfn/v2/merge-tests branch August 1, 2025 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

merge CFn V1 and CFnV2 tests #12944

merge CFn V1 and CFnV2 tests #12944

Uh oh!

pinzon commented Jul 31, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

simonrw Aug 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

merge CFn V1 and CFnV2 tests #12944

merge CFn V1 and CFnV2 tests #12944

Uh oh!

Conversation

pinzon commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (MA/MR) - Preflight, Unit

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64, MA/MR) - Acceptance

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64, MA/MR) - Integration, Bootstrap

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

simonrw Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pinzon commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading