CFNV2: remove/rescope CFNV2:Validation errors #12970

simonrw · 2025-08-07T15:34:26Z

Motivation

At time of writing, the CFNV2:Validation errors were the biggest class of errors in the new provider's test suite. This PR attempts to tackle (or at least reassign them) to make the biggest push towards parity.

Since we have made the validator a subclass of the preprocessor, we can now raise ValidationErrors in the preprocessor which will be turned into proper service exceptions. This is the natural place to put this functionality, but it now can be driven during CreateChangeSet|CreateStack|UpdateStack i.e. synchronously

Changes

Hide most logged exceptions from the logs unless CFN_VERBOSE_ERRORS=1
Move the validation of the logical resource ids to the preprocessor
Improve parity of error messages with looking up missing keys from mappings
The validator now inherits from the preprocessor to support resolving mapping values
Fail to create stack if a stack already exists with that name and is active
Unskiped 4 tests
Reassigned 5 tests (to CFNV2:Transform, sorry @pinzon!)
Added 2 (passing) tests

TODO

Perhaps rethink the parameter resolution from CFnv2: implement SSM parameter resolving #12934 to use the preprocessor

github-actions · 2025-08-07T15:48:09Z

Test Results - Preflight, Unit

22 063 tests ±0 20 329 ✅ ±0 6m 25s ⏱️ +12s
1 suites ±0 1 734 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit d878dfd. ± Comparison against base commit 4464cd2.

♻️ This comment has been updated with latest results.

github-actions · 2025-08-07T16:00:32Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 18s ⏱️ +6s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit d878dfd. ± Comparison against base commit 4464cd2.

♻️ This comment has been updated with latest results.

github-actions · 2025-08-07T16:02:08Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 22m 22s ⏱️ - 1h 23m 22s
553 tests - 4 070 329 ✅ - 3 857 224 💤 - 213 0 ❌ ±0
555 runs - 4 070 329 ✅ - 3 857 226 💤 - 213 0 ❌ ±0

Results for commit d878dfd. ± Comparison against base commit 4464cd2.

This pull request removes 4072 and adds 2 tests. Note that renamed tests count towards both.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

tests.aws.services.cloudformation.engine.test_mappings.TestCloudFormationMappings ‑ test_async_mapping_error_first_level_v2
tests.aws.services.cloudformation.engine.test_mappings.TestCloudFormationMappings ‑ test_async_mapping_error_second_level_v2

This pull request removes 215 skipped tests and adds 2 skipped tests. Note that renamed tests count towards both.

tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input4-FAILED]
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_deployed_infra_state
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_populate_data
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_user_clicks_are_stored
tests.aws.services.apigateway.test_apigateway_api.TestApiGatewayApiRestApi ‑ test_get_api_case_insensitive
tests.aws.services.apigateway.test_apigateway_api.TestApiGatewayApiRestApi ‑ test_update_rest_api_concatenation_of_errors
tests.aws.services.apigateway.test_apigateway_api.TestApigatewayIntegration ‑ test_integration_response_invalid_responsetemplates
tests.aws.services.apigateway.test_apigateway_api.TestApigatewayIntegration ‑ test_put_integration_request_parameter_bool_type
tests.aws.services.apigateway.test_apigateway_basic.TestAPIGateway ‑ test_api_gateway_authorizer_crud
…

tests.aws.services.cloudformation.engine.test_mappings.TestCloudFormationMappings ‑ test_async_mapping_error_first_level_v2
tests.aws.services.cloudformation.engine.test_mappings.TestCloudFormationMappings ‑ test_async_mapping_error_second_level_v2

♻️ This comment has been updated with latest results.

github-actions · 2025-08-07T16:05:08Z

Test Results (amd64) - Integration, Bootstrap

5 files 5 suites 33m 23s ⏱️
577 tests 354 ✅ 223 💤 0 ❌
583 runs 354 ✅ 229 💤 0 ❌

Results for commit d878dfd.

♻️ This comment has been updated with latest results.

pinzon

LGTM. 👍
Just some minor comments.

pinzon · 2025-08-08T17:34:43Z

tests/aws/services/cloudformation/test_template_engine.py

It's progress 👍

pinzon · 2025-08-08T17:40:12Z

tests/aws/services/cloudformation/engine/test_mappings.py

        snapshot.match("mapping_nonexisting_key_exc", e.value.response)

-    @skip_if_v2_provider(reason="CFNV2:Validation")
+    @skip_if_v2_provider(reason="CFNV2:Validation replaced with v2 test below")


we could consider removing this test.

pinzon · 2025-08-08T17:45:51Z

localstack-core/localstack/services/cloudformation/engine/v2/change_set_model_executor.py

                    "Resource provider operation failed: '%s'",
                    reason,
-                    exc_info=LOG.isEnabledFor(logging.DEBUG),
+                    exc_info=LOG.isEnabledFor(logging.DEBUG) and config.CFN_VERBOSE_ERRORS,


is this Flag new? I don't see it in the docs

It's not in the docs, we planned it to be mostly for getting more info from support cases. I was getting confused by tracebacks in logs which ones mattered and which didn't.

pinzon · 2025-08-08T17:46:08Z

localstack-core/localstack/services/cloudformation/engine/v2/change_set_model_preproc.py

simonrw added 7 commits August 7, 2025 15:08

Handle stack already exists

f84f2c3

Handle update stack with same template

8ace993

Preprocessor can emit validation errors

81f143e

Move some validations to the preprocessor

ecfae26

Handle second level mapping keys

f3f8f60

Fix wording of missing resources

8356f3e

Rescope or remove validation tags

36596e9

simonrw added this to the 4.8 milestone Aug 7, 2025

simonrw added semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases review: merge when ready Signals to the reviewer that a PR can be merged if accepted labels Aug 7, 2025

Hide exceptions unless CFN_VERBOSE_ERRORS is passed

aaacd07

simonrw added 5 commits August 7, 2025 18:12

Add v1 skips

5569519

Fix visiting in validator

56fedb6

Fix top level key lookup

9101dac

Fix validator to not evaluate deployed resource

6c03e4c

Merge remote-tracking branch 'origin/main' into cfn/v2/validation-2

d878dfd

simonrw marked this pull request as ready for review August 8, 2025 04:38

simonrw requested review from dominikschubert and pinzon as code owners August 8, 2025 04:38

pinzon approved these changes Aug 8, 2025

View reviewed changes

simonrw merged commit 809d971 into main Aug 8, 2025
40 checks passed

simonrw deleted the cfn/v2/validation-2 branch August 8, 2025 18:44

simonrw mentioned this pull request Sep 5, 2025

CFNV2: Switch provider defaults to new engine #13098

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

CFNV2: remove/rescope CFNV2:Validation errors #12970

CFNV2: remove/rescope CFNV2:Validation errors #12970

Uh oh!

simonrw commented Aug 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

pinzon left a comment

Uh oh!

pinzon Aug 8, 2025

Uh oh!

pinzon Aug 8, 2025

Uh oh!

pinzon Aug 8, 2025

Uh oh!

simonrw Aug 8, 2025

Uh oh!

pinzon Aug 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

CFNV2: remove/rescope CFNV2:Validation errors #12970

CFNV2: remove/rescope CFNV2:Validation errors #12970

Uh oh!

Conversation

simonrw commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

TODO

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

pinzon left a comment

Choose a reason for hiding this comment

Uh oh!

pinzon Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

pinzon Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

pinzon Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

simonrw Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

pinzon Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

simonrw commented Aug 7, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading