Skip to content

Conversation

@MEPalma
Copy link
Contributor

@MEPalma MEPalma commented Apr 17, 2025

Motivation

The current CloudFormation v2 engine has limitations in handling Ref intrinsic functions and does not populate PhysicalResourceId values during describe operations (e.g., DescribeChangeSet). The existing implementation is constrained to resource types only, and the logic for computing Ref values, both before and after changes, is complex and difficult to extend.
This PR introduces a redesigned approach to computing Ref intrinsic functions that generalizes the pattern beyond AWS::Resource types, making the engine more extensible and less convoluted in this area. It also adds support in the describer for including PhysicalResourceId values in describe steps.

Changes

  • Resource preprocessor objects carry PhysicalResourceId
  • Describer lists PhysicalResourceId values
  • Reworked the PreProc's design pattern for evaluating Ref intrinsic functions
  • Reflected PreProc design changes in the executor and describer
  • Removed PhysicalResourceId snapshot verify skips
  • Removed deprecated unit tests

@MEPalma MEPalma added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Apr 17, 2025
@MEPalma MEPalma added this to the 4.4 milestone Apr 17, 2025
@MEPalma MEPalma self-assigned this Apr 17, 2025
@github-actions
Copy link

github-actions bot commented Apr 17, 2025

LocalStack Community integration with Pro

  2 files  ±    0    2 suites  ±0   20m 36s ⏱️ - 1h 35m 50s
463 tests  - 3 910  314 ✅  - 3 702  149 💤  - 208  0 ❌ ±0 
465 runs   - 3 910  314 ✅  - 3 702  151 💤  - 208  0 ❌ ±0 

Results for commit 0587ff3. ± Comparison against base commit 96a0216.

This pull request removes 3910 tests.
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

Copy link
Contributor

@simonrw simonrw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general I'm ok with this but I have a couple of questions

Comment on lines 76 to 82
after_resource = delta.after
if after_resource is not None and delta.before != delta.after:
after_logical_id = after_resource.logical_id
after_physical_id = self._current_resource_physical_id(
resource_logical_id=after_logical_id
)
after_resource.physical_resource_id = after_physical_id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mutating the delta object right? I'm not keen on mutation - part of the reason the old engine was frustrating to work with was because of the mutation.

Copy link
Contributor Author

@MEPalma MEPalma Apr 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we are overriding the logic to compose the delta for resources and mutating the delta before it is returned upstream. I understand mutations can become tedious though. To avoid mutations I would probably mark the members of these deltas as finals, and then compute the references on demand rather than binding it to the resource delta. I moved away from this as it meant that downstream from the preproc are responsible for choosing where to sample this physical id from given the logical id (before and after). But it's definitely a good alternative to consider.

I think I am happy with this as we mutate the delta for a resource type in the one function that computes it; and it's also true that all the logics upstream from this point only need this entity as read-only (the resource was executed and others may depend on its computed values).

Can I propose I attempt the previous design again in a follow-up branch for us to reevaluate?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After discussion we think this approach is ok as the mutation is really hidden from the caller, and is an implementation detail. A low effort alternative would be to treat delta objects as immutable and construct new ones but at this stage this change is not critical.

after_physical_id = self._after_resource_physical_id(resource_logical_id)
if before_physical_id is None and after_physical_id is None:
raise RuntimeError(f"No PhysicalResourceId for '{resource_logical_id}'")
current_physical_id = after_physical_id or before_physical_id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry that this might lead to us incorrectly getting the physical resource id for a resource that has been deleted. Should this not enforce that the physical id is of a resource in the current stack? What is the use case for getting the physical resource of an object that is no longer in the stack?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should not happen. In a delta, deleted resources are represented as the current stack value in the before , and null values in the after. Therefore, we should never be able to sample the a physical id to assign it to the after of a resource delta, as the after of a delete resource delta does not need computing: it is null, doesn't exists (also in l77).
The executor may still record a duplicate of the deleted resource in the resources, which may be useful for the provider after execution (I'm not sure about this atm). Regardless, this is also fine for two reasons as far I can see as: (1) before definitions of resources are sampled strictly from before stores of resolved resources (as they should), and (2) even in the incorrect scenario that these are sampled from the 'current' or 'most updated value' (_current_resource_physical_id), the physical id can only be fore one for the resource being deleted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve excluded the physical ID from the delta’s comparison logic and now explicitly sample the physical resource ID from the after object of updated resources where necessary. This should make the intent behind which version of the physical ID is being used more clear in the code.

class ChangeSetModelExecutor(ChangeSetModelPreproc):
change_set: Final[ChangeSet]
# TODO: add typing.
_change_set: Final[ChangeSet]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm personally not too fond of "hiding" these properties. What is the motivation here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here (and in general) I make member variables private when they exist solely to support the class’s (or extensions) internal logic, and exposing them externally has no semantic purpose. By restricting visibility this way, I see it helps maintain clear abstraction boundaries it is in an effort to adhere to standard practices in this context where sensible. That being said, I'd be happy to revise this if you feel it's too different from the rest of this area of code, or it is simply needed

@MEPalma MEPalma merged commit 10bc97a into master Apr 23, 2025
31 checks passed
@MEPalma MEPalma deleted the MEP-CFN-physical_resource_id branch April 23, 2025 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants