bug: CloudFormation dynamic reference regression 4.7 to 4.8 #13169
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
I am creating a stack with the following CloudFormation template:
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Parameters:
Profile:
Type: String
Description: The stage to publish on deploy
AllowedValues:
- prod
- stag
- test
LambdaAlias:
Type: String
Description: The Lambda function alias to publish on deploy
AllowedValues:
- live
- latest
BuildId:
Type: String
Description: The build identifier
AllowedPattern: "^[0-9A-Za-z]+$"
CognitoUserPoolId:
Type: String
Description: The Cognito User Pool ID to use for the post confirmation hook
AllowedPattern: '^[a-z]{2}-[a-z]+-\d+_[A-Za-z0-9]{8,}$'
Bump:
Type: String
Description: An arbitrary parameter to force CloudFormation to update the stack
Default: "0"
AllowedPattern: "^[0-9A-Za-z]+$"
Conditions:
IsTesting: !Equals [ !Ref Profile, "test" ]
Mappings:
LambdaConfiguration:
prod:
StripeSecretName: foobar/prod/stripe
ApiKeySigningSecretName: /foobar/prod/signing
FreeAwsPlanParameterName: /foobar/prod/plans/free/aws
SmallAwsPlanParameterName: /foobar/prod/plans/small/aws
MediumAwsPlanParameterName: /foobar/prod/plans/medium/aws
LargeAwsPlanParameterName: /foobar/prod/plans/large/aws
stag:
StripeSecretName: foobar/stag/stripe
ApiKeySigningSecretName: /foobar/stag/signing
FreeAwsPlanParameterName: /foobar/stag/plans/free/aws
SmallAwsPlanParameterName: /foobar/stag/plans/small/aws
MediumAwsPlanParameterName: /foobar/stag/plans/medium/aws
LargeAwsPlanParameterName: /foobar/stag/plans/large/aws
test:
StripeSecretName: foobar/test/stripe
ApiKeySigningSecretName: /foobar/test/signing
FreeAwsPlanParameterName: /foobar/test/plans/free/aws
SmallAwsPlanParameterName: /foobar/test/plans/small/aws
MediumAwsPlanParameterName: /foobar/test/plans/medium/aws
LargeAwsPlanParameterName: /foobar/test/plans/large/aws
Resources:
# NOTE: This Lambda Function requires some clickops.
#
# You must create the following Secrets Manager secrets manually:
# - foobar/prod/stripe
# - foobar/stag/stripe
#
# You must create the following SSM Parameters manually:
# - /foobar/prod/plans/free/aws
# - /foobar/prod/plans/small/aws
# - /foobar/prod/plans/medium/aws
# - /foobar/prod/plans/large/aws
# - /foobar/stag/plans/free/aws
# - /foobar/stag/plans/small/aws
# - /foobar/stag/plans/medium/aws
# - /foobar/stag/plans/large/aws
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
Architectures:
- x86_64
AutoPublishAlias: !Ref LambdaAlias
# pragma package lambda-cognito-post-confirmation-hook.jar
CodeUri: "target/lambda-cognito-post-confirmation-hook.jar"
Description: Prepares new cognito accounts for Stripe usage
Environment:
Variables:
PROFILE: !Ref Profile
API_KEY_SIGNING_SECRET_NAME: !FindInMap [LambdaConfiguration, !Ref Profile, ApiKeySigningSecretName]
STRIPE_SECRET_NAME: !FindInMap [LambdaConfiguration, !Ref Profile, StripeSecretName]
COGNITO_USER_POOL_ID: !Ref CognitoUserPoolId
AWS_FREE_USAGE_PLAN_ID:
Fn::Sub:
- "{{resolve:ssm:${ParameterName}}}"
- ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, FreeAwsPlanParameterName]
AWS_SMALL_USAGE_PLAN_ID:
Fn::Sub:
- "{{resolve:ssm:${ParameterName}}}"
- ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, SmallAwsPlanParameterName]
AWS_MEDIUM_USAGE_PLAN_ID:
Fn::Sub:
- "{{resolve:ssm:${ParameterName}}}"
- ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, MediumAwsPlanParameterName]
AWS_LARGE_USAGE_PLAN_ID:
Fn::Sub:
- "{{resolve:ssm:${ParameterName}}}"
- ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, LargeAwsPlanParameterName]
BUMP: !Ref Bump
FunctionName: !Sub "foobar-${Profile}-stripe-cognito-post-confirmation-hook"
Handler: io.foobar.portal.backend.lambda.cognito.hook.postconfirmation.LambdaFunction
MemorySize: 4096
PackageType: Zip
Policies:
- AWSLambdaBasicExecutionRole
- AWSXRayDaemonWriteAccess
- Statement:
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
Resource:
- Fn::Sub:
- "arn:aws:secretsmanager:${Region}:${AccountId}:secret:${SecretName}-*"
- Region: !Ref AWS::Region
AccountId: !Ref AWS::AccountId
SecretName: !FindInMap [LambdaConfiguration, !Ref Profile, StripeSecretName]
- Fn::Sub:
- "arn:aws:secretsmanager:${Region}:${AccountId}:secret:${SecretName}-*"
- Region: !Ref AWS::Region
AccountId: !Ref AWS::AccountId
SecretName: !FindInMap [LambdaConfiguration, !Ref Profile, ApiKeySigningSecretName]
- Statement:
- Effect: Allow
Action:
- ssm:GetParameter
Resource:
- !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/foobar/${Profile}/*"
- Statement:
- Effect: Allow
Action:
- cognito-idp:AdminGetUser
- cognito-idp:AdminUpdateUserAttributes
Resource:
- !Sub "arn:aws:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${CognitoUserPoolId}"
- Statement:
# We need to manage API keys
- Effect: Allow
Action:
# List all API keys
- apigateway:GET
# Create new API keys
- apigateway:POST
Resource:
- !Sub "arn:aws:apigateway:${AWS::Region}::/apikeys"
# We need to create API keys for the free usage plan
- Effect: Allow
Action:
- apigateway:POST
Resource:
- Fn::Sub:
- "arn:aws:apigateway:${Region}::/usageplans/{{resolve:ssm:${ParameterName}}}/keys"
- Region: !Ref AWS::Region
ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, FreeAwsPlanParameterName]
# We need to delete API keys for all usage plans, just in case
- Effect: Allow
Action:
- apigateway:DELETE
Resource:
- Fn::Sub:
- "arn:aws:apigateway:${Region}::/usageplans/{{resolve:ssm:${ParameterName}}}/keys/*"
- Region: !Ref AWS::Region
ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, FreeAwsPlanParameterName]
- Fn::Sub:
- "arn:aws:apigateway:${Region}::/usageplans/{{resolve:ssm:${ParameterName}}}/keys/*"
- Region: !Ref AWS::Region
ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, SmallAwsPlanParameterName]
- Fn::Sub:
- "arn:aws:apigateway:${Region}::/usageplans/{{resolve:ssm:${ParameterName}}}/keys/*"
- Region: !Ref AWS::Region
ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, MediumAwsPlanParameterName]
- Fn::Sub:
- "arn:aws:apigateway:${Region}::/usageplans/{{resolve:ssm:${ParameterName}}}/keys/*"
- Region: !Ref AWS::Region
ParameterName: !FindInMap [LambdaConfiguration, !Ref Profile, LargeAwsPlanParameterName]
Runtime: java21
RuntimeManagementConfig:
UpdateRuntimeOn: FunctionUpdate
# This lambda function has no sophisticated state, so TURN OFF SnapStart!
# SnapStart:
# ApplyOn: PublishedVersions
Tags:
"aleph0": "true"
"aleph0:scope": !Ref Profile
"aleph0:product": foobar
Timeout: 30
Tracing: !If [IsTesting, !Ref AWS::NoValue, "Active"]
VersionDescription: !Ref BuildId
Outputs:
LambdaFunction:
Value: !GetAtt LambdaFunction.Arn
Description: The ARN of the Stripe Cognito Post Confirmation Hook Lambda function
Export:
Name: !Sub "foobar-${Profile}-stripe-cognito-post-confirmation-hook"Expected Behavior
Using LocalStack v4.7, the stack creates as expected. Using LocalStack v4.8, the create fails. Oddly, after I wait for the stack's status to settle, the stack reports status CREATE_COMPLETE. However, the following events appear in the event history, sorted by timestamp descending, from my logs:
[22:59:58.738] [ERROR] [main] LambdaFunctionITBase - Stack in state CREATE_COMPLETE, but at least one resource actually failed to create: LambdaFunction = An error occurred (ParameterNotFound) when calling the GetParameter operation: Parameter ${ParameterName} not found.
[22:59:58.748] [ERROR] [main] LambdaFunctionITBase - Stack event: 7bee3577-8f4b-4df6-bfe4-6ce980864900 2025-09-19T03:59:57.671973Z my-test-stack AWS::CloudFormation::Stack CREATE_COMPLETE null
[22:59:58.748] [ERROR] [main] LambdaFunctionITBase - Stack event: 58d9b7ef-1576-4ab9-bedc-af1233acd390 2025-09-19T03:59:57.671964Z my-test-stack AWS::CloudFormation::Stack ROLLBACK_COMPLETE null
[22:59:58.748] [ERROR] [main] LambdaFunctionITBase - Stack event: 0762417c-969f-4ec2-89c2-c4f87bb08ace 2025-09-19T03:59:57.671937Z my-test-stack AWS::CloudFormation::Stack CREATE_FAILED null
[22:59:58.748] [ERROR] [main] LambdaFunctionITBase - Stack event: f84893e9-29e5-451f-beeb-57d82691ca55 2025-09-19T03:59:57.671929Z LambdaFunction AWS::Lambda::Function CREATE_FAILED An error occurred (ParameterNotFound) when calling the GetParameter operation: Parameter ${ParameterName} not found.
[22:59:58.748] [ERROR] [main] LambdaFunctionITBase - Stack event: 596e56f8-cfb9-4a9a-bff9-773fad8a1319 2025-09-19T03:59:57.665751Z my-test-stack AWS::CloudFormation::Stack CREATE_IN_PROGRESS null
Note the presence of a CREATE_FAILED event on the LambdaFunction resource.
How are you starting LocalStack?
Custom (please describe below)
Steps To Reproduce
How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)
I'm starting LocalStack inside my Java JUnit5 tests using this code:
@Container
@SuppressWarnings("resource")
public static LocalStackContainer localstack =
new LocalStackContainer(DockerImageName.parse("localstack/localstack:4.7"))
.withServices(LocalStackContainer.Service.CLOUDFORMATION, LocalStackContainer.Service.S3,
LocalStackContainer.Service.LAMBDA, LocalStackContainer.Service.IAM,
LocalStackContainer.Service.SSM, LocalStackContainer.Service.SECRETSMANAGER)
// Let's get some debug output if things go wrong
// .withEnv("DEBUG", "1")
// Increase the Lambda timeout to 5 minutes to avoid intermittent failures
.withEnv("LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT", "300")
// Use the "local" executor to avoid intermittent failures
// .withEnv("LAMBDA_RUNTIME_EXECUTOR", "local")
// Give LocalStack enough time to start up
.withStartupTimeout(Duration.ofMinutes(1L));Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
I am creating the stack using the AWS Java SDK v2 CloudFormation client.
Environment
- OS: MacOS 15.6.1 (local), Ubuntu 24.04.3 LTS (GitHub Actions) both
- LocalStack:
LocalStack version: 4.7, 4.8
LocalStack Docker image sha: 4.8: a6d609a1714f, 4.7: ad4f76a02108
LocalStack build date:
LocalStack build git hash:Anything else?
My best guess about what's going on here is a change in "order of operations" for dynamic reference resolution. I noticed some PRs that were merged between 4.7 and 4.8 that seem relevant. I have not run the issue to ground, but these came up in my search for an existing issue, so I'm providing them here in case it's useful information:
- CFnv2: implement SSM parameter resolving #12934
- CFNv2: support resolve:ssm: and resolve:secretsmanager: strings #12965
I use these parameters in tests for creating the template:
Profile: test
LambdaAlias: test
BuildId: 1234
CognitoUserPoolId: us-east-1_GoodUserPoolId
These steps are required to "prepare" for creating that template, in Java code, from my tests:
ssm.putParameter(b -> b.name("/foobar/test/plans/free/aws").type(ParameterType.STRING)
.value("free.aws"));
ssm.putParameter(b -> b.name("/foobar/test/plans/small/aws").type(ParameterType.STRING)
.value("small.aws"));
ssm.putParameter(b -> b.name("/foobar/test/plans/medium/aws").type(ParameterType.STRING)
.value("medium.aws"));
ssm.putParameter(b -> b.name("/foobar/test/plans/large/aws").type(ParameterType.STRING)
.value("large.aws"));
secretsManager.createSecret(
b -> b.name("foobar/test/stripe").secretString("{\"apiKey\":\"S00p3rS3cret!\"}"));
secretsManager.createSecret(
b -> b.name("/foobar/test/signing").secretString("{\"hex\":\"1234\"}"));