Our serverless files we use get pretty complex which is why we thought the setting environment variables weren't working. Your example works without the modifications up until you start referencing AWS resources either through Ref, GetAtt or ImportValue. Looking at the existing serverless framework I think I found where the environment variables are currently being set. "lib/plugins/aws/invokeLocal/index.js". I'm thinking that I should remove that code I sent, and instead add an update to the previously mentioned file to interrogate the existing cloud formation deployment if it exists and translate those values.

Modified Example:
service: philipp-test-${self:custom.idx}

provider:
name: aws
runtime: nodejs10.x
versionFunctions: false
region: eu-central-1
stage: dev

environment:
PROVIDER: ${self:provider.name}
REGION: ${self:provider.region}
Test:
Ref: s3Bucket

custom:
idx: 0

functions:
hello:
handler: functions/handler.handler
environment:
HANDLER: ${self:functions.hello.handler}
IDX: ${self:custom.idx}

resources:
Resources:
s3Bucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: test

Issue with output:
Test: '[object Object]'

I've reverted the original modifications and made changes to get referenced and imported resources for SNS, SQS, DynamoDB, Lambda and ImportValue.

Here's the test yml I am using. To test import you will need to comment the reference to the export in the environment variables, deploy, then un-comment. Typically ImportValue would be used to get values from other cloudformation stacks.

service: philipp-test-${self:custom.idx}

provider:
name: aws
runtime: nodejs10.x
versionFunctions: false
region: eu-central-1
stage: dev

environment:
PROVIDER: ${self:provider.name}
REGION: ${self:provider.region}
Test:
Fn::GetAtt:
- dynamoDB
- Arn
snsTest:
Ref: sns
snsTestName:
Fn::GetAtt:
- sns
- TopicName
sqsQueueRef:
Ref: sqsQueue
sqsQueueArn:
Fn::GetAtt:
- sqsQueue
- Arn
sqsQueueName:
Fn::GetAtt:
- sqsQueue
- QueueName
imported:
Fn::ImportValue: ${self:service}-${self:provider.stage}-sqs

custom:
idx: 0

functions:
hello:
handler: functions/handler.handler
environment:
HANDLER: ${self:functions.hello.handler}
IDX: ${self:custom.idx}
refLambdaName:
Ref: Hello2LambdaFunction
refLambdaArn:
Fn::GetAtt:
- Hello2LambdaFunction
- Arn

hello2:
handler: functions/handler2.handler
environment:
HANDLER: ${self:functions.hello.handler}
IDX: ${self:custom.idx}

resources:
Resources:
dynamoDB:
Type: AWS::DynamoDB::Table
Properties:
TableName: test
BillingMode: PAY_PER_REQUEST
AttributeDefinitions:
- AttributeName: test
AttributeType: S
KeySchema:
- AttributeName: test
KeyType: HASH

sns:
  Type: AWS::SNS::Topic
  Properties: 
    TopicName: test-sns

sqsQueue:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: TestQueue

Outputs:
test:
Value:
Ref: sqsQueue
Export:
Name: ${self:service}-${self:provider.stage}-sqs

Hi Mariusz,

That statement isn't quite accurate. Due to the way AWS operates as a consordium of individual teams, there isn't a single definition of Ref vs GetAtt. At the same time it is well documented what should be returned for each. The apis for each also have quite a bit of variation to them, making it hard to make a single function perform the value retrieval operation. This is why in the framework the conversion is broken down per service rather than trying to consolidate it all into a single function.

Reference Documentation:
DynamoDB - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html#aws-resource-dynamodb-table-return-values
SNS - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sns-topic.html#aws-properties-sns-topic-return-values
SQS - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html#aws-properties-sqs-queues-return-values
Lambda - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html#aws-resource-lambda-function-return-values
S3 - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html#aws-properties-s3-bucket-return-values

The "lib/plugins/aws/invokeLocal/resolvers/index.js" file creates a registration process per handler as can be seen on lines 4 - 10. This is then used if a "Ref", "GetAtt" or "ImportValue" is used in the serverless yaml for the environment variables. As the values are retrieved they are cached in the "./.serverless" directory, which during a deployment will get removed as the cache might be invalid after the deployment.

While there are more services than just the 5 I implemented, these 5 are the most commonly used with serverless AWS implementations. For any project which is significant, the ability to leverage references rather than hard coded values becomes critical, and there for local debugging without this with serverless framework without this capability is useless.

With respect, as this is a primitive capability of AWS in the same way cloudformation is, this should be a framework level component as your user base shouldn't need to add a custom plugin just to be able to debug their basic stack functions.

@nkeating-mutualofenumclaw great thanks for further clarifications.

Indeed I was mislead by AWS support, I contacted them back on that, and they've responded:

I see now that I misunderstood your question in by the previous case note. As you have rightly pointed out, the values returned by intrinsic functions are documented in "Return Values -->Ref" for each resource respectively.

In light of that, indeed it makes sense to implement resolution for popular cases, and eventually add resoulution for others on the go, whenever demand appears.

Let me review the PR

I've implemented the changes suggested. Let me know what you think

I have on multiple times done what you had asked with more and new changes coming in the next review. I no longer have time to continue this back and forth, as I have other responsibilities.

At this point the updates work and improve the serverless framework beyond what it is currently capable of doing. Your other customers are asking when this update will be available to them. If there are more changes you'd like to make or try, you should feel free to make them. If you'd like to recreate the whole resolution code base feel free to.

@nkeating-mutualofenumclaw great thanks for this effort! It's a really worthwhile contribution, and I'm pretty sure many will appreciate it when it lands.

We'll try to do our best to finish it whenever we have some free resources for that.
As it's uncertain when it'll be the case, I'll mark it for now with help wanted badge.

If anyone wants to jump in, and finish this PR, it's very appreciated!

What's happening with this change?

@ionush we're looking for help, it's waiting for someone to take it over and finalize.

Can the pull request from @medikoo be merged?

Also interested in this change

@PierrickI3 @corbfon we're looking for help in finalizing that pull request. Are you in for that?

@medikoo In a project right now until the end of July and requires me to work in the evenings too. Not sure how much time I will be able to spare.

@medikoo I could probably take a look later this week

@medikoo I could probably take a look later this week

That'll be great, thanks @corbfon

Is anyone working on this PR?

@herebebogans I don't think anyone is. Do you want to finish it?

The problem with AWS Cloudformation intrinsic functions not being resolvable via environment variables when running locally (either via invoke local or serverless-offline) has come up many times and I've contributed to a few of those discussions.

My thinking is that this is better left for plugins to solve otherwise the invokeLocal resolver module being created in this PR will need to be continually be updated as AWS update Cloudformation with new services.

Theres a few plugins that already deal with this issue - though they have some limitations.

The author of this PR has

https://github.com/mutual-of-enumclaw/serverless-plugin-local-env

There is also.

https://www.npmjs.com/package/serverless-export-env

Perhaps the documentation for serverless invoke local should describe the issue and point to these plugins?

Perhaps the documentation for serverless invoke local should describe the issue and point to these plugins?

We should probably ensure unified behavior, and crash if we find that it can't be meet, so:

• Do not leak local env vars into invocation
• Abort local invocation with an error if function has configured env vars that require resolution via external service. (then we can also provide a hint pointing to plugins)

@nkeating-mutualofenumclaw

I'm guessing you raised this PR as it was difficult to implement this in a plugin? Like to get your input as you've put in a lot of work on it.

Perhaps the documentation for serverless invoke local should describe the issue and point to these plugins?

@herebebogans My opinion on this is that it's serverless itself should include these env variables, not a plugin. The ultimate goal is that invokeLocal should behave as close to cloud execution as possible. This means if a lambda function in AWS gets a variable, it should get the same variable locally.
I reckon the main reason why it was previously implemented as a plugin is that, as we've seen with this PR, it takes months of effort to get a change like this to get merged into serverless.

@herebebogans My opinion on this is that it's serverless itself should include these env variables, not a plugin. The ultimate goal is that invokeLocal should behave as close to cloud execution as possible. This means if a lambda function in AWS gets a variable, it should get the same variable locally.

I can understand how the surprise of [object Object] as an environment variable impacts those of us who like to develop locally but refer to AWS resources in a consistent way as happens in the lambda runtime.

Supporting the most commonly used resources ie .. S3, Dynamo , SQS etc as is being done in this PR would definitely help , however for this feature to be complete you would need to support all Cloudformation Resources which is a large initial and continous ongoing effort.

In addition do you draw the line only supporting the common cloudformation return types ie !Ref, !GetAtt , !ImportValue etc, or would this resolver need to handle all CF intrinsic functions? eg something like

!FnIf ['mycondition', !Ref ThisS3Bucket, !Ref ThatS3Bucket]

There's a larger issue already in serverless with CF intrinsic functions - they are supported and implemented differently across serverless. For example I can use !ImportValue in a sqs event definition but not in a kinesis event definition or for the provider.deployment bucket property .. Why?

I think adding more of this only adds to the problem?

Eventually all the scenarios should be covered. The problem is without a base of support, even around referenced variables, the framework itself starts becoming less useful for development purposes. This becomes even more stark when its the Solution Architect that chooses and typically writes the serverless file, and developers don't have the ability to do what other frameworks (like SAM) have baked in to their ability to debug.

The FnIf example isn't that hard to implement as an additional feature, nor are other cloudformation functions. Trying to solve everything in one go though is setting this process up to be a long drawn out process, when a more iterative process would allow this feature to be added and improved on much more quickly.

All good points, you've convinced me to finalise this PR ..

@herebebogans, any luck? This would be really useful

I had a look through the PR and started extending it to support more resources it but it's solely focused on supporting invoke local. I think the majority of users that want this feature are really wanting it to work with both serverless-offline and invoke local .

@medikoo can you think of a more general purpose way this can be solved in the framework to satisfy both.

@medikoo can you think of a more general purpose way this can be solved in the framework to satisfy both.

if you mean that resolution of env vars should be done in a way that serves both local invocation and eventual plugins. Then I think it can be achieved by secluding the resolution logic into serverless.service.resolveFunctionEnvVars(functionName) method which then could be used by both.

What do you think?

@nkeating-mutualofenumclaw great thanks for the effort! I'm going to close this as this was partially addressed by #8157 (added support for Fn::ImportValue resolution) and #8198 (added support for Ref resolution) and at current stage probably this branch has derived too far from master to continue work on it.

What's missing (and what was supposed to be covered with this PR) is support to resolve Fn::GetAtt. We're open for PR's that'll bring support for that function specifically.

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 28.75817% with 109 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.08%. Comparing base (526b21b) to head (cdf318b).

@@            Coverage Diff             @@
##           master    #7147      +/-   ##
==========================================
- Coverage   88.14%   87.08%   -1.06%     
==========================================
  Files         236      243       +7     
  Lines        8640     8776     +136     
==========================================
+ Hits         7616     7643      +27     
- Misses       1024     1133     +109     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.