Hey I pushed my code: https://github.com/dougmoscrop/serverless-plugin-split-stacks

I think I am taking a slightly different approach than @pmuens which is to say I am trying to use a quasi "pluggable" system for detecting resource hierarchies. I still resolve references of migrated resources using graph traversal, but I'm not using the graph as the basis for migration.

So far I have a working integration test and supporting unit tests. I've only tested with a subset of events, resources, and so on. Because of the detection method I am using, it's totally opt-in/explicit rather than implicit.

The plugin itself is not ready for release, but the code is there and I'm going to start testing - and fixing - it against our own serverless services.

That's super nice! Thanks for sharing @dougmoscrop

I'll look into it the upcoming week. Great to hear that you took a slightly different approach. Maybe this PR will be a little bit inspired by your plugin 😬

This PR is getting closer to work with all different setups. Had quite a trial and error time yesterday to figure out how the resources in the nested stack needs to be updated so that dependencies can be established between the parent and the nested stack.

@pmuens , just had a thought about the stack splitting. Is it possible to have that feature on by default but only have it split the stacks if the resource count is greater than the limit? Then we could offer a way to toggle off the stack splitting disableStackSplitting: true.

This way users will no longer run in to the resource count limit error but we won't split stacks until it's an issue.

@brianneisler that sounds like a good plan from a DX / UX perspective. However I'm not sure if it's too much magic and can cause unexpected behavior.

As a developer you'll have multiple ways to solve this kind of issue. The first one is the usage of automatic stack splitting (done by serverless). The second could be that you'll use your own custom splitting mechanism based on nested stack. The third (and "most professional / advanced") way would be to use imports and exports / cross-stack-references to solve this problem.

TBH currently I'm more in favor of showing an info message which outlines the different paths the user can take once he reaches the resource limit (with direct solutions how he can immediately fix it). This way he knows that there are more ways to resolve this issue.

Open for other ideas though...

I'm also having trouble enabling stack splitting for an existing stack. CFN does not seem to handle this very well at all.

Not to pile on, but migration breaks plugins as well - here's an example: https://github.com/gmetzker/serverless-plugin-lambda-dead-letter/blob/master/src/index.js#L112

I suppose you could try to intercept calls through provider and rewrite the names/stack names of things that were migrated. Ouch. Alternatively, some kind of actual layer of abstraction needs to go in top of resources, instead of just bare AWS calls :(

@pmuens is there any real forecast about when this issue is going to be closed?
It continuously gets pushed to the next release and we're holding back a release of our software waiting for it. I am sorry to point out this, stating the remarkable job you all did on Serverless.
Last commit is May 22nd and after a month I see no evidence of this being resolved.
Should we continue waiting for next release or do you suggest to implement a viable alternative?

We're also waiting for the resolution, so it could be useful to know what is going to be done, and if it's possible, when. Thanks

Hello everyone!

First of all: Thanks everyone for the great discussions in this PR and the related issues. Seems like there's huge demand and we agree, that it would be super nice to have such a functionality baked into core.

We've scheduled the fix for this for the upcoming release, however there are still some unknows which makes it hard to predict if we can ship a stable implementation in v1.17.

It turns out that splitting stacks into different nested stacks is not that easy as it looks at the first glance. There are many different edge cases one can run into.

However we brainstormed a little bit and came up with the following implementation proposal we'll work on:

• Make it possible for users to switch to automatic stack splitting once they hit a limitation (the current implementation doesn't allow this. So the user needs to do deployments with stack splitting enabled from the very beginning)
• Do the first splitting when the user hits a stack limitation (not earlier)
• Catch errors returned by AWS which indicate a resource limitation and kick-off the stack splitting process:
  • Query the current state and diff against the changes which should be made
  • Put new resources (which will exceed the stack limitations) into an own, nested stack
  • Fill up nested stack until this stack is full
  • Repeat this process and create new nested stacks as needed
• Make it possible to put "special" resources such as DynamoDB into own, separate stacks to ensure that they're e.g. not removeable

I'll open up a new PR with this new implementation since this PR is quite bloated. However I'll cherry-pick the "good parts" from this PR.

For everyone who needs such a functionality now.

Please take a look @dougmoscrop great "Split Stack" plugin: https://github.com/dougmoscrop/serverless-plugin-split-stacks

/cc @aletheia @helveticafire @eliasisrael @hughneale @emilioponce

@pmuens Do the first splitting when the user hits a stack limitation (not earlier)

This needs as prerequisite that all used plugins work with and support stack splitting. Otherwise a project might suddenly render undeployable.
All plugins that change the CF template before deployment are candidates to make this break!

I'm almost on the same situation as @aletheia. Almost because we're not totally dependent on this feature, yet.
But in my case, the proposal on #2995 would solve my problem, and I think the manual stack splitting is easier to code than an automatic way. So I was thinking if wouldn't be easier (and faster) to resolve the 200 resources limit problem with the manual proposal.

Manual splitting could be good for us too: our main issue is related to the high number of CF resource every endpoint uses to set up. Manual splitting endpoints in different stacks could be a viable solution

Thanks for the feedback everyone!

Just wanted to highlight that you could still use stack Outputs and Fn::ImportValue as another solution (or replacement for nested stacks) to manually work around the current limitations.

This needs as prerequisite that all used plugins work with and support stack splitting. Otherwise a project might suddenly render undeployable. All plugins that change the CF template before deployment are candidates to make this break!

Good catch @HyperBrain. Yes, I remember that you mentioned smth. like that in the PR reviews you did for this PR.

The main problem with that is that users run into the stack limitations throughout the lifecycle of their project so the automated stack splitting should kick in once they get to this problem. It shouldn't be a prerequisite to decide whether I want to use stack splitting or a "monostack" when starting a project (this would defeat the purpose the stack splitting should solve).

Still not sure how to introduce this in a non-breaking way for plugins 🤔.

Will wrap my head around this...

@pmuens our current project in in dev state when we hit the limit? if we do an 'sls remove', then redeploy the project with the stack splitting enabled, will it work properly?

I tried @dougmoscrop 's plugin, and it worked great for a while but then we hit the limit again.

Also, is there any easy way to install serverless from this branch then?

@dougmoscrop I changed some migrations in migrate-resources.js in your plugin for my project and it worked for more number of resources, would you be open to merging them if I fork and push?

@dougmoscrop I'm trying to implement the stack splitting plugin in my existing project, but I'm not exactly sure how to use it. Could you provide an example? I've installed the plugin and listed it under my plugins in the serverless.yml file just don't know where to go from there.

@selected-pixel-jameson that should be enough. On the next deployment you should see the differences.

Mind you, the plugin will change your URL, and needs Serverless version > 1.13 to work.

Ah!! I'm running serverless 1.10 locally due to some outstanding issues packaging individual java functions. WIll try to implement this on my build server and give it a whirl. If you have a custom URL for the endpoint will you need to remap it?

I believe you will as it allocates a new API Gateway Resource based on the nested Cloudformation template.
Havet tried so cant say 100% surely though.

Does it re-create the API every time or just with the initial update?

Just wondering what the status of this feature is? Has a new PR been created? The plugin works for now, but the fact that it deletes and re-creates the API is a major problem. If you're doing any automation surrounding the downloading of a RestAPI SDK you'll have to update the RestAPI ID all the time in your script or write something to search through your list. Thanks for all your hard work on this feature up to this point. Really appreciate.

@pmuens What is the status of this plugin? Has this been merged, still being worked on or new PR has been created? Any ETA on when this will be available? Thanks for the excellent work here, its really appreciated.

Wow, sorry I never got back to this. Trying to remember, but its definitely been working for me and I only saw it update the RestAPI ID in once instance. I think that maybe it happened when I initially integrated the plugin and hasn't recreated it since.

I am maintaining my plugin as best I can, dealing with different issues that come up when other plugins are mixed with it.

The first thing you should do is see if you can get away from hitting the CF limit. Can you split your services up in to multiple separate deployable units? Having an "API" tier and a "Worker" tier is straightforward. You can even share a git repo and just have two subfolders that import from a common lib folder. The next step is using CF nesting 'manually' - define some stuff outside of your stack.

If you intend to go in to automatic stack splitting, @medikoo has PR open from their fork which does the splitting based on functions instead of types. I still have not had a chance to test it on services of my own, but please consider it and provide feedback in the PR.

I can't speak for serverless themselves, but I think the behavior is so complex and prone to bugs that it should not go in to core.