Local Development and Testing of AWS Serverless Application

Context

For at least a few weeks now I was trying to figure out how serverless developers are:

My interest lies in AWS, Python and Serverless Framework so I’ll focus on them.

What is Already on The Market

There are a couple of players on this market like:

I’m pretty sure there are more of them. The issue I had is that I couldn’t find a solution that will be mature enough, often updated and will work with most current Serverless Framework and AWS APIs (e.g. S3 not available yet, Lambda not being found in localstack, Issue with DynamoDB Client). I ended up googling one issue after another spending time not how I believe I should be.

What I Needed was a Mindset Shift

And then I just realized that I’m trying to find a perfect solution similar to the environments I was working until now and try to run everything locally. Serverless might be a different story. It composes of:

In this fast pace world it is hard for any mock-platform provider to catch up with the cloud provider (take a look at AWS This Week to see how fast those services are evolving. Not as fast as new JavaScript frameworks are created but quite fast :-).

Understanding this made me to think about using direct AWS account / environment per-developer.

With proper separation (e.g. separate accounts within one organization) you can properly ring-fence yourself from the production environment. With proper configuration of resources and architecture the costs of such per-developer environments can be also reduced – e.g. do not use EC2 instances or configure DynamoDB instances always to work in ‘On-Demand’ payment mode.

At the same time you’re shortening the distance between local and production environment and gaining the possibility to find bugs with the integration layer sooner than before.

The developers are working all the time on real AWS environment we’ll use in production.

Disadvantages

Of course, it’s not so perfect if you take a closer look at the downsides of such solution.

Local development is slower.

For resources you need to go to AWS and depending on your internet connection, the information loop (write, run, test, verify, fix) will be longer. Imagine that when you change a piece of code, you need to deploy it to AWS and then execute it and repeat this routine many times during every-day development.

I tend to think that not being able to develop and test everything locally (as in your local-machine) is a huge downside. However, I have reviewed my approach – nowadays we got fast internet connection almost from every place on the world, At the same time, if you properly unit-test your lambda business code, developers are able to test them locally.

If you’re working on Lambdas triggered mainly via HTTP then a hybrid model is also tempting.
You could use some solutions that are already on the market and are working fine in this limited set of capabilities. This will give you a development boost.

What I mean is you could use serverless-offline and work on your Lambdas offline, test it using API Gateway on localhost then modify the code, refresh the web page and you’ll see the results.

At the same time other resources (DynamoDB, SNS, SQS, Cognito, S3 are taken from real AWS endpoint.

My Current Local Development and Testing Process

So this is how my current development looks like:

Summary

I’m definitely not saying the above assumptions are the best but that’s what right now I think makes your local development as close as possible to the production environment. At the same time it gives you at least a bit of a boost in development in form of those basic offline AWS services (Lambda + API GW).
Hence, that’s what I decided to go and verify how it works in a longer run.

Wonder if you guys have any experience with your own approach to serverless applications local development and testing. I would love to read it in comments!