AWS Serverless Integration Tests With Moto - What Can go Wrong?

2019-06-11

tags: aws, credentials, serverless, python, moto, testing, integration

Python logo

Context

I’m using moto for testing of my Python lambdas. It nicely mimics all most often used AWS services and lets you unit-test your code in isolation.

At the same time I write some integration tests that are working with real – i.e. not mocked – AWS resources.

I want to distinguish unit from integration tests so I chose to properly annotate them like this:

@pytest.mark.integration
def test_sending_noification_using_sns():
    # given
    (...)

For unit tests I’m not using any test type annotation. Well, you can add it, but I find it superfluous as great majority of the tests are unit tests anyway. @mock_dynamodb2 is related with moto and not the level of a test:

@mock_dynamodb2
def test_persist_entry():
    # given

Thanks to this I can call:

python -m pytest -m "not integration", for unit tests only
python -m pytest -m integration, for integration tests only

and Python will auto-discover tests and run only those with appropriate annotation, e.g.:

[Container] Running command python3 -m pytest -m "not integration" 
======================== test session starts ========================= 
[...]
collected 72 items / 11 deselected / 61 selected 
 
[...]

========= 61 passed, 11 deselected, 3 warnings in 6.57 seconds ========= 

Take a look at “11 deselected” - those are integration tests not being executed when "not integration" selector is used.

Moto Plays With AWS Credentials Environment Variables

Now, when you use moto you need to know that it sets AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to some fake values. More precisely: foobar_key and foobar_secret accordingly (see it in source code.)
I believe this is to make sure you won’t hit the real AWS services by any chance.

Now, for unit tests this sounds perfectly valid but for integration tests not really.

If you’re executing the integration tests then pytest already scans test methods for the annotations (including analyzing unit tests that does use moto.) So it hits the import moto statement and you end up with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY being populated by moto with mock values.

Workaround

In boto3 (Python library for accessing AWS resources) environment variables are very high in the hierarchy or a loading process. Higher are only in-code values which you don’t want to hard-code for security and configuration reasons.

Overriding environmental variables during a build is also not an option as it’s both: not secure and it will be anyway overridden by moto when running tests.

One possible workaround here is to keep the tests in separate directory like “/integration” and then execute pytest like this:

python -m pytest tests/integration

(alternatively add -m integration to be extra safe that unit tests – if put there mistakenly in this folder – will not be executed.)

This will make sure that unit tests won’t mix with integration tests and you won’t hit unneeded import moto statement.

Moto is missing an option to control this overriding behaviour.

I think that without large impact there might be a variable that will allow you to override the default behaviour and prevent moto from putting those fake values.