Contact Us

AWS Lambda & Serverless Development - Part 2: Testing & Debugging

author image Tom McLaughlin

May 16, 2018   //   serverless, tutorial, development

Beautiful divider with lightning bolt in the middle
 

This is part 2 of a 2 part series detailing an AWS Lambda and serverless development workflow.  In Part 1 we discussed everything we do before coding; design and project setup. This will cover the steps after you have done your coding; testing and debugging. We're building an AWS serverless application that publishes AWS Health notifications to Slack. The two services we use to construct this application are:

We're going to focus on how we test and debug the aws-sns-to-slack-publisher service.

serverless-dev-workflow

Once my initial code for a service is written I start writing tests. Adequate testing reduces the frustration that can occur by the slow feedback loop of code, deploy, fix, redeploy, repeat. I don’t go through the trouble of using something like localstack or other tools to run my functions locally. On the down side I can't check if my code, "works on my machine". On the plus side I can't use, "works on my machine" as en excuse to not know whether it works once deployed to AWS. This forces me to do decent testing and debugging which is better in the long run.

Testing

I’ll start writing tests to begin testing and debugging my code. Every deploy is going to cause you at least a one to two minute interrupt of waiting. If you’re not testing before you deploy then the last mile of completing a new service will become extremely frustrating. I have been through this. Catch as many errors as you can before you deploy.

Coming from an operations background, I understood the importance of testing in the abstract but hadn’t spent enough time actually doing it. I’ll also add that for operations engineers in a serverless environment, I suspect testing will become an integral part of our role. When you find a bug in a running system, after you’ve fixed it, then you should evaluate whether a test is warranted to prevent the issue in the future. This is one way you’re going to add value in your role. This is how you’re going to improve the reliability of the systems your team builds.

Since I’m using Python to write my services, I use pytest to write my tests.

Unit Tests

I’ll add some unit tests to the aws-sns-to-slack-publisher service. The service does the following actions.

  • Receives an SNS message.
  • Validates the data received from SNS.
  • Publishes the data to Slack as a message.
  • Optional: Publish Slack response to another SNS topic.

To aide writing simple and easy to understand unit tests, I try and keep my handler function very simple.

def handler(event, context):
    '''Function entry'''
    _logger.debug('Event received: {}'.format(json.dumps(event)))
    slack_message = _get_message_from_event(event)
    _validate_slack_message_schema(slack_message, SLACK_MESSAGE_SCHEMA)
    slack_channel = _sanitize_slack_channel_name(SLACK_DEFAULT_CHANNEL)
    _check_slack_channel_exists(SLACK_API_TOKEN, slack_channel)
    slack_response = _publish_slack_message(SLACK_API_TOKEN, slack_channel, slack_message)

    resp = {
        'slack_response': slack_response,
        'status': 'OK'
    }

    if SNS_PUBLISH_RESPONSE:
        sns_response = _publish_sns_message(RESPONSE_SNS_TOPIC_ARN, slack_response)
        resp['sns_response'] = sns_response

    _logger.debug('Response: {}'.format(json.dumps(resp)))
    return resp

 

I’ll create a unit test for _validate_slack_message_schema(). The function takes a takes a Slack message and compares it against the Slack chat.postMessage API message.

Remember when I said to create an event earlier? This is exactly why. This unit test is going to use the data in that file and pass it to _validate_slack_message_schema(). And how do we ensure the data returned is formatted correctly? I have a JSON schema file to compare it against. The code to do this looks as follows.

import json
import os

import jsonschema
import pytest

import handlers.aws_sns_to_slack_publisher as h

EVENT_FILE = os.path.join(
    os.path.dirname(__file__),
    '..',
    '..',
    'events',
    'aws_health_event_publisher.json'
)

SLACK_SCHEMA_FILE_PATH = os.path.join(
    os.path.dirname(__file__),
    '../../../slack-message-schema.json'
)


@pytest.fixture()
def event(event_file=EVENT_FILE):
    '''Trigger event'''
    with open(event_file) as f:
        return json.load(f)


@pytest.fixture()
def slack_message(event):
    '''Slack message'''
    return h._get_message_from_event(event)


@pytest.fixture()
def slack_message_schema():
    '''Slack message schema'''
    with open(SLACK_SCHEMA_FILE_PATH) as f:
        return json.load(f)


def test__validate_slack_message_schema(slack_message, slack_message_schema):
    '''Validate a Slack message.'''
    # Throws an exception on bad in put.
    h._validate_slack_message_schema(slack_message, slack_message_schema)

 

If you look at tests/unit/handlers/test_aws_sns_to_slack_publisher.py you'll see there's a variety of additional tests to ensure that we properly validate different messages.

Now that I know the Slack message is properly formatted I go ahead and test the SNS publishing. This adds a wrinkle. How do you test against AWS services locally? I mentioned earlier that I don't use localstack. I prefer using the python library moto for testing against AWS services. I’ve found moto quick and easy to use. Just import the needed service as a decorator on the test function and inside the test function use boto3 as normal. Because I use assumed roles to access AWS accounts such as my dev account, I have to add the @mock_sts decorator to my functions as well. It took me hours to initially discover this and make my tests work.

Publishing to SNS requires an SNS topic ARN and a message. After publishing the message we’re just going to ensure that we get a successful response back. The code for our unit test looks as follows. (I’ve removed the Slack related testing bits for clarity.)

import boto3
from moto import mock_sns, mock_sts
import pytest

import handlers.aws_sns_to_slack_publisher as h

SNS_TOPIC_NAME = "mock-aws-sns-to-slack-publisher-responses"


@pytest.fixture()
def sns_client():
    '''SNS client'''
    return boto3.client('sns')


@pytest.fixture()
def sns_message(event):
    '''SNS message'''
    return h._format_slack_message(event)


@pytest.fixture
def sns_topic_name():
    '''SNS topic name'''
    return SNS_TOPIC_NAME

@mock_sts
@mock_sns
def test__publish_sns_message(sns_client, sns_message, sns_topic_name):
    '''Test publish an SNS message.'''

    sns_create_topic_resp = sns_client.create_topic(Name=sns_topic_name)
    sns_publish_resp = h._publish_sns_message(
        sns_create_topic_resp.get('TopicArn'),
        sns_message
    )

    assert sns_publish_resp.get('ResponseMetadata').get('HTTPStatusCode') == 200

 

I’ll run my test now and ensure they all pass.

$ pytest -v tests/unit/
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.6.4, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /Users/tom/.pyenv/versions/3.6.4/envs/aws-sns-to-slack-publisher/bin/python3.6
cachedir: .pytest_cache
rootdir: /Users/tom/Source/serverlessops/aws-sns-to-slack-publisher, inifile: pytest.ini
plugins: pylint-0.9.0
collected 12 items

tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__check_slack_channel_exists SKIPPED                                                      [  8%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__get_message_from_event PASSED                                                           [ 16%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__publish_slack_message SKIPPED                                                           [ 25%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__publish_sns_message PASSED                                                              [ 33%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__sanitize_slack_channel_name_clean PASSED                                                [ 41%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__sanitize_slack_channel_name_dirty PASSED                                                [ 50%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__validate_slack_message_schema PASSED                                                    [ 58%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__validate_slack_message_schema_has_attachment PASSED                                     [ 66%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__validate_slack_message_schema_has_attachment_bad_type PASSED                            [ 75%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__validate_slack_message_schema_has_attachment_missing_text_property PASSED               [ 83%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__validate_slack_message_schema_has_attachment_has_fields PASSED                          [ 91%]
tests/unit/handlers/test_aws_sns_to_slack_publisher.py::test__validate_slack_message_schema_has_attachment_has_fields_missing_property PASSED         [100%]

=========================================================== 10 passed, 2 skipped in 0.71 seconds ============================================================

 

Once I’ve finished my unit testing and have successful tests then I deploy to my dev environment. The command to deploy is:

serverless deploy

 

By the way, there’s some added complexity to deploying aws-sns-to-slack-publisher because it needs a Slack API token and the name of an SNS topic to subscribe to. Since I’m not using AWS Systems Manager Parameter Store in this example I just pass in the values as environmental variables on the command line. The setup to accomplish this is in my serverless.yml file. The actual command I run and output is here:

$ SNS_PUBLISHER_TOPIC_EXPORT=twitter-to-slack-message-dev-SlackMessageSnsTopicArn SLACK_API_TOKEN=|REDACTED| SLACK_DEFAULT_CHANNEL="testing" sls deploy -v
Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (2.14 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - aws-sns-to-slack-publisher-dev
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::SNS::Topic - SlackResponseSnsTopic
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - SlackPublishLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::SNS::Topic - SlackResponseSnsTopic
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - SlackPublishLogGroup
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - SlackPublishLogGroup
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::SNS::Topic - SlackResponseSnsTopic
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - SlackPublishIamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - SlackPublishIamRoleLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - SlackPublishIamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - SlackPublishLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - SlackPublishLambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - SlackPublishLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - SlackPublishLambdaVersionIy4JHxZ7Db3c2ccBBfHpKPbIcc3JaLOeeDjvZ9TQAw
CloudFormation - CREATE_IN_PROGRESS - AWS::SNS::Subscription - EventPublishSnsSubscription
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - S3BillingItemWriterLambdaPermission
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - SlackPublishLambdaVersionIy4JHxZ7Db3c2ccBBfHpKPbIcc3JaLOeeDjvZ9TQAw
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - S3BillingItemWriterLambdaPermission
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - SlackPublishLambdaVersionIy4JHxZ7Db3c2ccBBfHpKPbIcc3JaLOeeDjvZ9TQAw
CloudFormation - CREATE_IN_PROGRESS - AWS::SNS::Subscription - EventPublishSnsSubscription
CloudFormation - CREATE_COMPLETE - AWS::SNS::Subscription - EventPublishSnsSubscription
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Permission - S3BillingItemWriterLambdaPermission
CloudFormation - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - aws-sns-to-slack-publisher-dev
CloudFormation - UPDATE_COMPLETE - AWS::CloudFormation::Stack - aws-sns-to-slack-publisher-dev
Serverless: Stack update finished...
Service Information
service: aws-sns-to-slack-publisher
stage: dev
region: us-east-1
stack: aws-sns-to-slack-publisher-dev
api keys:
  None
endpoints:
  None
functions:
  SlackPublish: aws-sns-to-slack-publisher-dev-SlackPublish

Stack Outputs
SlackResponseSnsTopicArn: arn:aws:sns:us-east-1:355364402302:aws-sns-to-slack-publisher-dev-SlackResponseSnsTopic-J0WVJ1ZQCATE
SlackPublishLambdaFunctionQualifiedArn: arn:aws:lambda:us-east-1:355364402302:function:aws-sns-to-slack-publisher-dev-SlackPublish:47
ServerlessDeploymentBucketName: aws-sns-to-slack-publish-serverlessdeploymentbuck-wygwuid0btg8

 

With the service deployed to AWS I can continue my testing.

Integration Tests

The importance of integration can’t be understated. Nearly every serverless application is a system. That is, a collection of independent parts that need to function together. Integration tests lets you test both the code and the infrastructure of the system to ensure they work together properly. Issues you might commonly see are:

  • Misconfigured function triggers
  • Misconfigured IAM roles
  • Incorrect credentials to third part services

I’ll add a quick integration test that invokes the Lambda function handler of the dev instance of the service deployed in AWS.

import json
import os

import boto3
import pytest

EVENT_FILE = os.path.join(
    os.path.dirname(__file__),
    '..',
    '..',
    'events',
    'aws_sns_to_slack_publisher.json'
)


@pytest.fixture()
def event(event_file=EVENT_FILE):
    '''Trigger event'''
    with open(event_file) as f:
        return json.load(f)


@pytest.fixture()
def cfn_stack_name():
    '''Return name of stack to get Lambda from'''
    # FIXME: We should eventually read serverless.yml to figure it out.
    # Handling different environments would be good too.
    return 'aws-sns-to-slack-publisher-dev'


@pytest.fixture()
def lambda_client():
    '''Lambda client'''
    return boto3.client('lambda')


@pytest.fixture()
def lambda_function(cfn_stack_name):
    '''Return Lambda function name'''
    return '-'.join([cfn_stack_name, 'SlackPublish'])


def test_handler(lambda_client, lambda_function, event):
    '''Test handler'''
    r = lambda_client.invoke(
        FunctionName=lambda_function,
        InvocationType='RequestResponse',
        Payload=json.dumps(event).encode()
    )

    lambda_return = r.get('Payload').read()
    slack_response = json.loads(lambda_return).get('slack_response')

    assert slack_response.get('ok') is True

 

Now I run the integration test and see the following:

pytest -v tests/integration/
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.6.4, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /Users/tom/.pyenv/versions/3.6.4/envs/aws-sns-to-slack-publisher/bin/python3.6
cachedir: .pytest_cache
rootdir: /Users/tom/Source/serverlessops/aws-sns-to-slack-publisher, inifile: pytest.ini
plugins: pylint-0.9.0
collected 1 item

tests/integration/handlers/test_aws_sns_to_slack_publisher.py::test_handler PASSED                                                                    [100%]

================================================================= 1 passed in 2.46 seconds ==================================================================

 

I admittedly need to clean this up to scale this solution. If you’re adopting Lambda as an organization keep the following in mind. Every engineer will be deploying their personal copies of the services they work on to AWS. Engineers are going to need to test the interactions between the different pieces that are a part of the system. And, unlike previously with AWS EC2, you’re not paying for idle so there’s no budgetary concern to deny them.

Debugging

The application is now deployed and passing tests. But what do you do if you’re still having problems or you want to test specific behaviors before you add more tests? Serverless Framework makes debugging pretty easy. The two commands to use are:

What these two commands help you replace are logging into the AWS console to manually invoke your function and then fumbling through CloudTrail logs. In a low traffic development environment these commands will help you to find errors. If you’ve captured an event that triggered a failure you can now resend the event and experiment to find the cause of the failure.

I’m going to use my test event again to invoke the SlackPublish function while I tail the logs.  In the two videos below, one shows invoking the function while the other shows tailing logs during the invocation.

serverless invoke -f SlackPublish -p tests/events/aws_sns_to_slack_publisher.json
serverless logs -f SlackPublish -t

The combination of the serverless invoke and logs subcommands will aide you in debugging and understanding failures and behavior your tests did not initially catch.

Wrapping Up

At this point I'd be ready to push my new serverless services through my CI/CD pipeline. That's a post for another time though. You should however be comfortable now understanding the processes to follow when doing serverless development. Follow the steps in this series and you'll avoid a lot of frustration.

Find what you've just read useful? Want to use serverless more in your organization? Have a look at the DevOps transformation and AWS cloud advisory services ServerlessOps provides.
Beautiful divider with lightning bolt in the middle
«
»

Contact Us

×
Looking to get in touch with a member of our team? Simply fill out the form below and we'll be in touch soon!