This is Part 5 of the series Hello Serverless, a guide aimed at the individual or team that is building their first serverless application. This series addresses the common conceptual and tactical questions that occur during the serverless development and operational lifecycle. This post covers the Hello Serverless application’s evolution to an asynchronous event-driven application.

In the previous blog posts, Hello Serverless has been a RESTful web API whose create, retrieve, update, and delete operations are triggered via synchronous web requests. But the application doesn’t have to be built this way. Here’s a line in the application requirements from a previous post explaining what Hello Serverless needs to do:

“unlike the system(s) retrieving data, the system(s) creating, updating, and deleting data don’t need to be informed that the operation was completed successfully”

Based on that information, it makes sense for the application’s retrieve operation to continue to be a synchronous operation using AWS API Gateway triggering a Lambda function. But it means that we can rethink how create, update, and delete operations are performed. What we are going to do in this blog post is replace API Gateway with AWS SNS as the trigger for those operations and explore asynchronous event-driven serverless functions.

This is what the new asynchronous event-driven serverless functions design will look like, and the code for the application can be found here on GitHub.

This will be the final architectural evolution of Hello Serverless. In subsequent blog posts, we’ll explore how to solve common requirements, needs, and issues of a serverless application.

What Does Asynchronous Event-Driven Mean

Asynchronous event-driven architecture starts with an event that can be the result of an action occurring, a change in state, or a command submission. In the case of Hello Serverless, that could be a user sending a message, editing the previous message, or requesting a message to be deleted. An event-driven architecture is composed of three parts: an event producer, an event router, and an event consumer. A service produces an event and publishes it to the event router and then an independent service, decoupled from the producer, receives that event and then performs an action based on it.

How is this different from the previous iteration of Hello Serverless, a RESTful API using AWS API Gateway and Lambda? In that previous iteration, operations are handled synchronously. It’s up to the client calling the service to ensure the operation was completed successfully. If there’s a failure then it’s up to the client to perform retries. With asynchronous, you can think of the event producer as adopting a fire and forget mentality. It produces an event that it sends to the event router and its job is done. It’s now on the event consumer(s) to ensure that they properly handle the event, which means handling retries in the event of failure on their own.

The biggest benefit of this approach is the decoupling of event producers and consumers. Both need to be aware of the event router, but do not need to be aware of each other. In a non-event-driven architecture, to add a new feature or service, you would need to update a client to make requests to the new service. In an event-driven architecture, we don’t need to inform the event producer that a new event consumer exists. We just point the event consumer at the event router. This avoids the need to update an existing service to introduce a new one, both in terms of adding new code and potentially coordinating with a different team who owns that service, shortening development cycle time.

There are a few drawbacks to this architecture. The first is the possibility that an upstream service is unaware of all its downstream consumers. Try and deprecate an HTTP microservice and one of the tasks you will probably perform is tracking down all the clients that make requests to it. If you don’t have a system for easily tracking event consumers, you won’t have an ability to understand how your producer’s data is being used or what downstream effects might occur if a producer suffers issues. Event-driven architecture also requires different systems engineering skills than in other architectures. For example, event producers are not going to retry, and different event routers have different retry behavior. Event producers are solely responsible for successfully processing events. This combined with potentially more independent moving parts in an application means developers are going to need to utilize their systems engineering skills even more than previously required. Utilizing these skills isn’t a drawback, in fact it can even be characterized as a positive, but for some there will be a learning curve and ramp-up time required.

Creating Asynchronous Event-Driven Functions

Our application is structured no differently than it was in the previous blog post. What we’re going to do here is change the event source on create, update, and delete operations from API Gateway to AWS SNS, which is a managed publish-subscribe (pub/sub) messaging service. There are other event sources that we could use, such as AWS SQS or AWS Kinesis, but SNS fits our needs, which is passing events to subscribers with low latency that do not require being processed in order. It is possible for us to do an event-driven retrieve operation, but that complexity is beyond the scope of this blog post.

We should also point out that Hello Serverless isn’t a perfect example of an event-driven application. In our application, the event router is coupled with the event consumers. In an ideal event-driven architecture, the event router would be independent or possibly coupled with the event producer. If we were to repeat the pattern used by Hello Serverless to introduce an additional service, then we’d end up with a second event router which the same event producer would need to be aware of, thus defeating having event producers and consumers unaware of each other. However, what we have is fine for now and very possibly a pattern you will use as you begin to adopt event-driven architecture.

Application Initialization & Route Definitions

Now that we’re no longer using HTTP endpoints for create, update, and delete operations, what do application initialization and route definitions mean? In the previous blog post, we already mentioned that there is no application to initialize after having moved to using multiple serverless functions. Each serverless function just runs when triggered by an event source. What really matters is the event source and how we route events to the appropriate serverless function. And that brings us to route definitions. Up till now, routes had been HTTP endpoints configured on API Gateway. Send a request to an endpoint, and a serverless function executes. To move to an asynchronous event-driven architecture, we’ll replace API Gateway with an SNS topic, and instead of configuring HTTP routes, we’ll create SNS topic filter policies.

The SNS topic is an event router. Messages are published to the topic and are then delivered to subscribers, which may be Lambda functions, SQS topics, HTTP endpoints, or other possible subscriber types. In our case, those subscribers will be our application’s Lambda functions. Below is the SAM configuration to add the SNS topic to this application. We don’t even define any configuration for the SNS topic.

MessageItemTopic:
  Type: AWS::SNS::Topic

When we separated Hello Serverless into multiple serverless functions, we had to define API Gateway endpoints that would trigger each function individually instead of sending the event to every serverless function (a configuration I’m not even sure API Gateway will let you do but I’ve never bothered to try). By default, every SNS topic subscriber will receive every message and therefore each Lambda function would execute on each message. So, a message to create a message item would also trigger an update and a delete. That’s where SNS filtering policies come in.

To better understand SNS topic filtering, let’s first start by showing what an SNS event sent to a Lambda function looks like.

{
  "Records": [
    {
      "EventSource": "aws:sns",
      "EventVersion": "1.0",
      "EventSubscriptionArn": "arn:aws:sns:us-east-1::ExampleTopic",
      "Sns": {
        "Type": "Notification",
        "MessageId": "95df01b4-ee98-5cb9-9903-4c221d41eb5e",
        "TopicArn": "arn:aws:sns:us-east-1:123456789012:ExampleTopic",
        "Subject": "example subject",
        "Message": "{\"message\": \"hello world\"}",
        "Timestamp": "1970-01-01T00:00:00.000Z",
        "SignatureVersion": "1",
        "Signature": "EXAMPLE",
        "SigningCertUrl": "EXAMPLE",
        "UnsubscribeUrl": "EXAMPLE",
        "MessageAttributes": {
          "Action": {
            "Type": "String",
            "Value": "CREATE"
          }
        }
     }
  ]
}

The event contains a single record and under the JSON path Records[0].Sns, you’ll see the metadata about the SNS message, which includes the Message key that contains the message with the data that the CreateMessageItem Lambda function needs to create a message item. You’ll also see a key named MessageAttributes where an SNS message publisher (our event publisher) can define additional metadata. This message has an attribute named Action with a string value of CREATE, which is the metadata that we will use to filter messages to the correct function. Update messages will have an action of UPDATE, and delete messages will have an action of DELETE.

Here’s the configuration change we make to the CreateMessageItem Lambda function so that the function will be triggered by SNS and only by messages with metadata that indicate a “CREATE” operation.

--- serverless-hello-world-functions-py/template.yaml   2020-05-09 18:02:33.000000000 -0400
+++ serverless-hello-world-py/template.yaml     2020-05-23 20:52:45.000000000 -0400
@@ -93,12 +93,13 @@
         Variables:
           DDB_TABLE_NAME: !Ref DynamoDBTable
       Events:
-        ApiGateway:
-          Type: Api
+        SnsTopic:
+          Type: SNS
           Properties:
-            Path: /message
-            Method: POST
-            RestApiId: !Ref ApiGatewayApi
+            Topic: !Ref MessageItemTopic
+            FilterPolicy:
+              Action:
+                - CREATE

We’ve changed the event type from Api to SNS, and instead of configuring a REST API ID with a path and method, we configure the SNS topic and a filter policy that will only trigger this function if the Action message attribute has the value CREATE.

Request Logic & Business Logic

In the previous blog post, for each operation we combined the request and business logic into a single serverless function but used separate Python functions. The CreateServerlessMessageItem serverless function had a Python function named handler(), which pulled the message data out of the event and passed it to the _create_item() Python function, which in turn put that data into DynamoDB. Using two separate Python functions was on purpose, and I’ll argue a good practice to get into doing. By separating the two, we can change the handler to accommodate a different event source, or in some cases even add additional event source handling without having to touch the serverless function’s core business logic.

For CreateServerlessMessageItem to go from being triggered by API Gateway to being triggered by SNS this is the extent of the changes necessary in the function’s code.:

--- serverless-hello-world-functions-py/src/handlers/CreateMessageItem/function.py      2020-05-13 21:09:37.000000000 -0400
+++ serverless-hello-world-py/src/handlers/CreateMessageItem/function.py        2020-05-23 17:41:44.000000000 -0400
@@ -33,18 +33,9 @@

 def handler(event, context):
     '''Function entry'''
-    message = json.loads(event.get('body'))
+    message = json.loads(event['Records'][0]['Sns']['Message'])

-    message_id = _create_item(message)
-
-    body = {
-        'success': True,
-        'message': message_id
-    }
-    resp = {
-        "statusCode": 200,
-        "body": json.dumps(body)
-    }
+    resp = _create_item(message)

     return resp

We only made two changes. First, the shape of an API Gateway and SNS event are different so we updated the line of code that extracts the message data from the event. Second, we updated what data the handler() function returns. When this function was triggered by API Gateway data had to be returned by the function in a certain way. Since we’ve moved to SNS we don’t actually have to return anything but I do and I’ll get to why in a later blog post on testing.

Making This Production Ready

The Hello Serverless application has evolved from a Python Flask application to an asynchronous event-driven application, and the final architectural form the application will take. However, we’re not done if we want to make this a production-ready application.