Kick off async lambda processing from API Gateway

Asynchronous lambda invocation allows for a quick response back (e.g. 202 Accepted) while allowing the lambda to continue running. This would be desired if you wish to kick off a lambda that runs longer then the API GW 30 second timeout, or if the application needs a quick response without the need for direct success for failure of the longer running lambda.

At the time of this post this feature is not natively supported by serverless framework, but it looks trivial to set it up via a resource override. More information at https://github.com/serverless/serverless/issues/4862

Purchase my Serverless book at https://leanpub.com/serverless

Dynamodb ttl expiration is not minute precise

Finally figured out why some of my expired items fail to delete when expected. (bold mine)

DynamoDB typically deletes expired items within 48 hours of expiration. The exact duration within which an item truly gets deleted after expiration is specific to the nature of the workload and the size of the table. Items that have expired and not been deleted will still show up in reads, queries, and scans. These items can still be updated and successful updates to change or remove the expiration attribute will be honored.

Bottom line: you can't rely on DynamoDB's ttl column based expiration to do minute-precise expiration of things in your serverless app. Luckily for my purposes, I don't need minute precision. I do however need hour precision, so I'm planning to write a little scheduled lambda that kicks off every sixty minutes and reaps whatever records have expired but not deleted yet, and delete them manually.

Purchase my Serverless book at https://leanpub.com/serverless

Limitations of Dynamodb KeyConditionExpression

Dynamodb has a query system with a particularly hard learning curve (in my opinion.) Even though I've been using it for years now, I still run into some sticky problems like this one today when I tried to use an IN operator in a query expression. Should be fine, right?

    const key = {
      IndexName: 'by_label_and_status',
      KeyConditionExpression:
        'labelId = :labelId and submissionStatus in (:sent, :opened, :listened)',
      ExpressionAttributeValues: {
        ':labelId': event.pathParameters.labelId,
        ':sent': 'sent',
        ':opened': 'opened',
        ':listened': 'listened',
      },
    };
    const result = await dynamodb.query('submissions', key);

The code above actually fails. Dynamodb is kind enough to report Invalid operator used in KeyConditionExpression: IN, but finding the reason why took me a bit of digging.

It turns out that indeed if you go to the guide for Dynamodb expression syntax, you'll see that the allowed comparators for KeyConditionExpression are limited.

Valid comparisons for the sort key condition are as follows:

sortKeyName = :sortkeyval - true if the sort key value is equal to :sortkeyval.

sortKeyName < :sortkeyval - true if the sort key value is less than :sortkeyval.

sortKeyName <= :sortkeyval - true if the sort key value is less than or equal to :sortkeyval.

sortKeyName > :sortkeyval - true if the sort key value is greater than :sortkeyval.

sortKeyName >= :sortkeyval - true if the sort key value is greater than or equal to :sortkeyval.

sortKeyName BETWEEN :sortkeyval1 AND :sortkeyval2 - true if the sort key value is greater than or equal to :sortkeyval1, and less than or equal to :sortkeyval2.

begins_with ( sortKeyName, :sortkeyval ) - true if the sort key value begins with a particular operand. (You cannot use this function with a sort key that is of type Number.) Note that the function name begins_with is case-sensitive.

Use the ExpressionAttributeValues parameter to replace tokens such as :partitionval and :sortval with actual values at runtime.

Since the system I'm working on is brand new, I was able to get the code example working by changing the values of our statuses. Instead of having completely disparate statuses for sent, clicked, and opened, I changed them to be sent, sent_clicked, and sent_opened. They are semantically correct, since clicked and opened states are also sent. That change allowed us to use begins_with instead of the prohibited IN operator.

    const key = {
      IndexName: 'by_label_and_status',
      KeyConditionExpression:
        'labelId = :labelId and begins_with(submissionStatus, :submissionStatus)',
      ExpressionAttributeValues: {
        ':labelId': event.pathParameters.labelId,
        ':submissionStatus': 'sent',
      },
    };

Purchase my Serverless book at https://leanpub.com/serverless

Careful sending complex objects to lambda callback

Ran into an issue today with the following code. It's the second or third time it's bitten me, and I think this time I finally am learning the general guideline: don't pass complex objects into your lambda success/failure callbacks!

export async function main(event, context, callback) {
  const id = event.pathParameters.id;
  try {
    const result = await startExecution(
      process.env.statemachine_arn, JSON.stringify({ id })
    );
    callback(null, success({ status: true, result }));
  } catch (error) {
    console.log(error.message);
    callback(error, failure({ status: false, message: error.message }));
  }
}

This lambda will fail with Converting circular structure to JSON exception, because the result object returned from the startExecution function is a big complex object that doesn't work with JSON.stringify. I've had the same thing happen to me with the results of axios requests, as well as other AWS service calls.

To fix, don't pass result but instead pass only what you need returned, or simply true.

How to properly do SNS in Serverless

Figuring this out has taken a few hours out of my day.

Assumption

Your system is comprised of multiple CloudFormation stacks (aka Serverless Framework services aka microservices)

What not to do

functions:
  dispatcher:
    handler: dispatcher.dispatch
    events:
      - sns: 'demo_submitted'

It is critically important to not create SNS topic subscriptions on the fly as lambda event triggers. The reason is that Serverless framework tries to be helpful by implicitly auto-magically creating resource declarations for those topics, effectively within the boundaries of the consumer service. Unless parts of your microservice are talking amongst themselves using SNS (which would be weird), you usually subscribe to SNS topics that originate in other services, not the one where you are consuming it.

Why not to do it

Let's say you try to subscribe to the same topic with a lambda in another service OR you go down the road of explicitly declaring topics as resources in the services that broadcast to the topics AND you've implicitly created those very same SNS topics already.

Then when you try to deploy you will get an error that looks something like this:

Serverless Error ---------------------------------------
  An error occurred: DemoSubmittedTopic - demo_submitted already exists in stack arn:aws:cloudformation:us-east-1:423:stack/producers-service-dev/5332a.

Resources must be unique and in this case, you already created that SNS topic in the consumer, not where it belongs and is expected to live.

What to do instead

When you subscribe a lambda to an SNS topic, use the ARN notation as described in https://serverless.com/framework/docs/providers/aws/events/sns#using-a-pre-existing-topic

functions:
  dispatcher:
    handler: dispatcher.dispatch
    events:
      - sns:
          arn: arn:xxx

The ARN notation in the subscription prevents the auto-magical undesirable creation of an SNS resource.

Vanity columns in join tables

Today as I was looking at a table in DynamoDB via the AWS console. This table is basically a queue of items for a particular user to look at or dismiss, and right now it only has two key columns: one for the user, and one for the item.

It struck me that it would be worthwhile to add a few extra columns to this table just to make administration and debugging a little easier. Namely, we could add the name of the user and the name of the item. At worst the tradeoff is that the read/write throughput for the table would be a little higher.