Lambda on Geoff Baskwill

Using Step Functions to build CloudFormation custom resources

Sat, 24 Apr 2021 00:00:00 +0000

Photo by Serban Mestecaneanu on Unsplash

My team uses AWS CloudFormation to provision our cloud infrastructure using code. Most of the time we can get what we need with the set of resources that AWS provides.

However, sometimes CloudFormation support for a service or a particular feature takes a while to arrive, and we need to fill in the gap ourselves. CloudFormation gives us the ability to fill these gaps by building “custom resources” that can literally run any logic you need in AWS Lambda, and we’ve used these when we needed to.

Sometimes Lambda isn’t the right answer

Sometimes even Lambda fails us, though, as some resources can potentially take a long time to set up, and we don’t particularly want to have a Lambda function sitting idle or potentially timing out halfway through the resource setup.

The great thing about Lambda functions is that you only pay for them when they’re running. This makes them great for places where you need to run a quick task or handle an API request. You lose some of the benefits when your Lambda function looks like this:

start_something()

while not_done():
    time.sleep(30)

finish()

because you’re paying for the Lambda to run while you’re waiting for your operation to complete. Worse, Lambda functions have a maximum lifetime of 15 minutes, so if your process takes longer than 15 minutes, you have to do weird hacks to make it work with a pure Lambda solution.

When we found out that CloudFormation didn’t have support for creating DynamodDB Global Tables, our first thought was “we know how to do this, a Lambda function custom resource can handle it.” However, as we dug into the details and tried it out, we quickly learned that we could get into a scenario where creating the initial replica set or updating the replicas could easily exceed the 15-minute Lambda timeout.

Step Functions to the rescue!

Here’s where AWS Step Functions comes in. Step Functions make the task of orchestrating processes easier. They have built-in support for looping, waiting, and integrating with different functions and services, which makes them perfect for this sort of thing.

One of our team members put together this Step Function definition for creating a DynamoDB global table. It starts out by checking the state of the table, waiting until the table is ready for updates, then comparing the set of replicas with the desired set. You can only add one replica at a time, so the step function repeats the UpdateReplicas step until the actual state matches the desired state. Each step is very small and self-contained, usually only one or two API calls, and all of the waiting is done by Step Functions instead of in the Lambda function, so we’re not paying for idle time! Best of all, Step Functions can run for up to a week, so we didn’t need to worry about the 15-minute timeout any more.

There’s a small catch…

I wish I could use Step Functions directly to build CloudFormation custom resources instead of having to have a Lambda function that triggers the Step Function. #awswishlist
— Geoff Baskwill (@geoff_baskwill) February 24, 2021

CloudFormation doesn’t support direct integration with Step Functions as a custom resource provider yet, but we can use our old Lambda function trick to trigger the Step Function execution, and send the response back to CloudFormation when we get to an end state.

Wrapping up

When you love infrastructure-as-code and need a custom resource for something that CloudFormation doesn’t support, Lambda is usually a great solution. When you need a bigger hammer for complex orchestration or operations with lots of idle time, Step Functions can help get you there.

The goal is to retire this particular resource soon, as AWS tells us that they’ll have built-in support in CloudFormation for DynamoDB Global Tables in the very near future. That said, my team is happy that we were able to deliver an initial implementation with this workaround and provide value to our customers!

Behaviour of background goroutines in AWS Lambda

Mon, 02 Nov 2020 00:00:00 +0000

I was curious what would happen with a time.Ticker running in an AWS Lambda function.

It was pretty obvious that the ticker wouldn’t tick while the container was hibernating, but what about when it woke up? Would a bunch of ticks happen if the container had been hibernating for a long time? Would you get a tick at all?

tl;dr

It looks like you get one tick when the function comes out of hibernation (assuming the ticker duration has passed at least once), and if the function never hibernates the ticks happen on time, which for my purposes is pretty much perfect.

The code

I set up a small AWS Lambda function that responds to API Gateway requests and has a goroutine running with a time.Ticker that ticks every minute:

package main

import (
  "context"
  "fmt"
  "net/http"
  "sync"
  "time"

  "github.com/aws/aws-lambda-go/events"
  "github.com/aws/aws-lambda-go/lambda"
)

func main() {
  wg := &sync.WaitGroup{}

  ticker := time.NewTicker(time.Second * 60)
  go func() {
    for t := range ticker.C {
      // let the WaitGroup know that we're working
      wg.Add(1)

      // tell the observer that the ticker actually ticked
      fmt.Println("tick at", t)
      
      // simulate a background process that takes a bit to complete
      time.Sleep(time.Second)
      
      // let the WaitGroup know that we're done
      wg.Done()
    }
  }()

  lambda.Start(func(ctx context.Context, req events.APIGatewayProxyRequest)(events.APIGatewayProxyResponse, error) {
    resp, err := realHandler(ctx, req)
    
    // If the ticker is running, wait for it.
    wg.Wait()

    return resp, err
  })
}

func realHandler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
  return events.APIGatewayProxyResponse{
    StatusCode: http.StatusPaymentRequired,
  }, nil
}

You’ll see how I’ve split out the real handler and the ticker-waiting “middleware” … this keeps the handler logic separated from the bit that waits for the background goroutine to complete what it’s doing before returning.

The test

I called the function a couple of times, waited a few minutes, and called the function again. The logs told me what happened:

2020-11-02T14:11:51.061-05:00 START RequestId: f2604bb1-8bc2-445a-b7b2-3893d8d3416d Version: $LATEST
2020-11-02T14:11:51.063-05:00 END RequestId: f2604bb1-8bc2-445a-b7b2-3893d8d3416d
2020-11-02T14:11:51.063-05:00 REPORT RequestId: f2604bb1-8bc2-445a-b7b2-3893d8d3416d Duration: 1.39 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 35 MB Init Duration: 66.79 ms XRAY TraceId: 1-5fa059f6-10e9a1715cb581111721b9a8 SegmentId: 0467a00a5d719145 Sampled: true

2020-11-02T14:11:54.171-05:00 START RequestId: 7a39872e-4b18-42da-b2ff-315a6590a1b6 Version: $LATEST
2020-11-02T14:11:54.174-05:00 END RequestId: 7a39872e-4b18-42da-b2ff-315a6590a1b6
2020-11-02T14:11:54.174-05:00 REPORT RequestId: 7a39872e-4b18-42da-b2ff-315a6590a1b6 Duration: 0.68 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 42 MB XRAY TraceId: 1-5fa059fa-0be52e2845702dd0365eeb74 SegmentId: 5d5f37f527b7de1a Sampled: true

The first two log blocks show the first two calls, at 14:11:51-05:00 and at 14:11:54-05:00. We wouldn’t expect to see a tick log here, and as expected we do not.

2020-11-02T14:16:28.076-05:00 START RequestId: 488123cd-43bc-4567-9970-00c35bfc91b8 Version: $LATEST
2020-11-02T14:16:28.076-05:00 tick at 2020-11-02 19:16:28.076072523 +0000 UTC m=+277.052607202
2020-11-02T14:16:28.078-05:00 END RequestId: 488123cd-43bc-4567-9970-00c35bfc91b8
2020-11-02T14:16:28.078-05:00 REPORT RequestId: 488123cd-43bc-4567-9970-00c35bfc91b8 Duration: 0.59 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 42 MB XRAY TraceId: 1-5fa05b0c-35afecef72dc3fb25cf4eeef SegmentId: 045243422ccaea72 Sampled: true

2020-11-02T14:20:15.133-05:00 START RequestId: 2dc415ef-f53f-4e35-865d-21b89cd9147e Version: $LATEST
2020-11-02T14:20:15.133-05:00 tick at 2020-11-02 19:20:15.133194003 +0000 UTC m=+504.107026404
2020-11-02T14:20:15.137-05:00 END RequestId: 2dc415ef-f53f-4e35-865d-21b89cd9147e
2020-11-02T14:20:15.137-05:00 REPORT RequestId: 2dc415ef-f53f-4e35-865d-21b89cd9147e Duration: 0.62 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 43 MB XRAY TraceId: 1-5fa05bef-5e8453ea22ff3fa4454e0af6 SegmentId: 183982d42c0b3848 Sampled: true

The next two log blocks show calls at 14:16:28-05:00 and 14:20:15-05:00 and one tick happening each time the code is invoked. This would seem to indicate that even if multiple ticks would normally have happened in the time period, they are consolidated into one.

Neat!