Cloudformation on Geoff Baskwill

Custom CloudFormation resource types

Mon, 13 Nov 2023 00:00:00 +0000

I recently had the opportunity to explore building custom CloudFormation resource types as a mechanism to support a stream-aligned team. It was intriguing to play with this new-to-me tool and see how it could reduce complexity for the team.

I’ve written before about using Step Functions to build custom resources in CloudFormation. This builds on the old-school custom resources mechanism where you provide a Lambda handler as part of your stack. This is convenient for very small or one-off duct-tape situations, but if you need more complex resource logic or if you want to provide a simple resource interface to a bunch of teams, there is a better way.

The AWS CloudFormation registry and the CloudFormation command-line interface are this better way. They allow you to write your own resource provider that will be exposed using the exact same mechanisms that CloudFormation uses for many AWS-provided resource types.

You can do all kinds of super-neat things with resource types, and there is a neat pattern to report progress when your resource may take a while to create or when there are multiple steps to create your resource.

Batteries not always included

Resource providers can expose configuration data; this is super-handy if you want to attach “account-level” configuration (really region-level configuration) to your type’s handler. Unfortunately, while an API exists to programmatically set the configuration data, tragically and ironically there is no CloudFormation support for setting the configuration data on a resource type.

Because of the super-cool open way that AWS CloudFormation resource providers are built, a person could theoretically open a pull request on the aws-cloudformation/aws-cloudformation-resource-providers-cloudformation repository, duplicating the aws-cloudformation-hooktypeconfig folder and refactoring it to implement AWS::CloudFormation::ResourceTypeConfig, but I haven’t gone there yet. Probably should open a roadmap issue for discussion first.

I used an old-school custom resource to set the configuration for my custom types. 🤦‍♂️

Using Step Functions to build CloudFormation custom resources

Sat, 24 Apr 2021 00:00:00 +0000

Photo by Serban Mestecaneanu on Unsplash

My team uses AWS CloudFormation to provision our cloud infrastructure using code. Most of the time we can get what we need with the set of resources that AWS provides.

However, sometimes CloudFormation support for a service or a particular feature takes a while to arrive, and we need to fill in the gap ourselves. CloudFormation gives us the ability to fill these gaps by building “custom resources” that can literally run any logic you need in AWS Lambda, and we’ve used these when we needed to.

Sometimes Lambda isn’t the right answer

Sometimes even Lambda fails us, though, as some resources can potentially take a long time to set up, and we don’t particularly want to have a Lambda function sitting idle or potentially timing out halfway through the resource setup.

The great thing about Lambda functions is that you only pay for them when they’re running. This makes them great for places where you need to run a quick task or handle an API request. You lose some of the benefits when your Lambda function looks like this:

start_something()

while not_done():
    time.sleep(30)

finish()

because you’re paying for the Lambda to run while you’re waiting for your operation to complete. Worse, Lambda functions have a maximum lifetime of 15 minutes, so if your process takes longer than 15 minutes, you have to do weird hacks to make it work with a pure Lambda solution.

When we found out that CloudFormation didn’t have support for creating DynamodDB Global Tables, our first thought was “we know how to do this, a Lambda function custom resource can handle it.” However, as we dug into the details and tried it out, we quickly learned that we could get into a scenario where creating the initial replica set or updating the replicas could easily exceed the 15-minute Lambda timeout.

Step Functions to the rescue!

Here’s where AWS Step Functions comes in. Step Functions make the task of orchestrating processes easier. They have built-in support for looping, waiting, and integrating with different functions and services, which makes them perfect for this sort of thing.

One of our team members put together this Step Function definition for creating a DynamoDB global table. It starts out by checking the state of the table, waiting until the table is ready for updates, then comparing the set of replicas with the desired set. You can only add one replica at a time, so the step function repeats the UpdateReplicas step until the actual state matches the desired state. Each step is very small and self-contained, usually only one or two API calls, and all of the waiting is done by Step Functions instead of in the Lambda function, so we’re not paying for idle time! Best of all, Step Functions can run for up to a week, so we didn’t need to worry about the 15-minute timeout any more.

There’s a small catch…

I wish I could use Step Functions directly to build CloudFormation custom resources instead of having to have a Lambda function that triggers the Step Function. #awswishlist
— Geoff Baskwill (@geoff_baskwill) February 24, 2021

CloudFormation doesn’t support direct integration with Step Functions as a custom resource provider yet, but we can use our old Lambda function trick to trigger the Step Function execution, and send the response back to CloudFormation when we get to an end state.

Wrapping up

When you love infrastructure-as-code and need a custom resource for something that CloudFormation doesn’t support, Lambda is usually a great solution. When you need a bigger hammer for complex orchestration or operations with lots of idle time, Step Functions can help get you there.

The goal is to retire this particular resource soon, as AWS tells us that they’ll have built-in support in CloudFormation for DynamoDB Global Tables in the very near future. That said, my team is happy that we were able to deliver an initial implementation with this workaround and provide value to our customers!

CloudFormation, Route 53, and ... EKS?

Thu, 22 Apr 2021 00:00:00 +0000

The other day I got halfway through writing a very irate support ticket to AWS, stopped to do some fact checking, and learned something deeply annoying.

One of the teams I work manages a bunch of services. One of these “services” is some common Amazon Route 53 infrastructure that is set up using AWS CloudFormation, and over the history of the project the deployment in the development account the team uses has been a little flaky. Each time we hit a deployment problem, the problem always turned out to be rate limiting. It never happened in production, so the flakiness didn’t quite get the attention that it could have.

Rate limiting in Route 53

Some background: Route 53 has a hard limit of 5 control plane requests per second per AWS account. For most folks, this is fine. However, it wasn’t working well for this team.

The team had raised support tickets and gotten advice like “CloudFormation will attempt to create your resources in parallel. One option to avoid rate limiting is to add DependsOn links to serialize the resource creation.” We weren’t super-happy with that answer, and there’s a CloudFormation roadmap item to fix it, but we needed something in the interim, and it worked … mostly.

On this day a deployment had failed again, and after going after the usual suspects and making sure that the resources were properly serialized, I got very irate. I was halfway through writing a support ticket:

We are still encountering Route 53 rate limits and our CloudFormation stack deployment / updates are intermittently failing, sometimes after only two resources are created. There are no Route 53 API calls being made by our applications, only through CloudFormation.

We are quite frustrated at this point and would like to request a session with a solution architect to help us understand how we should be doing this and

and I paused.

Check yourself before you wreck yourself

“There is no point in using the word ‘impossible’ to describe something that has clearly happened.” — Douglas Adams, Dirk Gently’s Holistic Detective Agency

“Is it true that there are no other Route 53 API calls being made?” I asked myself. A quick jaunt into AWS CloudTrail told me the answer, and also opened a gaping pit beneath my feet.

There were 436 Route 53 API calls made in the 2-minute period surrounding our CloudFormation failure. If you do the math, that’s 3.6 requests per second on average, so it’s not at all surprising that we maybe tipped over the limit of 5 at some point in there.

“But where are these coming from?” was my immediate question, and it was immediately answered.

Virtually all of these requests were being made by an EC2 instance that was part of an Amazon Elastic Kubernetes Service (EKS) cluster.

Talking through this with some other folks, I learned that they’d configured external-dns on the cluster, and that this behaviour is actually documented.

The production account doesn’t have the EKS cluster, so it’s not overwhelmed with Route 53 API calls, which explains why deployment never failed there.

Buh-bye

I wanted to decommission the cluster immediately, but unfortunately some teams still need it, so I wasn’t able to.

The external-dns documentation says that one workaround for the issue of the controller eating your entire Route 53 request budget is to extend the interval that the controller reconciliation loop runs at. In this particular cluster, the reconciliation loop was running every minute (the default!) to reconcile a set of records that change approximately never. I followed the instructions, set the interval to a week, and settled in to see what happened.

The first thing I noticed was that immediately the calls to Route 53 stopped. Not surprising, but great to see the confirmation. Several hours after the change, there were still no calls from the previously-misbehaving cluster.

All is well now, and I get to put away my detective hat for another day.

“The light works,” he said, indicating the window, “the gravity works,” he said, dropping a pencil on the floor. “Anything else we have to take our chances with.” — Douglas Adams, Dirk Gently’s Holistic Detective Agency

What I learned

First, CloudTrail was instrumental here. I’m still a novice, but I’m learning how powerful a tool it is. Once I knew what to look for, it was immediately obvious what the source of the rate limiting was. The events in CloudTrail identified the EC2 instance and even made it clear that the source of the requests was in an EKS cluster.

Second, I was reminded that Kubernetes is not a get-out-of-ops-free card. There is a lot of expertise involved in running Kubernetes well, even when you’re using a managed service like EKS. I knew this before, but this was an example of a cluster I didn’t even know existed (don’t worry: someone more responsible did know!) having side effects way outside its scope.