Geoff Baskwill

Custom CloudFormation resource types

Mon, 13 Nov 2023 00:00:00 +0000

I recently had the opportunity to explore building custom CloudFormation resource types as a mechanism to support a stream-aligned team. It was intriguing to play with this new-to-me tool and see how it could reduce complexity for the team.

I’ve written before about using Step Functions to build custom resources in CloudFormation. This builds on the old-school custom resources mechanism where you provide a Lambda handler as part of your stack. This is convenient for very small or one-off duct-tape situations, but if you need more complex resource logic or if you want to provide a simple resource interface to a bunch of teams, there is a better way.

The AWS CloudFormation registry and the CloudFormation command-line interface are this better way. They allow you to write your own resource provider that will be exposed using the exact same mechanisms that CloudFormation uses for many AWS-provided resource types.

You can do all kinds of super-neat things with resource types, and there is a neat pattern to report progress when your resource may take a while to create or when there are multiple steps to create your resource.

Batteries not always included

Resource providers can expose configuration data; this is super-handy if you want to attach “account-level” configuration (really region-level configuration) to your type’s handler. Unfortunately, while an API exists to programmatically set the configuration data, tragically and ironically there is no CloudFormation support for setting the configuration data on a resource type.

Because of the super-cool open way that AWS CloudFormation resource providers are built, a person could theoretically open a pull request on the aws-cloudformation/aws-cloudformation-resource-providers-cloudformation repository, duplicating the aws-cloudformation-hooktypeconfig folder and refactoring it to implement AWS::CloudFormation::ResourceTypeConfig, but I haven’t gone there yet. Probably should open a roadmap issue for discussion first.

I used an old-school custom resource to set the configuration for my custom types. 🤦‍♂️

About me

Sun, 29 Oct 2023 00:00:00 +0000

Hi? If you’re here and not a robot, a) wow; b) ok? I’m flattered and deeply curious about what brings you to this little corner of the Internets.

Most of what I do day-to-day is not shareable or even relevant outside a very tightly-scoped context, but sometimes I get to roll some thoughts together into a post.

I hope that what you’ll find here is useful or interesting, though the real point of this site is to keep a bunch of the shareable material I’ve created in one place for easier reference.

Work connections

I’m currently employed at Tackle.io, where my role is “software architect”, which I see as more “enabling” than “stream-aligned” if you are familiar with Team Topologies.

This is a super-fun role but also super-hard to nail down concisely. You might get a decent idea from the posts I’ve made here. One important aspect is that I do not speak on behalf of Tackle. For any official commentary, you really need to go elsewhere.

You can find me on LinkedIn if you’re curious about my career path, though I’ll warn you that it’s pretty sparse over there. If you’re looking for details, the best way is to get in touch directly and we can have a chat.

Where I live

I live just outside of Ottawa, Ontario, Canada, on the ancestral and unceded territory of the Algonquin Anishinaabe Nation.

Fun times with AWS STS AssumeRole

Mon, 21 Aug 2023 00:00:00 +0000

I had an AWS STS AssumeRole problem today when I was working with the AWS SDK; digging in it turned out the problem was that AssumeRole was not actually being invoked. Poking around I was unsurprised to find that @ben11kehoe had already clearly stated the problem and GitHub user petitout had example Go code with the right way to do it (link).

Sometimes you need to assume another role than your default execution role in order to accomplish a task. You can explicitly call STS AssumeRole yourself and configure the SDK client to use the returned credentials, or you can configure your SDK client to call AssumeRole itself when it needs the credentials. I like the second way because it means that credentials aren’t something that my code has access to.

Sample code for not assuming a role correctly; you might have stumbled on this the same way I did:

// almost certainly not what you want
cfg, _ := config.LoadDefaultConfig(
	ctx,
	config.WithAssumeRoleCredentialOptions(func(aro *stscreds.AssumeRoleOptions) {
		aro.Client = stsClient
		aro.RoleARN = arn
		aro.RoleSessionName = sessionName
	}),
)

A person might think that the code above is right, because it has all kinds of the right keywords that trigger pattern recognition. The comments on WithAssumeRoleCredentialOptions read:

// WithAssumeRoleCredentialOptions is a helper function to construct functional
// options that sets a function to use stscreds.AssumeRoleOptions on config's
// LoadOptions. If assume role credentials options is set to nil, the assume
// role credentials value will be ignored. If multiple WithAssumeRoleCredentialOptions
// calls are made, the last call overrides the previous call values.

I think that what this actually does is … you know, I truly have no idea, and don’t intend to dig into what exactly this does. It definitely does not do what I thought it did.

Sample code for probably doing what you want:

// almost certainly what you do want instead
cfg, err := config.LoadDefaultConfig(
	ctx, 
	config.WithCredentialsProvider(stscreds.NewAssumeRoleProvider(
		stsClient, arn, func(aro *stscreds.AssumeRoleOptions) {
			aro.RoleSessionName = sessionName
		}
	))
)

When the SDK needs credentials, this setup uses STS AssumeRole to obtain credentials with the STS client and role ARN that you provide. The stsClient in this example can be configured to take credentials from the environment.

Hope posting this helps make the solution findable for other folks.

re-posting and expanding on my original Mastodon post

SQS, DLQs, and KMS, oh my

Mon, 06 Mar 2023 00:00:00 +0000

Recently one of the teams I work with had a fun time (note: it was not fun in the moment) with Amazon Simple Queue Service (SQS), dead-letter queues (DLQs), and AWS Key Management Service (KMS).

I thought I’d share because we learned something pretty important.

If you’re new to SQS, you may be kind of aware that it’s a generalized queue service that lets you post messages onto the queue and then process queued messages in a handler, kind of like this:

There is an event source mapping in between the queue and the handler; we’re going to gloss over that detail for a minute.

SQS is super-handy, reliable, and amazingly scalable; AWS guides us to use it anywhere that we need to buffer communications between components of a solution.

There’s even a mechanism to deal with failures. You can configure a policy on your queue so that if SQS fails to deliver¹ a message to the handler enough times, the message gets moved to another queue for special handling. This second queue is called a “dead-letter” queue, named after the post office concept where messages that were undeliverable got sent to a special place for handling.

¹ “SQS fails to deliver” is not technically accurate, as SQS does not deliver messages, rather the event source mapping retrieves the message, attempts to deliver it to the Lambda function, and deletes the message if the message was handled successfully. If there was an error retrieving or processing the message, SQS will make the message available again after the visibility period expires. Eventually the message will expire, either after a specified time period or after a specified number of retrievals from the queue.

You may have a mental model of this setup that looks a bit like this:

SQS also lets you encrypt messages in the queue; you can use an AWS-managed key or a customer-managed key. Our teams always use customer-managed keys, so we have an SQS queue with a KMS CMK and DLQs (say that three times fast!). The updated model looks a bit like this:

If you’re an SQS expert, someone who has read the documentation carefully, or someone who read very carefully through the paragraphs above, you will instantly see the problem with this picture. If you didn’t catch it, you are not alone.

The key phrasing above is that undeliverable messages are moved from the primary queue to the dead-letter queue. They’re not retrieved from the primary queue and then sent onto the dead-letter queue, they’re moved. This isn’t explicitly stated in the documentation we have been able to find, but it’s clear from the behaviour we experienced and a couple of other hints in the documentation.

Our team discovered that the DLQ handler wasn’t processing messages, and in fact its event source mapping (the invisible thing we agreed to gloss over) was entirely disabled. An extremely-helpful AWS support person was able to tell us why.

When messages are put onto a primary queue that has KMS encryption enabled, they are encrypted using that queue’s KMS key. The handler must also have permissions to decrypt messages using the key, and we had that all working perfectly.

The Lambda function does not use the decryption permission itself, but rather “loans” that permission to the event source mapping. In our case, the Lambda function’s execution role and therefore event source mapping did not have sufficient permissions to decrypt the message, so the mapping failed to decrypt the message after retrieving it from the DLQ and did not invoke the Lambda function.

If you have a flawed understanding of the DLQ like we did, you might expect that messages would be decrypted and then re-encrypted when they are sent on the DLQ. However, they’re not decrypted and re-encrypted. They’re not even sent on the DLQ. They’re moved from the primary queue to the DLQ.

We had our DLQ set up with a separate key, and the DLQ handler had permissions to use the key, and in tests where we sent messages to the DLQ the handler worked perfectly. However, in a full end-to-end scenario where messages fell off the primary queue and were moved to the DLQ, the event source mapping failed to decrypt the message (it was set up with permissions on the DLQ key, not the primary queue key) and did not invoke the handler.

AWS is super-smart, and they know not to try things forever when there’s persistent failure. The event source mapping eventually disables itself and emits a CloudTrail event when it does so. If you’re vigilant or, preferably, if you set up a CloudWatch alarm based on the ApproximateAgeOfOldestMessage metric on the DLQ, you’ll be able to detect that messages are queuing up in the DLQ and not getting handled.

It’s important to remember that because messages are moved and not re-sent on the DLQ, your ApproximateAgeOfOldestMessage does not reflect the time that the message is in the DLQ but rather the time since the message was originally sent on the primary queue. This is also an important thing to remember when you are configuring your DLQ: if you use the same policy to age messages out of your DLQ as you have in your primary queue, you may find that messages disappear without getting handled the way you expected.

I did mention that AWS documents the requirement to use the same key:

Your source queue and its corresponding dead-letter queue need to share the same KMS key.

Oops. 😳

We’re looking into creating a cfn-lint custom rule to prevent us from tripping over this issue again.

re:Inforce 2022 - IAM306: Managing B2B identity at scale

Wed, 27 Jul 2022 00:00:00 +0000

I had the amazing opportunity to co-present a session at AWS re:Inforce 2022. My co-presenter was Suresh Sridharan, a product manager on the Amazon Cognito team.

If you’re not familiar with AWS re:Inforce, it’s an annual Amazon Web Services conference focused on security. It’s quite a bit smaller than re:Invent, their conference targeting builders, but that is actually a massive draw for me because I’ve found that re:Invent has gotten a bit big. I appreciate the focus and the relative ease of navigating a single site rather than what seems to be the entire Las Vegas Strip.

Suresh talked about a variety of options that Cognito customers have for managing multi-tenant solutions, and I talked about the particular choices that our team has made and what took us down this path.

For the full details, check out the slides and the recording.

Enriching Amazon Cognito features with an Amazon API Gateway proxy

Wed, 30 Mar 2022 00:00:00 +0000

I read a great post about protecting Amazon Cognito with an Amazon CloudFront proxy and wondered if using an Amazon API Gateway proxy instead would work better.

It really really did. Being able to use the full power of Lambda was fantastic, and the performance was improved significantly.

This approach also let us quickly implement a few tiny features that Cognito doesn’t have yet, like server-side attribute validation and user lifecycle events.

I got in touch with the author of the original post and he was generous enough to offer a collaboration on a follow-up post on the AWS architecture blog.

Check out the full post for all the details.

Sustainability and systemic change

Mon, 07 Jun 2021 00:00:00 +0000

Photo by Ross Findon on Unsplash

If you’re here looking for answers, I’m afraid I don’t have any. I do have questions, though, and maybe by putting them out there we can find a next step on the path forward together.

I have the great fortune of working with Madeline Van Der Paelt, and she sent me a note after a particularly-frustrating meeting recently. It was a quick one-liner, “I’m curious what you thought of the meeting this afternoon.” That small prompt turned into an hour-long conversation about how so many of our efforts to Do Better™ fail, and why that might be.

I think that economics is a contributing factor.

Behavioural economics

Behavioural economics tells us that rational people usually make rational decisions based on the information and incentives presented to them. (Important note: I am not a behavioural economist and have no training in the field, I could be completely wrong in my understanding.)

As a case in point, Chelsea Troy has a great article Why your efforts to make your company inclusive aren’t working. In it, she says:

[E]veryone kind of thinks they are paying attention. But they ain’t. And the reason they ain’t is that, usually, the information has no bearing on their career trajectory.

Bias trainings are interesting, but after they’re over employees don’t focus there because they’re not evaluated on their inclusion literacy. Instead, employees focus on the skills they need to get ahead—writing code, appeasing the boss, establishing influence.

Individual people treat inclusion like an elective because the company’s incentive system treats inclusion like an elective.

Replace “bias training” with “security training”, “architecture reviews”, or pretty much anything that is not explicitly on a team’s quarterly objectives, and we often see the same pattern. With few exceptions, quarterly objectives win, because that is what all of our incentives are tied to. In some ways, that’s kind of the point of those objectives in the first place: providing folks with the tools to focus on what’s important and rewarding people for meeting goals. However, we need to make sure that we’re making conscious decisions about what we’re leaving off the list, and ask ourselves whether we’re making the right choices.

It’s important to remember that lots of organizational values will never show up on an OKR (Objectives and Key Results) list, because they’re supposed to be overriding concerns that everyone knows to do. However, sometimes in the effort of pursuing an OKR, or when working with a rapidly-growing team, we can forget some of those fundamental values.

When this happens, we can end up with the same results that Chelsea describes: a flurry of activity to check off a box, with no follow-up or real systemic improvement. An hour of training doesn’t address systemic problems: we need to change the system.

Our objectives reflect what we think is important

It seems silly and obvious to point this out, but our objectives are the definition of what we think is important. In the presence of infinite resources, we can do everything, but in more realistic situations, we often have to make trade-offs.

In order for us to make real change that sticks, we need to make sure that our objectives reflect not only “Ship feature X and get feedback from N customers” but also the other activities that are important to running the business successfully, while being decent humans.

Mechanisms are a key to success

Adrian Hornsby, a principal engineer at Amazon Web Services, writes about how Amazon achieves operational excellence through mechanisms that encode their processes. He quotes Thomas Edison, saying:

A good intention, with a bad approach, often leads to a poor result.

We can have all the good intentions in the world, but unless we have supporting processes in place, it can be very difficult to make those intentions reality.

In the article, Adrian talks about the virtuous cycle of tools, adoption, and audits. I’m extrapolating a bit, but my understanding of the essence is: Create a tool, encourage adoption through education, and audit its use. Use the audit results to improve the tool and increase adoption.

Making change sustainable

There are so many things that I can be better at, and I often feel so overwhelmed that I … do nothing. I fall into the trap of wanting to be perfect and I let that get in the way of being better. One motivational image I’ve seen that sometimes helps me get past being overwhelmed has an equation:

1.01³⁶⁵ = 37.8

If you’re not mathematically inclined, this might look a bit confusing, but it says “if you take a number greater than one, even a little bit, and multiply it by itself a lot, you get a much bigger number”. If you improve a little bit every day, at the end of a year you will be a lot better.

What mechanism can I use to avoid being overwhelmed and trick myself into making that 1% daily improvement that will compound over years?

Can we nudge ourselves towards the changes we want?

I have a list of questions that I ask myself every week. It’s a tool I picked up from working with Madeline and her team. Their questions are a bit different, but mine are:

What did you do for our customers this week?
What did you do for other Trenders this week?
What did you do for yourself this week?

Every week I open a new document, paste in these questions, and as the week progresses I add answers. I find it helps me keep my head in the right place.

I wonder what would happen if I set up a bot to nudge me with an extra question, maybe a new one each week, that helped remind me of the behaviours I want to improve? What if I took all of the items in Chelsea Troy’s rubric for inclusiveness, turned them into “how did you demonstrate X” questions, and used a different question each week to help me remember to be more inclusive? What if I added in questions for other behaviours? There are all kinds of examples:

How did you build up a team’s capacity to be more self-sufficient this week?
How did you help someone become a better code reviewer this week?
How did you help a team improve their operational effectiveness this week?

These work well for my role, your set of prompts might be different!

I’m going to give it a try

We’ve got a hack day coming up, so I think my project is going to be a nudge-bot that sends me a question of the week to add to my list. I’m curious to see how it works out. I’m hopeful that this mechanism will help me make small sustainable changes in my behaviour and surface some of the things that aren’t always included in the big-O Objectives.

What questions do you think I should ask? What behaviours are you looking to amplify in your teams and organizations? I’d love to hear your thoughts.

Many thanks to Madeline Van Der Paelt for her thoughts on the initial draft of this post.

Using Step Functions to build CloudFormation custom resources

Sat, 24 Apr 2021 00:00:00 +0000

Photo by Serban Mestecaneanu on Unsplash

My team uses AWS CloudFormation to provision our cloud infrastructure using code. Most of the time we can get what we need with the set of resources that AWS provides.

However, sometimes CloudFormation support for a service or a particular feature takes a while to arrive, and we need to fill in the gap ourselves. CloudFormation gives us the ability to fill these gaps by building “custom resources” that can literally run any logic you need in AWS Lambda, and we’ve used these when we needed to.

Sometimes Lambda isn’t the right answer

Sometimes even Lambda fails us, though, as some resources can potentially take a long time to set up, and we don’t particularly want to have a Lambda function sitting idle or potentially timing out halfway through the resource setup.

The great thing about Lambda functions is that you only pay for them when they’re running. This makes them great for places where you need to run a quick task or handle an API request. You lose some of the benefits when your Lambda function looks like this:

start_something()

while not_done():
    time.sleep(30)

finish()

because you’re paying for the Lambda to run while you’re waiting for your operation to complete. Worse, Lambda functions have a maximum lifetime of 15 minutes, so if your process takes longer than 15 minutes, you have to do weird hacks to make it work with a pure Lambda solution.

When we found out that CloudFormation didn’t have support for creating DynamodDB Global Tables, our first thought was “we know how to do this, a Lambda function custom resource can handle it.” However, as we dug into the details and tried it out, we quickly learned that we could get into a scenario where creating the initial replica set or updating the replicas could easily exceed the 15-minute Lambda timeout.

Step Functions to the rescue!

Here’s where AWS Step Functions comes in. Step Functions make the task of orchestrating processes easier. They have built-in support for looping, waiting, and integrating with different functions and services, which makes them perfect for this sort of thing.

One of our team members put together this Step Function definition for creating a DynamoDB global table. It starts out by checking the state of the table, waiting until the table is ready for updates, then comparing the set of replicas with the desired set. You can only add one replica at a time, so the step function repeats the UpdateReplicas step until the actual state matches the desired state. Each step is very small and self-contained, usually only one or two API calls, and all of the waiting is done by Step Functions instead of in the Lambda function, so we’re not paying for idle time! Best of all, Step Functions can run for up to a week, so we didn’t need to worry about the 15-minute timeout any more.

There’s a small catch…

I wish I could use Step Functions directly to build CloudFormation custom resources instead of having to have a Lambda function that triggers the Step Function. #awswishlist
— Geoff Baskwill (@geoff_baskwill) February 24, 2021

CloudFormation doesn’t support direct integration with Step Functions as a custom resource provider yet, but we can use our old Lambda function trick to trigger the Step Function execution, and send the response back to CloudFormation when we get to an end state.

Wrapping up

When you love infrastructure-as-code and need a custom resource for something that CloudFormation doesn’t support, Lambda is usually a great solution. When you need a bigger hammer for complex orchestration or operations with lots of idle time, Step Functions can help get you there.

The goal is to retire this particular resource soon, as AWS tells us that they’ll have built-in support in CloudFormation for DynamoDB Global Tables in the very near future. That said, my team is happy that we were able to deliver an initial implementation with this workaround and provide value to our customers!

CloudFormation, Route 53, and ... EKS?

Thu, 22 Apr 2021 00:00:00 +0000

The other day I got halfway through writing a very irate support ticket to AWS, stopped to do some fact checking, and learned something deeply annoying.

One of the teams I work manages a bunch of services. One of these “services” is some common Amazon Route 53 infrastructure that is set up using AWS CloudFormation, and over the history of the project the deployment in the development account the team uses has been a little flaky. Each time we hit a deployment problem, the problem always turned out to be rate limiting. It never happened in production, so the flakiness didn’t quite get the attention that it could have.

Rate limiting in Route 53

Some background: Route 53 has a hard limit of 5 control plane requests per second per AWS account. For most folks, this is fine. However, it wasn’t working well for this team.

The team had raised support tickets and gotten advice like “CloudFormation will attempt to create your resources in parallel. One option to avoid rate limiting is to add DependsOn links to serialize the resource creation.” We weren’t super-happy with that answer, and there’s a CloudFormation roadmap item to fix it, but we needed something in the interim, and it worked … mostly.

On this day a deployment had failed again, and after going after the usual suspects and making sure that the resources were properly serialized, I got very irate. I was halfway through writing a support ticket:

We are still encountering Route 53 rate limits and our CloudFormation stack deployment / updates are intermittently failing, sometimes after only two resources are created. There are no Route 53 API calls being made by our applications, only through CloudFormation.

We are quite frustrated at this point and would like to request a session with a solution architect to help us understand how we should be doing this and

and I paused.

Check yourself before you wreck yourself

“There is no point in using the word ‘impossible’ to describe something that has clearly happened.” — Douglas Adams, Dirk Gently’s Holistic Detective Agency

“Is it true that there are no other Route 53 API calls being made?” I asked myself. A quick jaunt into AWS CloudTrail told me the answer, and also opened a gaping pit beneath my feet.

There were 436 Route 53 API calls made in the 2-minute period surrounding our CloudFormation failure. If you do the math, that’s 3.6 requests per second on average, so it’s not at all surprising that we maybe tipped over the limit of 5 at some point in there.

“But where are these coming from?” was my immediate question, and it was immediately answered.

Virtually all of these requests were being made by an EC2 instance that was part of an Amazon Elastic Kubernetes Service (EKS) cluster.

Talking through this with some other folks, I learned that they’d configured external-dns on the cluster, and that this behaviour is actually documented.

The production account doesn’t have the EKS cluster, so it’s not overwhelmed with Route 53 API calls, which explains why deployment never failed there.

Buh-bye

I wanted to decommission the cluster immediately, but unfortunately some teams still need it, so I wasn’t able to.

The external-dns documentation says that one workaround for the issue of the controller eating your entire Route 53 request budget is to extend the interval that the controller reconciliation loop runs at. In this particular cluster, the reconciliation loop was running every minute (the default!) to reconcile a set of records that change approximately never. I followed the instructions, set the interval to a week, and settled in to see what happened.

The first thing I noticed was that immediately the calls to Route 53 stopped. Not surprising, but great to see the confirmation. Several hours after the change, there were still no calls from the previously-misbehaving cluster.

All is well now, and I get to put away my detective hat for another day.

“The light works,” he said, indicating the window, “the gravity works,” he said, dropping a pencil on the floor. “Anything else we have to take our chances with.” — Douglas Adams, Dirk Gently’s Holistic Detective Agency

What I learned

First, CloudTrail was instrumental here. I’m still a novice, but I’m learning how powerful a tool it is. Once I knew what to look for, it was immediately obvious what the source of the rate limiting was. The events in CloudTrail identified the EC2 instance and even made it clear that the source of the requests was in an EKS cluster.

Second, I was reminded that Kubernetes is not a get-out-of-ops-free card. There is a lot of expertise involved in running Kubernetes well, even when you’re using a managed service like EKS. I knew this before, but this was an example of a cluster I didn’t even know existed (don’t worry: someone more responsible did know!) having side effects way outside its scope.

Why I Use Go (for now)

Thu, 03 Dec 2020 00:00:00 +0000

Recently someone asked me why I prefer using Go to build services instead of Python.

It struck me as an odd question, mostly because the way it was phrased and how I skimmed it made me think they were asking why I recommended using Python to build services, which is something I have never done. Once we got past my misunderstanding and I had apologized for not reading carefully, I took a few minutes to think about it.

The first point to keep in mind is that I would not call myself a competent Python programmer. It is entirely possible that if I were more comfortable with Python, I would pick it up as a tool more often. Some Very Smart People™ I know use Python for Real Services In Production™. I wouldn’t, but as I mentioned, I’m not exactly well-versed. I expect that if I were dropped into an environment where I could learn from those Very Smart People, I might grow to understand why they have made that choice.

It would be fairer to ask why I use Go instead of Java, as there was a time that I considered myself not-incompetent in Java, and I railed mightily against the project lead who decreed that I would learn Go.

I use and recommend Go these days because it has a lot of tools for writing robust, predictable, and reliable code, concisely. I have found over the last few years of learning Go that I can write better services than I ever did in Java, and I can do it faster and more easily. The end product is also more performant and more predictable with less effort on my part, not to mention ridiculously small compared to stuff built on the Java Virtual Machine.

As a tool, Go fits well with the problems that I am trying to solve today.

Are there things that are annoying and things I really don’t understand all that well yet? For sure. These things don’t really get in the way of creating value for customers, though.

Your Mileage May Vary.

Behaviour of background goroutines in AWS Lambda

Mon, 02 Nov 2020 00:00:00 +0000

I was curious what would happen with a time.Ticker running in an AWS Lambda function.

It was pretty obvious that the ticker wouldn’t tick while the container was hibernating, but what about when it woke up? Would a bunch of ticks happen if the container had been hibernating for a long time? Would you get a tick at all?

tl;dr

It looks like you get one tick when the function comes out of hibernation (assuming the ticker duration has passed at least once), and if the function never hibernates the ticks happen on time, which for my purposes is pretty much perfect.

The code

I set up a small AWS Lambda function that responds to API Gateway requests and has a goroutine running with a time.Ticker that ticks every minute:

package main

import (
  "context"
  "fmt"
  "net/http"
  "sync"
  "time"

  "github.com/aws/aws-lambda-go/events"
  "github.com/aws/aws-lambda-go/lambda"
)

func main() {
  wg := &sync.WaitGroup{}

  ticker := time.NewTicker(time.Second * 60)
  go func() {
    for t := range ticker.C {
      // let the WaitGroup know that we're working
      wg.Add(1)

      // tell the observer that the ticker actually ticked
      fmt.Println("tick at", t)
      
      // simulate a background process that takes a bit to complete
      time.Sleep(time.Second)
      
      // let the WaitGroup know that we're done
      wg.Done()
    }
  }()

  lambda.Start(func(ctx context.Context, req events.APIGatewayProxyRequest)(events.APIGatewayProxyResponse, error) {
    resp, err := realHandler(ctx, req)
    
    // If the ticker is running, wait for it.
    wg.Wait()

    return resp, err
  })
}

func realHandler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
  return events.APIGatewayProxyResponse{
    StatusCode: http.StatusPaymentRequired,
  }, nil
}

You’ll see how I’ve split out the real handler and the ticker-waiting “middleware” … this keeps the handler logic separated from the bit that waits for the background goroutine to complete what it’s doing before returning.

The test

I called the function a couple of times, waited a few minutes, and called the function again. The logs told me what happened:

2020-11-02T14:11:51.061-05:00 START RequestId: f2604bb1-8bc2-445a-b7b2-3893d8d3416d Version: $LATEST
2020-11-02T14:11:51.063-05:00 END RequestId: f2604bb1-8bc2-445a-b7b2-3893d8d3416d
2020-11-02T14:11:51.063-05:00 REPORT RequestId: f2604bb1-8bc2-445a-b7b2-3893d8d3416d Duration: 1.39 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 35 MB Init Duration: 66.79 ms XRAY TraceId: 1-5fa059f6-10e9a1715cb581111721b9a8 SegmentId: 0467a00a5d719145 Sampled: true

2020-11-02T14:11:54.171-05:00 START RequestId: 7a39872e-4b18-42da-b2ff-315a6590a1b6 Version: $LATEST
2020-11-02T14:11:54.174-05:00 END RequestId: 7a39872e-4b18-42da-b2ff-315a6590a1b6
2020-11-02T14:11:54.174-05:00 REPORT RequestId: 7a39872e-4b18-42da-b2ff-315a6590a1b6 Duration: 0.68 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 42 MB XRAY TraceId: 1-5fa059fa-0be52e2845702dd0365eeb74 SegmentId: 5d5f37f527b7de1a Sampled: true

The first two log blocks show the first two calls, at 14:11:51-05:00 and at 14:11:54-05:00. We wouldn’t expect to see a tick log here, and as expected we do not.

2020-11-02T14:16:28.076-05:00 START RequestId: 488123cd-43bc-4567-9970-00c35bfc91b8 Version: $LATEST
2020-11-02T14:16:28.076-05:00 tick at 2020-11-02 19:16:28.076072523 +0000 UTC m=+277.052607202
2020-11-02T14:16:28.078-05:00 END RequestId: 488123cd-43bc-4567-9970-00c35bfc91b8
2020-11-02T14:16:28.078-05:00 REPORT RequestId: 488123cd-43bc-4567-9970-00c35bfc91b8 Duration: 0.59 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 42 MB XRAY TraceId: 1-5fa05b0c-35afecef72dc3fb25cf4eeef SegmentId: 045243422ccaea72 Sampled: true

2020-11-02T14:20:15.133-05:00 START RequestId: 2dc415ef-f53f-4e35-865d-21b89cd9147e Version: $LATEST
2020-11-02T14:20:15.133-05:00 tick at 2020-11-02 19:20:15.133194003 +0000 UTC m=+504.107026404
2020-11-02T14:20:15.137-05:00 END RequestId: 2dc415ef-f53f-4e35-865d-21b89cd9147e
2020-11-02T14:20:15.137-05:00 REPORT RequestId: 2dc415ef-f53f-4e35-865d-21b89cd9147e Duration: 0.62 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 43 MB XRAY TraceId: 1-5fa05bef-5e8453ea22ff3fa4454e0af6 SegmentId: 183982d42c0b3848 Sampled: true

The next two log blocks show calls at 14:16:28-05:00 and 14:20:15-05:00 and one tick happening each time the code is invoked. This would seem to indicate that even if multiple ticks would normally have happened in the time period, they are consolidated into one.

Neat!

API versioning

Wed, 09 Sep 2020 00:00:00 +0000

There are lots of ways to implement API versioning. Picking one is an exercise in balance: what voices will you listen to?

For us, the most important voice is the voice of the customer. There are lots of customers, though, and many of them just want to get the job done. They are expecting us to be the experts and make it easy for them to build integrations that Just Work™.

People using our APIs don’t really care whether we’re 100% Fielding-Certified REST™ (not a real thing). They want something that’s simple and predictable.

How can we generally meet people where they’re at, and give everyone the benefit of versioning without creating work for them?

I consider Stripe’s API versioning strategy to be the gold standard. If you read their post, you’ll see how they carefully balance the ideal of “we will never break your integration with us” with “sometimes we need to make breaking changes”.

When do we need a new version?

Most people agree that you need a new API version every time you have a breaking change.

What’s a breaking change? It depends. Some people are really strict: if there is a new attribute, that’s a breaking change in their minds. I tend to be more lenient. If an attribute changes type (for example, from a simple type to an object) or if a required attribute in a response goes away, that’s a breaking change. It gets a little trickier when you modify an enumerated type, so you need to understand your audience and how they are using your API.

What’s a good version string look like?

The Stripe folks use a date to represent an API version; it’s handy as people generally understand how to compare dates. Some people get confused when they see an “old” API version and think that your documentation is out of date, but those people are rare.

Other folks prefer to use semantic versioning (v1.0.3), or use a simple string like v1 and move on.

I feel like v1 isn’t particularly granular, semantic versioning is a bit sketchy from an API perspective and too much like a release number, and prefer date strings.

Options

Just put a version in the URL

also known as “What’s wrong with /v1/widgets/123456?”

Some people love putting the API version in the path portion of the URL. I don’t particularly like this myself. If I have one resource one day that has a breaking change, then I have to break out /v2. The next day, another breaking change to a resource, /v3. Where does it end? What do you do with the resources that didn’t change?

From a dogmatic perspective, putting the API version in the URL path means that your resources at /v1/* and at /v2/* and at /v3/* are different things, not different representations of the same things. People with a Job To Be Done won’t care, but it’s a bit irksome to have something so flagrantly against how the Web is supposed to work. It’s handy from a service developer’s perspective, as you can “just” put a new endpoint handler in place, but if only one resource has changed then it gets messy.

Use content negotiation like Fielding intended

also known as “What’s wrong with Accept: application/x-widget+json;v=1? Accept: application/x-widget.v1+json?”

For a long time I believed that content negotiation was Totally Awesome™. Imagine: something built into the HTTP standards that lets you not only discover what potential representation options exist for any given resource, but also transparently negotiate upgrades on either the client or the server side.

I was so enamoured with this idea I did a conference talk on it, claiming that using content negotiation was level π of the Richardson Maturity Model (because it’s transcendent and slightly greater than 3).

I’ve mellowed since then. I still believe that content negotiation is an awesome mechanism, but it’s a bit too clever for some folks, and unfortunately framework support is rather sparse for parameterized content types on both the client and server side.

Forcing people to learn about content negotiation before they can use your API also means that some people will just walk away.

Use an API version header

also known as “What’s wrong with Api-Version: 2020-04-01?”

I haven’t managed to find any concrete reasons why you shouldn’t use a custom header for passing the API version. The best people have been able to give me is “it’s harder to set up header-based routing”, which is sometimes true and totally valid for people in that position.

If you allow caching of API responses, you should include a Vary: Api-Version header in your responses if you go this route; it will tell any caching infrastructure that the response may be different if the API version in the request is different.

The Stripe folks have a really smart mechanism: if you don’t specify the API version in your request, they look at what API version your account is pinned at. If it’s your first time calling the API, they pin you to the current version. You can change your pinned API version through the console (and presumably through the account management API as well).

I haven’t yet gone so far as to implement API version pinning. The best I’ve been able to do is default to the latest version when the header isn’t present – this gives folks the most chance of seeing responses that match the current API docs, but it means that folks who don’t provide an API version header are more vulnerable to breaking changes.

Pagination and cursors

Wed, 07 Mar 2018 00:00:00 +0000

Whenever you retrieve a list of things in an API I’ve managed to influence, unless the size of the list is internally capped at an incredibly small number, you’ll see that the API supports limit and cursor query parameters to allow you to page through the results. This post will explore what those parameters do, how they work, and some of the limitations of the approach.

Why pagination?

Before we get into the details of how the solution works, we should properly understand the problem. Most APIs start out very simple, with a trivial GET operation on a container resource that returns a list of objects, and everyone is happy:

GET /widgets HTTP/1.1
Accept: application/json
...

HTTP/1.1 200 OK
...
Content-Type: application/json

[ { widget 1 }, { widget 2 }, ... ]

The happiness starts to go away when the server starts handling hundreds or thousands of widgets. API response times start getting long, payloads are huge, and memory usage on the server gets really spiky as it tries to serialize the entire contents of the database in response to every request.

Most people come to the quick realization that it’s better to only ask for a smaller amount of data to start, and only load additional data on demand:

GET /widgets?limit=10 HTTP/1.1
Accept: application/json
...

HTTP/1.1 200 OK
...
Content-Type: application/json

[ { widget 1, { widget 2 }, ... ]

But wait! With this data structure, how is the client supposed to know whether there’s more data to load?

Some APIs simply say “Give me the start point and page size, and I’ll figure out the rest.” OK, no problem. But what’s the start point? Primary key? What if you’re sorting on something else? Things get complicated quickly. Some folks try to use an offset parameter, but that can also get complicated.

Cursors can help

Inspired by a very detailed post by the kind folks at Slack, I tried implementing cursors in an API framework I was building. Briefly, in this context a cursor is a token that can be used to resume the query when more than one page of results is available.

When someone makes a request to list resources, our API returns a structure that contains one attribute with the list of requested resources, and if there is more than one page of results, it will also include a next attribute containing an encoded token that can be used in a subsequent request to get the next page of results.

GET /widgets?limit=10 HTTP/1.1
Accept: application/json
...

HTTP/1.1 200 OK
Content-Type: application/json
...

{ "widgets": [...], "next": "bmljZSB0cnkK" }

If the next attribute isn’t there, there is no more data to be had.

The next attribute is a base64-encoded token that the server knows how to decode into a database query that corresponds to the next page of data. It includes information about the sort order, starting point, and anything else that the server needs in order to do its job. The API consumer doesn’t need to know anything other than “as long as there’s a next field, take the value out of the result and paste it into the cursor query parameter in the next request”, and I’m free to make changes to the internal logic at any time without affecting consumers, which is pretty amazing from a service development perspective. Different resources can use different logic internally without exposing the details to the API user.

Even better, the next attribute is self-contained. The server doesn’t have to maintain any state about the query or the cursor, so no resources are consumed if the caller never comes back.

Other ways to communicate the cursor

Some folks use a Link header to carry the full URL for the next page; I think that is a fantastic idea and recommend implementing that as well:

Link: </api/widgets?cursor=aGFwcHkgZWFzdGVyIQo>; rel="next"

Security considerations

If someone weren’t careful, they could easily take the idea above and say “I’ll just base64-encode a JSON blob with a bunch of parameters, call it a cursor, and take the rest of the day off!”

Of course, we want to keep our attack surface to a minimum. To help with this, we encrypt the cursor with an algorithm that verifies the integrity of the protected content, and we salt the input to make it harder to attack with known plaintext. For added protection, we also rotate the encryption key regularly, so that if someone does manage to break the key they can only attempt further attacks for a limited period of time.

Limitations

The main limitation in this scheme is that it’s pretty focused on unidirectional scrolling – the cursor only goes forward (or back, depending on the sort order). There are ways to deal with this, but they’re far more complicated than I wanted to get into, and more importantly they’re not relevant to our current use cases.

AWS re:Invent 2017

Tue, 05 Dec 2017 00:00:00 +0000

tl;dr

If I had to pick one word to describe AWS re:Invent 2017, it would be “popular”. There were over 43 000 attendees this year. The conference organizers and staff did an amazing job rising to the challenge, keeping folks corralled and working hard to manage expectations. I had a fun time; it was great to meet up with old friends, make a few new ones, and learn stuff at the same time.

Sessions had about 75% reserved seating and about 25% of the seats available for walk-ups, but with so many folks at the conference, popular sessions had a lot of waiting in line and a lot of unhappy faces on the people who didn’t make it in.

The conference was spread over a massive campus: 6 properties from the MGM at one end to the Venetian at the other; a good 30-minute hike if you’re fast and don’t get lost or caught in construction. Shuttles were available, but lines were long and people reported transit times of 60–90 minutes, so other than an early-morning trip on Monday, I just walked. A lot. I put in just over 60km during the week and am very happy to have had good shoes.

There were lots of interesting breakout sessions, chalk talks, and workshops; reports are favourable on the jams and hackathons. Based on this year, if I get to go again I’m going to try to go to chalk talks exclusively and fill in remaining time with a hackathon and a jam or two. The sessions are great, but they’re recorded and you can watch them in your copious spare time once you get home.

General activities

Registration

Registration opened just before 1pm on Sunday and was incredibly efficient. There were tons of folks available to hand out badges, and they had a separate section for people to try on this year’s hoodie before heading over to pick up their size, which eliminated a bottleneck from last year. I was in and out in single-digit minutes.

Sadly, I lost my hoodie in the crush of the crowd on Monday and had to use last year’s instead. Life is pain.

Midnight Madness

Modeled after an NCAA basketball tradition, AWS had a big party and then hit the court at midnight exactly with a new service annoucement: Amazon Sumerian, a 3D, VR, and AR service that lets folks create interactive environments. So far environments are single-user, but it’s not hard to see multi-user coming. There will be a marketplace for folks to create and sell 3D models that anyone can use in their environments, and there’s already support for chatbot actors called Hosts that can interact with users.

At 1am, Amazon announced support for AWS Elemental, a full suite of media services targeting folks who broadcast live and pre-recorded video.

This is the last time I’m going to mention new releases; you can get a full list from the AWS product announcements page. Be prepared to scroll.

Welcome Reception

The expo hall opened at 5pm Tuesday for the usual welcome reception; it had been open for most of the day but the reception added food and drinks. It’s very popular and there’s always a mad rush for swag. I had some good chats with vendors and the very smart folks in the AWS section, including the Education folks hidden away in the back corner. The most interesting thing I took away was that AWS has a program for high-school and college students that students can sign up for themselves or with a teacher’s help and learn all kinds of stuff about the cloud, with tons of content and credits available.

Sadly, I wasn’t able to say the right things to the Yubico folks to snag a free key. I want to get a 4 so that I can experiment with U2F, and I doubt they were handing those out.

Tuesday Night Live with Peter DeSantis (video)

This was a very interesting update on the vast networking infrastructure that AWS has around the world, and segued into a description of the incredible work they’ve done in compute.

The most inspiring part of the evening, though, was the Autodesk presenter talking about how advances in compute have enabled entirely new ways of working for their customers and how that’s going to help the world be a better place for the 10 billion people that will soon be here.

Wednesday Keynote with Andy Jassy (video)

This keynote came with a lot of mind-bubbling announcements, but I promised not to list them. It was a fun time and clearly Andy Jassy was excited to share the news with the world.

Thursday Keynote with Werner Vogels (video)

Werner Vogels’ keynote was more retrospective. I didn’t hike over to the MGM to watch the show and elected to stay in a remote viewing room. I think this was a mistake as I missed out on the energy and wasn’t as engaged.

re:Play

I’m not a big party person, so I just picked up a T-shirt for the folks at home and toured around the megastructures. Yes, megastructures — if you’re going to put on a party for 43 000 people, you need space. There were 3 huge tents and tons of activities, from video games to mini-golf to archery tag to dodgeball to a ball pit and many more, all set to the soothing tones of Team Ezy, Jen Lasher, and DJ Snake.

Breakout sessions and workshops (AWS video catalog)

I’m just going to dump in the list of sessions that I wanted to go to for future reference; there will be some sparse commentary for the ones that I actually made it to. There were over 1300 breakout sessions spread out over the five properties, so there was really no way to cover even a noticeable fraction of the ones I wanted to see.

Community Day

DVC202 — The Open Guide to AWS: A GitHub-Based Resource for Practitioners

The Open Guide to AWS is an open source writing project, which over the past year has become one of the most popular AWS resources on the web. It’s both a written resource on…

DVC302 — And You Thought You Knew Amazon EC2

When is the last time you explored the nooks and crannies of Amazon EC2? While you weren’t looking, AWS leveled it up with more features and capabilities than you can shake a…

DVC304 — Compliance and Top Security Threats in the Cloud — Are You Protected?

Compliance is necessary and a good thing. However, many compliant companies are still getting breached. In this talk, we discuss the importance of using a risk model to…

Enterprise

ENT346 — How AWS Runs Our Weekly Operations Meetings (Chalk Talk)

For more than 10 years, AWS has held a weekly meeting to review the operational performance of our services, identify new best practices, and share lessons learned from the…

IoT

IOT311 — Customer Stories of Things, Cloud, and Analytics on AWS

In this session, AWS IoT customers talk about the nuances, successes, and challenges of running large-scale IoT deployments on AWS. Hear from customers who have been…

I stumbled into this talk accidentally; I was waiting for a friend who was going to the session, so decided to wait in the walk-up line and ended up going in. The talk was a great discussion of how iRobot does IoT, I really enjoyed seeing the details and challenges that they encountered.

Containers

CON201 — Containers on AWS — State of the Union

Just over four years after the first public release of Docker, and three years to the day after the launch of Amazon EC2 Container Service, the use of containers has surged…

This was an interesting outline of what’s come out and what’s coming soon for containers in AWS. Container product manager Deepak Singh hinted that AWS Fargate, the new container cluster service, was only a step towards what they really want to do, which is allow customers to run containers without worrying at all about the servers that they’re running on.

CON213 — Hands-on Deployment of Kubernetes on AWS (Workshop)

In this workshop, attendees get hands-on experience using Kubernetes and Kops (Kubernetes Operations), as described in our recent blog. Attendees learn how to provision a…

CON302 — Building a CI/CD Pipeline for Containers on Amazon ECS

Containers can make it easier to scale applications in the cloud, but how do you set up your CI/CD workflow to automatically test and deploy code to containerized apps? In…

CON402 — Advanced Patterns in Microservices Implementation with Amazon ECS

Scaling a microservice-based infrastructure can be challenging in terms of both technical implementation and developer workflow. In this talk, AWS Solutions Architect Pierre…

Media

MAE401 — Designing for Disney/Marvel Studio-Grade Security (video)

Security is paramount for media storage and workloads and can directly impact a studio’s bottom line. As core media workloads move to the cloud, it’s imperative to examine…

Content Distribution

CTD301 — Amazon CloudFront Flash Talks: Best Practices on Configuring, Securing, Customizing, and Monitoring Your Distribution. (video)

In this series of technical flash talks, learn directly from Amazon CloudFront engineers about best practices on security, caching, measuring performance using Real User…

CTD401 — Taking DevOps Closer to the AWS Edge

Since last year’s ‘Taking DevOps to the Edge’, and with the introduction of AWS Lambda@Edge, the tools available to apply DevOps practices to your application edge have…

Serverless

SRV213 — Thirty Serverless Architectures in 30 Minutes (Chalk Talk)

Don’t blink because in this session, we quickly show you thirty different architectural patterns that you can use with AWS Lambda to solve everything from basic…

This talk was insanely popular. There were multiple repeats and I didn’t make any of them, but I really wanted to. Maybe there’s a recorded version of this talk somewhere out there? 爛

SRV301 — Optimizing Serverless Application Data Tiers with Amazon DynamoDB

As a fully managed database service, Amazon DynamoDB is a natural fit for serverless architectures. In this session, we dive deep into why and how to use DynamoDB in…

This was a cool session that covered (among other things) how to solve a couple of use cases that I’ve always struggled with, so I walked out a more-informed person.

SRV302 — Building CI/CD Pipelines for Serverless Applications (video)

Building and deploying serverless applications introduces new challenges for developers whose development workflows are optimized for traditional VM-based applications. In…

SRV305 — What’s New in Serverless (video)

Join us to learn what’s new in serverless computing and AWS Lambda. Dr. Tim Wagner, General Manager of AWS Lambda and Amazon API Gateway, will share the latest developments…

Fun times are always had in serverless-town. This session started off with cutting a server in half.

One thing I walked away from the session with is that cold start times have been cut by 80% this year, which is amazing. In a post-session chat, I found out that VPC cold starts are still going to be slow until they re-architect how Lambda attaches the ENI; I’d hoped that some of the container networking work released this year would help but apparently the sausage isn’t made that way.

SRV307 — Operating Your Serverless API in Production at Scale (video)

In this session, you will learn how to deploy, monitor and manage your serverless APIs in production. We will deep dive into advanced capabilities of API Gateway that enable…

SRV308 — Securing Serverless Applications Step-by-step (video)

You need a new approach to security for serverless applications. Classic approaches just don’t make sense, because tools and process can only take you so far…

This was an excellent talk. Mark Nunnikhoven’s practical, clear approach to application security made this a fun walk-through of practical steps that developers and organizations can take to make their serverless applications more secure. 100% recommended.

SRV310 — Designing Microservices with Serverless

When designing microservices there are a number of things to think about. Just for starters, the bounds of their functionality, how they communicate with their dependencies,…

SRV320 — Best Practices for Using AWS Lambda with RDS/RDBMS Solutions (Chalk Talk)

Are you considering using AWS Lambda with your RDS/RDBMS solutions, but don’t know where to start? In this session, we look at recommended architectural patterns and best…

SRV321 — Serverless Reactive Microservices on AWS (Chalk Talk)

In this talk, we will cover the concepts of Reactive Microservices Architecture, a set of patterns and best practices for implementing backend systems with very high scaling…

SRV329 — Lessons in Serverless Architecture for IoT from iRobot (Chalk Talk)

The event-driven nature of serverless architecture is a natural fit for IoT use cases. In this chalk talk, we cover iRobot’s use case and requirements, detail the serverless…

SRV401 — Become a Serverless Black Belt: Optimizing Your Serverless Applications (video)

Are you an experienced serverless developer who wants a handy guide to unleash the full power of serverless architectures for your production workloads? Do you have questions…

SRV403 — Serverless Authentication and Authorization: Identity Management for Serverless Applications

Many serverless applications need a way to manage end user identities and support sign-ups and sign-ins. Join this session to learn real-world design patterns for…

SRV425 — Serverless OAuth: Authorizing Third-Party Applications to Your Serverless API (Workshop)

By using serverless architectures, startups, and enterprises are building and running modern applications and services with increased agility and simplified scalability, all…

Security

SID205 — Building the Largest Repo for Serverless Compliance-as-Code (video)

When you use the cloud to enable speed and agility, how do you know if you did it right? We are on a mission to help builders follow industry best practices within security…

SID312 — DevSecOps Capture the Flag (Workshop)

Notice: This Workshop requires a laptop computer and an active AWS account with Administrator privileges. In this Capture the Flag workshop, we divide groups into teams and…

SID332 — Identity Management for Your Users and Apps: A Deep Dive on Amazon Cognito (video)

Learn how to set up an end-user directory, secure sign-up and sign-in, manage user profiles, authenticate and authorize your APIs, federate from enterprise and social…

SID343 — User Management and App Authentication with Amazon Cognito (Workshop)

Notice: This Workshop requires a laptop computer and an active AWS account with Administrator privileges. Are you curious about how to authenticate and authorize your…

SID401 — Let’s Dive Deep Together: Advancing Web Application Security (Chalk Talk)

Beginning with a recap of best practices in CloudFront, AWS WAF, Route 53, and Amazon VPC security, we break into small teams to work together on improving the security of a…

SID402 — An AWS Security Odyssey: Implementing Security Controls in the World of Internet, Big Data, IoT and E-Commerce Platforms (Workshop)

Notice: This Workshop requires a laptop computer and an active AWS account with Administrator privileges. This workshop will give participants the opportunity to take a…

Financial Services

FSV301 — Security Anti-Patterns: Mistakes to Avoid

At AWS, security is job zero. Our infrastructure is architected for the most data-sensitive, financial services companies in the world. We have worked with global enterprises…

Big Data

ABD301 — Analyzing Streaming Data in Real Time with Amazon Kinesis

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. In this session,…

ABD401 — How Netflix Monitors Applications in Near Real-Time with Amazon Kinesis

Thousands of services work in concert to deliver millions of hours of video streams to Netflix customers every day. These applications vary in size, function, and technology,…

Architecture

ARC316 — Getting from Here to There: A Journey from On-premises to Serverless Architecture

In this session, go on a journey from traditional, on-premises applications and architecture to pure cloud-native environments. This transformative approach highlights the…

ARC401 — Serverless Architectural Patterns and Best Practices

As serverless architectures become more popular, customers need a framework of patterns to help them identify how they can leverage AWS to deploy their workloads without…

Networking

NET203 — Using Amazon VPC Flow Logs to Do Predictive Security Analytics (Workshop)

Ready to secure your network and application in near real-time using Amazon VPC flow logs and AWS Web Application Firewall (WAF)? In this workshop, we incorporate advanced…

This seemed like it was going to be a really interesting workshop; unfortunately the majority of the time was spent in lead-up and very little time was spent on the parts that I was interested in. I’m hoping to put together something that covers the data enrichment parts of the process and gets into the machine learning — certainly not an area of expertise for me, but perhaps something I can post about as I learn.

NET403 — Deep Dive: AWS Direct Connect and VPNs

As enterprises move to the cloud, robust connectivity is often an early consideration. AWS Direct Connect provides a more consistent network experience for accessing your AWS…

DevOps

DEV401 — Advanced DevOps Practices for AWS (Workshop)

Advanced DevOps Practices for AWS is designed to provide concise and prescriptive guidance on how to build proper DevOps solutions in AWS using common patterns and best…

Mobile

MBL305 — Implement User Onboarding, Sign-Up, and Sign-In for Mobile and Web Applications with Amazon Cognito

Learn how to use Amazon Cognito to build the user identity management workflows, including user on-boarding, sign-up, and sign-on for mobile and web applications.

Machine Learning

MCL212 — AWS DeepLens workshop: Building Computer Vision Applications (Workshop)

In this workshop, developers have the opportunity to learn how to build and deploy computer vision models, such as face detection and object analysis using the AWS DeepLens…

There was no way that I was going to make it into one of these sessions; they were easily the most popular workshops of the week, announced during the Andy Jassy keynote, with 15 repeat sessions throughout Wednesday and Thursday. Two lucky colleagues did get in and had fun with their new DeepLens devices; I’m hoping they share their experience with the rest of us.

Databases

DAT405 — Workshop on Advanced Design Patterns for Amazon DynamoDB (Workshop)

Join us for the first-ever Amazon DynamoDB practical hands-on workshop. This session is designed for developers, engineers, and database administrators who are involved in…

I am so bummed that I missed this. It would have been fantastic to get some non-trivial practical experience with DynamoDB. I wish AWS posted their workshop material online so that folks could play around with the examples.

I feel like a traitor. Or a fool.

Sun, 18 Jun 2017 00:00:00 +0000

https://www.flickr.com/photos/andrewfhart/8106200690

Recently I signed up for the A Cloud Guru AWS advanced networking certification prep course put together by Adrian Cantrill.

After the obligatory OSI 7-layer modules, which I watched because you never know, things rapidly got interesting. I found myself learning about BGP, AWS VPC peering, how to multi-home EC2 instances with ENIs, how to set up a shared-services network that could talk to peered VPCs with overlapping CIDR ranges (turns out it’s not even that difficult), and all kinds of really cool stuff.

I got nerd-sniped. Big time. Networking was a big part of my early career, and the course scraped a bunch of rust off and added a whole new layer of awesome things to learn about. My brain was all hopped up on learning-juice, spinning and carousing and saying to itself “THIS IS SO COOL.”

Then something hit me like a cross-town bus.

Everything I’d just been geeking out about is completely irrelevant in the serverless world.

Last year I went to the first Serverlessconf. It was great, tons of fun; I met some really smart folks and learned that people are doing real work in a whole new way. There is a whole new solution space becoming available, a solution space where, as Tim Wagner put it, you don’t have to worry about whether you have cattle or pets, you just show up at the drive-through and get the burger you were looking for. It’s arguable that the vast majority of problems people are trying to solve today fit into this space: take an event, do some transforms on it, send it somewhere else, maybe do some aggregation and correlation later. It’s a way to focus on your core competency and let other folks worry about the servers, the mechanics of routing those events to your code, instantiating it, and keeping everything available.

As someone who’s pretty keenly interested in the future of serverless, it seems either traitorous or foolish to be taking a course on advanced networking.

What faithful, intelligent person would invest time in learning all sorts of networking shenanigans when there’s so much to learn in serverless space?

It’s a week later now, and I’ve realized that the answer to my question is “people like me.” People who think that even though there is a huge bright shiny serverless future out there, someone still has to know how it all fits together. People who think that the road between today and that shiny future will have more than one bump in it. People who just like learning new things every day.

AWS Certified Security Specialty

Mon, 21 Nov 2016 00:00:00 +0000

A good friend of mine has a habit of throwing gauntlets around like they’re nothing. “So, Geoff,” he’ll start casually, “did you know there are some new AWS specialty certifications in beta now?”

This is usually about the time that I realize I’m in for a world of pain, uh, learning. It turns out that this time AWS has come up with three exams that I’ll conservatively call pro-level.

I’ve been learning a lot over the past year and a bit, negotiating a career transition from telecom to security, so I’d consider myself a firm novice when it comes to all this stuff, but it’s not like we were doing nothing over in the old world, so I figured I’d check out the landscape for the security certification. I find that preparing for certifications is a great way to learn at least the theory, and sometimes get a bit of practice in as well. When some detail doesn’t click right away, it’s easy enough to spin something up and try it out to figure out how things are working.

Keep in mind that the “AWS recommendations” includes, as a single bullet point among several, “AWS documentation.” So, since we’re talking about security, let’s take that as the reference docs, programmers’ guides, and best practices for 50+ services.

Next is the alphabet soup: as @marknca says, “candidates should be familiar with ISO 27001, PCI DSS, HIPAA, SAS 70, SOC 1, FISMA, and NIST.” That’s “and”, not “or”, folks.

Right now I’m on the edge. The reading so far has been pretty much stuff I’ve seen before one way or another, so maybe, just maybe, I’ll give the exam a shot.