Aws on Geoff Baskwill

Custom CloudFormation resource types

Mon, 13 Nov 2023 00:00:00 +0000

I recently had the opportunity to explore building custom CloudFormation resource types as a mechanism to support a stream-aligned team. It was intriguing to play with this new-to-me tool and see how it could reduce complexity for the team.

I’ve written before about using Step Functions to build custom resources in CloudFormation. This builds on the old-school custom resources mechanism where you provide a Lambda handler as part of your stack. This is convenient for very small or one-off duct-tape situations, but if you need more complex resource logic or if you want to provide a simple resource interface to a bunch of teams, there is a better way.

The AWS CloudFormation registry and the CloudFormation command-line interface are this better way. They allow you to write your own resource provider that will be exposed using the exact same mechanisms that CloudFormation uses for many AWS-provided resource types.

You can do all kinds of super-neat things with resource types, and there is a neat pattern to report progress when your resource may take a while to create or when there are multiple steps to create your resource.

Batteries not always included

Resource providers can expose configuration data; this is super-handy if you want to attach “account-level” configuration (really region-level configuration) to your type’s handler. Unfortunately, while an API exists to programmatically set the configuration data, tragically and ironically there is no CloudFormation support for setting the configuration data on a resource type.

Because of the super-cool open way that AWS CloudFormation resource providers are built, a person could theoretically open a pull request on the aws-cloudformation/aws-cloudformation-resource-providers-cloudformation repository, duplicating the aws-cloudformation-hooktypeconfig folder and refactoring it to implement AWS::CloudFormation::ResourceTypeConfig, but I haven’t gone there yet. Probably should open a roadmap issue for discussion first.

I used an old-school custom resource to set the configuration for my custom types. 🤦‍♂️

Fun times with AWS STS AssumeRole

Mon, 21 Aug 2023 00:00:00 +0000

I had an AWS STS AssumeRole problem today when I was working with the AWS SDK; digging in it turned out the problem was that AssumeRole was not actually being invoked. Poking around I was unsurprised to find that @ben11kehoe had already clearly stated the problem and GitHub user petitout had example Go code with the right way to do it (link).

Sometimes you need to assume another role than your default execution role in order to accomplish a task. You can explicitly call STS AssumeRole yourself and configure the SDK client to use the returned credentials, or you can configure your SDK client to call AssumeRole itself when it needs the credentials. I like the second way because it means that credentials aren’t something that my code has access to.

Sample code for not assuming a role correctly; you might have stumbled on this the same way I did:

// almost certainly not what you want
cfg, _ := config.LoadDefaultConfig(
	ctx,
	config.WithAssumeRoleCredentialOptions(func(aro *stscreds.AssumeRoleOptions) {
		aro.Client = stsClient
		aro.RoleARN = arn
		aro.RoleSessionName = sessionName
	}),
)

A person might think that the code above is right, because it has all kinds of the right keywords that trigger pattern recognition. The comments on WithAssumeRoleCredentialOptions read:

// WithAssumeRoleCredentialOptions is a helper function to construct functional
// options that sets a function to use stscreds.AssumeRoleOptions on config's
// LoadOptions. If assume role credentials options is set to nil, the assume
// role credentials value will be ignored. If multiple WithAssumeRoleCredentialOptions
// calls are made, the last call overrides the previous call values.

I think that what this actually does is … you know, I truly have no idea, and don’t intend to dig into what exactly this does. It definitely does not do what I thought it did.

Sample code for probably doing what you want:

// almost certainly what you do want instead
cfg, err := config.LoadDefaultConfig(
	ctx, 
	config.WithCredentialsProvider(stscreds.NewAssumeRoleProvider(
		stsClient, arn, func(aro *stscreds.AssumeRoleOptions) {
			aro.RoleSessionName = sessionName
		}
	))
)

When the SDK needs credentials, this setup uses STS AssumeRole to obtain credentials with the STS client and role ARN that you provide. The stsClient in this example can be configured to take credentials from the environment.

Hope posting this helps make the solution findable for other folks.

re-posting and expanding on my original Mastodon post

SQS, DLQs, and KMS, oh my

Mon, 06 Mar 2023 00:00:00 +0000

Recently one of the teams I work with had a fun time (note: it was not fun in the moment) with Amazon Simple Queue Service (SQS), dead-letter queues (DLQs), and AWS Key Management Service (KMS).

I thought I’d share because we learned something pretty important.

If you’re new to SQS, you may be kind of aware that it’s a generalized queue service that lets you post messages onto the queue and then process queued messages in a handler, kind of like this:

There is an event source mapping in between the queue and the handler; we’re going to gloss over that detail for a minute.

SQS is super-handy, reliable, and amazingly scalable; AWS guides us to use it anywhere that we need to buffer communications between components of a solution.

There’s even a mechanism to deal with failures. You can configure a policy on your queue so that if SQS fails to deliver¹ a message to the handler enough times, the message gets moved to another queue for special handling. This second queue is called a “dead-letter” queue, named after the post office concept where messages that were undeliverable got sent to a special place for handling.

¹ “SQS fails to deliver” is not technically accurate, as SQS does not deliver messages, rather the event source mapping retrieves the message, attempts to deliver it to the Lambda function, and deletes the message if the message was handled successfully. If there was an error retrieving or processing the message, SQS will make the message available again after the visibility period expires. Eventually the message will expire, either after a specified time period or after a specified number of retrievals from the queue.

You may have a mental model of this setup that looks a bit like this:

SQS also lets you encrypt messages in the queue; you can use an AWS-managed key or a customer-managed key. Our teams always use customer-managed keys, so we have an SQS queue with a KMS CMK and DLQs (say that three times fast!). The updated model looks a bit like this:

If you’re an SQS expert, someone who has read the documentation carefully, or someone who read very carefully through the paragraphs above, you will instantly see the problem with this picture. If you didn’t catch it, you are not alone.

The key phrasing above is that undeliverable messages are moved from the primary queue to the dead-letter queue. They’re not retrieved from the primary queue and then sent onto the dead-letter queue, they’re moved. This isn’t explicitly stated in the documentation we have been able to find, but it’s clear from the behaviour we experienced and a couple of other hints in the documentation.

Our team discovered that the DLQ handler wasn’t processing messages, and in fact its event source mapping (the invisible thing we agreed to gloss over) was entirely disabled. An extremely-helpful AWS support person was able to tell us why.

When messages are put onto a primary queue that has KMS encryption enabled, they are encrypted using that queue’s KMS key. The handler must also have permissions to decrypt messages using the key, and we had that all working perfectly.

The Lambda function does not use the decryption permission itself, but rather “loans” that permission to the event source mapping. In our case, the Lambda function’s execution role and therefore event source mapping did not have sufficient permissions to decrypt the message, so the mapping failed to decrypt the message after retrieving it from the DLQ and did not invoke the Lambda function.

If you have a flawed understanding of the DLQ like we did, you might expect that messages would be decrypted and then re-encrypted when they are sent on the DLQ. However, they’re not decrypted and re-encrypted. They’re not even sent on the DLQ. They’re moved from the primary queue to the DLQ.

We had our DLQ set up with a separate key, and the DLQ handler had permissions to use the key, and in tests where we sent messages to the DLQ the handler worked perfectly. However, in a full end-to-end scenario where messages fell off the primary queue and were moved to the DLQ, the event source mapping failed to decrypt the message (it was set up with permissions on the DLQ key, not the primary queue key) and did not invoke the handler.

AWS is super-smart, and they know not to try things forever when there’s persistent failure. The event source mapping eventually disables itself and emits a CloudTrail event when it does so. If you’re vigilant or, preferably, if you set up a CloudWatch alarm based on the ApproximateAgeOfOldestMessage metric on the DLQ, you’ll be able to detect that messages are queuing up in the DLQ and not getting handled.

It’s important to remember that because messages are moved and not re-sent on the DLQ, your ApproximateAgeOfOldestMessage does not reflect the time that the message is in the DLQ but rather the time since the message was originally sent on the primary queue. This is also an important thing to remember when you are configuring your DLQ: if you use the same policy to age messages out of your DLQ as you have in your primary queue, you may find that messages disappear without getting handled the way you expected.

I did mention that AWS documents the requirement to use the same key:

Your source queue and its corresponding dead-letter queue need to share the same KMS key.

Oops. 😳

We’re looking into creating a cfn-lint custom rule to prevent us from tripping over this issue again.

re:Inforce 2022 - IAM306: Managing B2B identity at scale

Wed, 27 Jul 2022 00:00:00 +0000

I had the amazing opportunity to co-present a session at AWS re:Inforce 2022. My co-presenter was Suresh Sridharan, a product manager on the Amazon Cognito team.

If you’re not familiar with AWS re:Inforce, it’s an annual Amazon Web Services conference focused on security. It’s quite a bit smaller than re:Invent, their conference targeting builders, but that is actually a massive draw for me because I’ve found that re:Invent has gotten a bit big. I appreciate the focus and the relative ease of navigating a single site rather than what seems to be the entire Las Vegas Strip.

Suresh talked about a variety of options that Cognito customers have for managing multi-tenant solutions, and I talked about the particular choices that our team has made and what took us down this path.

For the full details, check out the slides and the recording.

Enriching Amazon Cognito features with an Amazon API Gateway proxy

Wed, 30 Mar 2022 00:00:00 +0000

I read a great post about protecting Amazon Cognito with an Amazon CloudFront proxy and wondered if using an Amazon API Gateway proxy instead would work better.

It really really did. Being able to use the full power of Lambda was fantastic, and the performance was improved significantly.

This approach also let us quickly implement a few tiny features that Cognito doesn’t have yet, like server-side attribute validation and user lifecycle events.

I got in touch with the author of the original post and he was generous enough to offer a collaboration on a follow-up post on the AWS architecture blog.

Check out the full post for all the details.

Using Step Functions to build CloudFormation custom resources

Sat, 24 Apr 2021 00:00:00 +0000

Photo by Serban Mestecaneanu on Unsplash

My team uses AWS CloudFormation to provision our cloud infrastructure using code. Most of the time we can get what we need with the set of resources that AWS provides.

However, sometimes CloudFormation support for a service or a particular feature takes a while to arrive, and we need to fill in the gap ourselves. CloudFormation gives us the ability to fill these gaps by building “custom resources” that can literally run any logic you need in AWS Lambda, and we’ve used these when we needed to.

Sometimes Lambda isn’t the right answer

Sometimes even Lambda fails us, though, as some resources can potentially take a long time to set up, and we don’t particularly want to have a Lambda function sitting idle or potentially timing out halfway through the resource setup.

The great thing about Lambda functions is that you only pay for them when they’re running. This makes them great for places where you need to run a quick task or handle an API request. You lose some of the benefits when your Lambda function looks like this:

start_something()

while not_done():
    time.sleep(30)

finish()

because you’re paying for the Lambda to run while you’re waiting for your operation to complete. Worse, Lambda functions have a maximum lifetime of 15 minutes, so if your process takes longer than 15 minutes, you have to do weird hacks to make it work with a pure Lambda solution.

When we found out that CloudFormation didn’t have support for creating DynamodDB Global Tables, our first thought was “we know how to do this, a Lambda function custom resource can handle it.” However, as we dug into the details and tried it out, we quickly learned that we could get into a scenario where creating the initial replica set or updating the replicas could easily exceed the 15-minute Lambda timeout.

Step Functions to the rescue!

Here’s where AWS Step Functions comes in. Step Functions make the task of orchestrating processes easier. They have built-in support for looping, waiting, and integrating with different functions and services, which makes them perfect for this sort of thing.

One of our team members put together this Step Function definition for creating a DynamoDB global table. It starts out by checking the state of the table, waiting until the table is ready for updates, then comparing the set of replicas with the desired set. You can only add one replica at a time, so the step function repeats the UpdateReplicas step until the actual state matches the desired state. Each step is very small and self-contained, usually only one or two API calls, and all of the waiting is done by Step Functions instead of in the Lambda function, so we’re not paying for idle time! Best of all, Step Functions can run for up to a week, so we didn’t need to worry about the 15-minute timeout any more.

There’s a small catch…

I wish I could use Step Functions directly to build CloudFormation custom resources instead of having to have a Lambda function that triggers the Step Function. #awswishlist
— Geoff Baskwill (@geoff_baskwill) February 24, 2021

CloudFormation doesn’t support direct integration with Step Functions as a custom resource provider yet, but we can use our old Lambda function trick to trigger the Step Function execution, and send the response back to CloudFormation when we get to an end state.

Wrapping up

When you love infrastructure-as-code and need a custom resource for something that CloudFormation doesn’t support, Lambda is usually a great solution. When you need a bigger hammer for complex orchestration or operations with lots of idle time, Step Functions can help get you there.

The goal is to retire this particular resource soon, as AWS tells us that they’ll have built-in support in CloudFormation for DynamoDB Global Tables in the very near future. That said, my team is happy that we were able to deliver an initial implementation with this workaround and provide value to our customers!

CloudFormation, Route 53, and ... EKS?

Thu, 22 Apr 2021 00:00:00 +0000

The other day I got halfway through writing a very irate support ticket to AWS, stopped to do some fact checking, and learned something deeply annoying.

One of the teams I work manages a bunch of services. One of these “services” is some common Amazon Route 53 infrastructure that is set up using AWS CloudFormation, and over the history of the project the deployment in the development account the team uses has been a little flaky. Each time we hit a deployment problem, the problem always turned out to be rate limiting. It never happened in production, so the flakiness didn’t quite get the attention that it could have.

Rate limiting in Route 53

Some background: Route 53 has a hard limit of 5 control plane requests per second per AWS account. For most folks, this is fine. However, it wasn’t working well for this team.

The team had raised support tickets and gotten advice like “CloudFormation will attempt to create your resources in parallel. One option to avoid rate limiting is to add DependsOn links to serialize the resource creation.” We weren’t super-happy with that answer, and there’s a CloudFormation roadmap item to fix it, but we needed something in the interim, and it worked … mostly.

On this day a deployment had failed again, and after going after the usual suspects and making sure that the resources were properly serialized, I got very irate. I was halfway through writing a support ticket:

We are still encountering Route 53 rate limits and our CloudFormation stack deployment / updates are intermittently failing, sometimes after only two resources are created. There are no Route 53 API calls being made by our applications, only through CloudFormation.

We are quite frustrated at this point and would like to request a session with a solution architect to help us understand how we should be doing this and

and I paused.

Check yourself before you wreck yourself

“There is no point in using the word ‘impossible’ to describe something that has clearly happened.” — Douglas Adams, Dirk Gently’s Holistic Detective Agency

“Is it true that there are no other Route 53 API calls being made?” I asked myself. A quick jaunt into AWS CloudTrail told me the answer, and also opened a gaping pit beneath my feet.

There were 436 Route 53 API calls made in the 2-minute period surrounding our CloudFormation failure. If you do the math, that’s 3.6 requests per second on average, so it’s not at all surprising that we maybe tipped over the limit of 5 at some point in there.

“But where are these coming from?” was my immediate question, and it was immediately answered.

Virtually all of these requests were being made by an EC2 instance that was part of an Amazon Elastic Kubernetes Service (EKS) cluster.

Talking through this with some other folks, I learned that they’d configured external-dns on the cluster, and that this behaviour is actually documented.

The production account doesn’t have the EKS cluster, so it’s not overwhelmed with Route 53 API calls, which explains why deployment never failed there.

Buh-bye

I wanted to decommission the cluster immediately, but unfortunately some teams still need it, so I wasn’t able to.

The external-dns documentation says that one workaround for the issue of the controller eating your entire Route 53 request budget is to extend the interval that the controller reconciliation loop runs at. In this particular cluster, the reconciliation loop was running every minute (the default!) to reconcile a set of records that change approximately never. I followed the instructions, set the interval to a week, and settled in to see what happened.

The first thing I noticed was that immediately the calls to Route 53 stopped. Not surprising, but great to see the confirmation. Several hours after the change, there were still no calls from the previously-misbehaving cluster.

All is well now, and I get to put away my detective hat for another day.

“The light works,” he said, indicating the window, “the gravity works,” he said, dropping a pencil on the floor. “Anything else we have to take our chances with.” — Douglas Adams, Dirk Gently’s Holistic Detective Agency

What I learned

First, CloudTrail was instrumental here. I’m still a novice, but I’m learning how powerful a tool it is. Once I knew what to look for, it was immediately obvious what the source of the rate limiting was. The events in CloudTrail identified the EC2 instance and even made it clear that the source of the requests was in an EKS cluster.

Second, I was reminded that Kubernetes is not a get-out-of-ops-free card. There is a lot of expertise involved in running Kubernetes well, even when you’re using a managed service like EKS. I knew this before, but this was an example of a cluster I didn’t even know existed (don’t worry: someone more responsible did know!) having side effects way outside its scope.

AWS re:Invent 2017

Tue, 05 Dec 2017 00:00:00 +0000

tl;dr

If I had to pick one word to describe AWS re:Invent 2017, it would be “popular”. There were over 43 000 attendees this year. The conference organizers and staff did an amazing job rising to the challenge, keeping folks corralled and working hard to manage expectations. I had a fun time; it was great to meet up with old friends, make a few new ones, and learn stuff at the same time.

Sessions had about 75% reserved seating and about 25% of the seats available for walk-ups, but with so many folks at the conference, popular sessions had a lot of waiting in line and a lot of unhappy faces on the people who didn’t make it in.

The conference was spread over a massive campus: 6 properties from the MGM at one end to the Venetian at the other; a good 30-minute hike if you’re fast and don’t get lost or caught in construction. Shuttles were available, but lines were long and people reported transit times of 60–90 minutes, so other than an early-morning trip on Monday, I just walked. A lot. I put in just over 60km during the week and am very happy to have had good shoes.

There were lots of interesting breakout sessions, chalk talks, and workshops; reports are favourable on the jams and hackathons. Based on this year, if I get to go again I’m going to try to go to chalk talks exclusively and fill in remaining time with a hackathon and a jam or two. The sessions are great, but they’re recorded and you can watch them in your copious spare time once you get home.

General activities

Registration

Registration opened just before 1pm on Sunday and was incredibly efficient. There were tons of folks available to hand out badges, and they had a separate section for people to try on this year’s hoodie before heading over to pick up their size, which eliminated a bottleneck from last year. I was in and out in single-digit minutes.

Sadly, I lost my hoodie in the crush of the crowd on Monday and had to use last year’s instead. Life is pain.

Midnight Madness

Modeled after an NCAA basketball tradition, AWS had a big party and then hit the court at midnight exactly with a new service annoucement: Amazon Sumerian, a 3D, VR, and AR service that lets folks create interactive environments. So far environments are single-user, but it’s not hard to see multi-user coming. There will be a marketplace for folks to create and sell 3D models that anyone can use in their environments, and there’s already support for chatbot actors called Hosts that can interact with users.

At 1am, Amazon announced support for AWS Elemental, a full suite of media services targeting folks who broadcast live and pre-recorded video.

This is the last time I’m going to mention new releases; you can get a full list from the AWS product announcements page. Be prepared to scroll.

Welcome Reception

The expo hall opened at 5pm Tuesday for the usual welcome reception; it had been open for most of the day but the reception added food and drinks. It’s very popular and there’s always a mad rush for swag. I had some good chats with vendors and the very smart folks in the AWS section, including the Education folks hidden away in the back corner. The most interesting thing I took away was that AWS has a program for high-school and college students that students can sign up for themselves or with a teacher’s help and learn all kinds of stuff about the cloud, with tons of content and credits available.

Sadly, I wasn’t able to say the right things to the Yubico folks to snag a free key. I want to get a 4 so that I can experiment with U2F, and I doubt they were handing those out.

Tuesday Night Live with Peter DeSantis (video)

This was a very interesting update on the vast networking infrastructure that AWS has around the world, and segued into a description of the incredible work they’ve done in compute.

The most inspiring part of the evening, though, was the Autodesk presenter talking about how advances in compute have enabled entirely new ways of working for their customers and how that’s going to help the world be a better place for the 10 billion people that will soon be here.

Wednesday Keynote with Andy Jassy (video)

This keynote came with a lot of mind-bubbling announcements, but I promised not to list them. It was a fun time and clearly Andy Jassy was excited to share the news with the world.

Thursday Keynote with Werner Vogels (video)

Werner Vogels’ keynote was more retrospective. I didn’t hike over to the MGM to watch the show and elected to stay in a remote viewing room. I think this was a mistake as I missed out on the energy and wasn’t as engaged.

re:Play

I’m not a big party person, so I just picked up a T-shirt for the folks at home and toured around the megastructures. Yes, megastructures — if you’re going to put on a party for 43 000 people, you need space. There were 3 huge tents and tons of activities, from video games to mini-golf to archery tag to dodgeball to a ball pit and many more, all set to the soothing tones of Team Ezy, Jen Lasher, and DJ Snake.

Breakout sessions and workshops (AWS video catalog)

I’m just going to dump in the list of sessions that I wanted to go to for future reference; there will be some sparse commentary for the ones that I actually made it to. There were over 1300 breakout sessions spread out over the five properties, so there was really no way to cover even a noticeable fraction of the ones I wanted to see.

Community Day

DVC202 — The Open Guide to AWS: A GitHub-Based Resource for Practitioners

The Open Guide to AWS is an open source writing project, which over the past year has become one of the most popular AWS resources on the web. It’s both a written resource on…

DVC302 — And You Thought You Knew Amazon EC2

When is the last time you explored the nooks and crannies of Amazon EC2? While you weren’t looking, AWS leveled it up with more features and capabilities than you can shake a…

DVC304 — Compliance and Top Security Threats in the Cloud — Are You Protected?

Compliance is necessary and a good thing. However, many compliant companies are still getting breached. In this talk, we discuss the importance of using a risk model to…

Enterprise

ENT346 — How AWS Runs Our Weekly Operations Meetings (Chalk Talk)

For more than 10 years, AWS has held a weekly meeting to review the operational performance of our services, identify new best practices, and share lessons learned from the…

IoT

IOT311 — Customer Stories of Things, Cloud, and Analytics on AWS

In this session, AWS IoT customers talk about the nuances, successes, and challenges of running large-scale IoT deployments on AWS. Hear from customers who have been…

I stumbled into this talk accidentally; I was waiting for a friend who was going to the session, so decided to wait in the walk-up line and ended up going in. The talk was a great discussion of how iRobot does IoT, I really enjoyed seeing the details and challenges that they encountered.

Containers

CON201 — Containers on AWS — State of the Union

Just over four years after the first public release of Docker, and three years to the day after the launch of Amazon EC2 Container Service, the use of containers has surged…

This was an interesting outline of what’s come out and what’s coming soon for containers in AWS. Container product manager Deepak Singh hinted that AWS Fargate, the new container cluster service, was only a step towards what they really want to do, which is allow customers to run containers without worrying at all about the servers that they’re running on.

CON213 — Hands-on Deployment of Kubernetes on AWS (Workshop)

In this workshop, attendees get hands-on experience using Kubernetes and Kops (Kubernetes Operations), as described in our recent blog. Attendees learn how to provision a…

CON302 — Building a CI/CD Pipeline for Containers on Amazon ECS

Containers can make it easier to scale applications in the cloud, but how do you set up your CI/CD workflow to automatically test and deploy code to containerized apps? In…

CON402 — Advanced Patterns in Microservices Implementation with Amazon ECS

Scaling a microservice-based infrastructure can be challenging in terms of both technical implementation and developer workflow. In this talk, AWS Solutions Architect Pierre…

Media

MAE401 — Designing for Disney/Marvel Studio-Grade Security (video)

Security is paramount for media storage and workloads and can directly impact a studio’s bottom line. As core media workloads move to the cloud, it’s imperative to examine…

Content Distribution

CTD301 — Amazon CloudFront Flash Talks: Best Practices on Configuring, Securing, Customizing, and Monitoring Your Distribution. (video)

In this series of technical flash talks, learn directly from Amazon CloudFront engineers about best practices on security, caching, measuring performance using Real User…

CTD401 — Taking DevOps Closer to the AWS Edge

Since last year’s ‘Taking DevOps to the Edge’, and with the introduction of AWS Lambda@Edge, the tools available to apply DevOps practices to your application edge have…

Serverless

SRV213 — Thirty Serverless Architectures in 30 Minutes (Chalk Talk)

Don’t blink because in this session, we quickly show you thirty different architectural patterns that you can use with AWS Lambda to solve everything from basic…

This talk was insanely popular. There were multiple repeats and I didn’t make any of them, but I really wanted to. Maybe there’s a recorded version of this talk somewhere out there? 爛

SRV301 — Optimizing Serverless Application Data Tiers with Amazon DynamoDB

As a fully managed database service, Amazon DynamoDB is a natural fit for serverless architectures. In this session, we dive deep into why and how to use DynamoDB in…

This was a cool session that covered (among other things) how to solve a couple of use cases that I’ve always struggled with, so I walked out a more-informed person.

SRV302 — Building CI/CD Pipelines for Serverless Applications (video)

Building and deploying serverless applications introduces new challenges for developers whose development workflows are optimized for traditional VM-based applications. In…

SRV305 — What’s New in Serverless (video)

Join us to learn what’s new in serverless computing and AWS Lambda. Dr. Tim Wagner, General Manager of AWS Lambda and Amazon API Gateway, will share the latest developments…

Fun times are always had in serverless-town. This session started off with cutting a server in half.

One thing I walked away from the session with is that cold start times have been cut by 80% this year, which is amazing. In a post-session chat, I found out that VPC cold starts are still going to be slow until they re-architect how Lambda attaches the ENI; I’d hoped that some of the container networking work released this year would help but apparently the sausage isn’t made that way.

SRV307 — Operating Your Serverless API in Production at Scale (video)

In this session, you will learn how to deploy, monitor and manage your serverless APIs in production. We will deep dive into advanced capabilities of API Gateway that enable…

SRV308 — Securing Serverless Applications Step-by-step (video)

You need a new approach to security for serverless applications. Classic approaches just don’t make sense, because tools and process can only take you so far…

This was an excellent talk. Mark Nunnikhoven’s practical, clear approach to application security made this a fun walk-through of practical steps that developers and organizations can take to make their serverless applications more secure. 100% recommended.

SRV310 — Designing Microservices with Serverless

When designing microservices there are a number of things to think about. Just for starters, the bounds of their functionality, how they communicate with their dependencies,…

SRV320 — Best Practices for Using AWS Lambda with RDS/RDBMS Solutions (Chalk Talk)

Are you considering using AWS Lambda with your RDS/RDBMS solutions, but don’t know where to start? In this session, we look at recommended architectural patterns and best…

SRV321 — Serverless Reactive Microservices on AWS (Chalk Talk)

In this talk, we will cover the concepts of Reactive Microservices Architecture, a set of patterns and best practices for implementing backend systems with very high scaling…

SRV329 — Lessons in Serverless Architecture for IoT from iRobot (Chalk Talk)

The event-driven nature of serverless architecture is a natural fit for IoT use cases. In this chalk talk, we cover iRobot’s use case and requirements, detail the serverless…

SRV401 — Become a Serverless Black Belt: Optimizing Your Serverless Applications (video)

Are you an experienced serverless developer who wants a handy guide to unleash the full power of serverless architectures for your production workloads? Do you have questions…

SRV403 — Serverless Authentication and Authorization: Identity Management for Serverless Applications

Many serverless applications need a way to manage end user identities and support sign-ups and sign-ins. Join this session to learn real-world design patterns for…

SRV425 — Serverless OAuth: Authorizing Third-Party Applications to Your Serverless API (Workshop)

By using serverless architectures, startups, and enterprises are building and running modern applications and services with increased agility and simplified scalability, all…

Security

SID205 — Building the Largest Repo for Serverless Compliance-as-Code (video)

When you use the cloud to enable speed and agility, how do you know if you did it right? We are on a mission to help builders follow industry best practices within security…

SID312 — DevSecOps Capture the Flag (Workshop)

Notice: This Workshop requires a laptop computer and an active AWS account with Administrator privileges. In this Capture the Flag workshop, we divide groups into teams and…

SID332 — Identity Management for Your Users and Apps: A Deep Dive on Amazon Cognito (video)

Learn how to set up an end-user directory, secure sign-up and sign-in, manage user profiles, authenticate and authorize your APIs, federate from enterprise and social…

SID343 — User Management and App Authentication with Amazon Cognito (Workshop)

Notice: This Workshop requires a laptop computer and an active AWS account with Administrator privileges. Are you curious about how to authenticate and authorize your…

SID401 — Let’s Dive Deep Together: Advancing Web Application Security (Chalk Talk)

Beginning with a recap of best practices in CloudFront, AWS WAF, Route 53, and Amazon VPC security, we break into small teams to work together on improving the security of a…

SID402 — An AWS Security Odyssey: Implementing Security Controls in the World of Internet, Big Data, IoT and E-Commerce Platforms (Workshop)

Notice: This Workshop requires a laptop computer and an active AWS account with Administrator privileges. This workshop will give participants the opportunity to take a…

Financial Services

FSV301 — Security Anti-Patterns: Mistakes to Avoid

At AWS, security is job zero. Our infrastructure is architected for the most data-sensitive, financial services companies in the world. We have worked with global enterprises…

Big Data

ABD301 — Analyzing Streaming Data in Real Time with Amazon Kinesis

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. In this session,…

ABD401 — How Netflix Monitors Applications in Near Real-Time with Amazon Kinesis

Thousands of services work in concert to deliver millions of hours of video streams to Netflix customers every day. These applications vary in size, function, and technology,…

Architecture

ARC316 — Getting from Here to There: A Journey from On-premises to Serverless Architecture

In this session, go on a journey from traditional, on-premises applications and architecture to pure cloud-native environments. This transformative approach highlights the…

ARC401 — Serverless Architectural Patterns and Best Practices

As serverless architectures become more popular, customers need a framework of patterns to help them identify how they can leverage AWS to deploy their workloads without…

Networking

NET203 — Using Amazon VPC Flow Logs to Do Predictive Security Analytics (Workshop)

Ready to secure your network and application in near real-time using Amazon VPC flow logs and AWS Web Application Firewall (WAF)? In this workshop, we incorporate advanced…

This seemed like it was going to be a really interesting workshop; unfortunately the majority of the time was spent in lead-up and very little time was spent on the parts that I was interested in. I’m hoping to put together something that covers the data enrichment parts of the process and gets into the machine learning — certainly not an area of expertise for me, but perhaps something I can post about as I learn.

NET403 — Deep Dive: AWS Direct Connect and VPNs

As enterprises move to the cloud, robust connectivity is often an early consideration. AWS Direct Connect provides a more consistent network experience for accessing your AWS…

DevOps

DEV401 — Advanced DevOps Practices for AWS (Workshop)

Advanced DevOps Practices for AWS is designed to provide concise and prescriptive guidance on how to build proper DevOps solutions in AWS using common patterns and best…

Mobile

MBL305 — Implement User Onboarding, Sign-Up, and Sign-In for Mobile and Web Applications with Amazon Cognito

Learn how to use Amazon Cognito to build the user identity management workflows, including user on-boarding, sign-up, and sign-on for mobile and web applications.

Machine Learning

MCL212 — AWS DeepLens workshop: Building Computer Vision Applications (Workshop)

In this workshop, developers have the opportunity to learn how to build and deploy computer vision models, such as face detection and object analysis using the AWS DeepLens…

There was no way that I was going to make it into one of these sessions; they were easily the most popular workshops of the week, announced during the Andy Jassy keynote, with 15 repeat sessions throughout Wednesday and Thursday. Two lucky colleagues did get in and had fun with their new DeepLens devices; I’m hoping they share their experience with the rest of us.

Databases

DAT405 — Workshop on Advanced Design Patterns for Amazon DynamoDB (Workshop)

Join us for the first-ever Amazon DynamoDB practical hands-on workshop. This session is designed for developers, engineers, and database administrators who are involved in…

I am so bummed that I missed this. It would have been fantastic to get some non-trivial practical experience with DynamoDB. I wish AWS posted their workshop material online so that folks could play around with the examples.

I feel like a traitor. Or a fool.

Sun, 18 Jun 2017 00:00:00 +0000

https://www.flickr.com/photos/andrewfhart/8106200690

Recently I signed up for the A Cloud Guru AWS advanced networking certification prep course put together by Adrian Cantrill.

After the obligatory OSI 7-layer modules, which I watched because you never know, things rapidly got interesting. I found myself learning about BGP, AWS VPC peering, how to multi-home EC2 instances with ENIs, how to set up a shared-services network that could talk to peered VPCs with overlapping CIDR ranges (turns out it’s not even that difficult), and all kinds of really cool stuff.

I got nerd-sniped. Big time. Networking was a big part of my early career, and the course scraped a bunch of rust off and added a whole new layer of awesome things to learn about. My brain was all hopped up on learning-juice, spinning and carousing and saying to itself “THIS IS SO COOL.”

Then something hit me like a cross-town bus.

Everything I’d just been geeking out about is completely irrelevant in the serverless world.

Last year I went to the first Serverlessconf. It was great, tons of fun; I met some really smart folks and learned that people are doing real work in a whole new way. There is a whole new solution space becoming available, a solution space where, as Tim Wagner put it, you don’t have to worry about whether you have cattle or pets, you just show up at the drive-through and get the burger you were looking for. It’s arguable that the vast majority of problems people are trying to solve today fit into this space: take an event, do some transforms on it, send it somewhere else, maybe do some aggregation and correlation later. It’s a way to focus on your core competency and let other folks worry about the servers, the mechanics of routing those events to your code, instantiating it, and keeping everything available.

As someone who’s pretty keenly interested in the future of serverless, it seems either traitorous or foolish to be taking a course on advanced networking.

What faithful, intelligent person would invest time in learning all sorts of networking shenanigans when there’s so much to learn in serverless space?

It’s a week later now, and I’ve realized that the answer to my question is “people like me.” People who think that even though there is a huge bright shiny serverless future out there, someone still has to know how it all fits together. People who think that the road between today and that shiny future will have more than one bump in it. People who just like learning new things every day.

AWS Certified Security Specialty

Mon, 21 Nov 2016 00:00:00 +0000

A good friend of mine has a habit of throwing gauntlets around like they’re nothing. “So, Geoff,” he’ll start casually, “did you know there are some new AWS specialty certifications in beta now?”

This is usually about the time that I realize I’m in for a world of pain, uh, learning. It turns out that this time AWS has come up with three exams that I’ll conservatively call pro-level.

I’ve been learning a lot over the past year and a bit, negotiating a career transition from telecom to security, so I’d consider myself a firm novice when it comes to all this stuff, but it’s not like we were doing nothing over in the old world, so I figured I’d check out the landscape for the security certification. I find that preparing for certifications is a great way to learn at least the theory, and sometimes get a bit of practice in as well. When some detail doesn’t click right away, it’s easy enough to spin something up and try it out to figure out how things are working.

Keep in mind that the “AWS recommendations” includes, as a single bullet point among several, “AWS documentation.” So, since we’re talking about security, let’s take that as the reference docs, programmers’ guides, and best practices for 50+ services.

Next is the alphabet soup: as @marknca says, “candidates should be familiar with ISO 27001, PCI DSS, HIPAA, SAS 70, SOC 1, FISMA, and NIST.” That’s “and”, not “or”, folks.

Right now I’m on the edge. The reading so far has been pretty much stuff I’ve seen before one way or another, so maybe, just maybe, I’ll give the exam a shot.