One aspect I did not include in My Personal Ecosystem is the way I use AWS Lambda to perform certain tasks using serverless code. I will start off by saying that I do not think serverless code is a good idea as a general purpose solution. It has been my experience that such solutions tend to be more expensive than running a pod in a cluster I already have in place. However, it is sometimes useful for certain specific cases. I have the following Lambdas in my setup:

EC2-Watch: This is a straight up rip-off of something we did at ZipRecruiter. Any time an EC2 instance comes up or down, this Lambda is triggered and it examines the characteristics of the new instance to decide how to tag it and its volumes. For example, it can detect persistent volumes attached to Kubernetes nodes and tag those so they get backed up.
EC2-Backup: This is the counterpart to EC2-Watch and ensures that periodic snapshots get made of EBS volumes marked for backup by EC2-Watch. It also deletes old snapshots. As of this writing, this Lambda is pretty idle since none of my current containers need backups. (But, if I decide to setup a Minecraft server again, that will change… :-p)
WP-Notifier: This is supposed to send me a text message every time a new version of WordPress is released. (Not working at the moment because I have to fill out paperwork that I haven’t gotten around to, not technical ones.)
Refresh-WP: This performs a periodic check of the public WordPress image that my personal WordPress image is based upon. Whenever that image is update, I trigger a build to ensure that my WordPress images are very up to date.
S3-Backup: This performs backups of the certain S3 buckets that I feel are vulnerable to data loss (mostly because I don’t trust my own code 100% and want a way to recover in case I make a mistake).

To layout how I make use Lambda, I’m going to single out the newest one, the S3-Backup and share how it works and how I deploy it.

The Code

The code itself is very typical and nothing special. I use no abstractions and nothing fancy. It just needs to copy a file from one bucket to another. The files I am backing up are rarely uploaded and so this will trigger once in a blue moon.

package main

import (
	"context"
	"path"

	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
	"go.uber.org/zap"
)

const backupBucket = "backup-bucket"

func handleBackup(ctx context.Context, s3e events.S3Event) {
	prodLogger, err := zap.NewProduction()
	if err != nil {
		panic(err)
	}

	logger := prodLogger.Sugar()

	sess, err := session.NewSession()
	if err != nil {
		logger.Panic(err)
	}

	s3c := s3.New(sess)

	for _, r := range s3e.Records {
		srcBkt, srcObj := r.S3.Bucket.Name, r.S3.Object.Key
		if srcBkt == backupBucket {
			logger.Errorf("Cannot backup to self: %q", backupBucket)
			continue
		}

		src := path.Join(srcBkt, srcObj)

		_, err := s3c.CopyObjectWithContext(ctx, &s3.CopyObjectInput{
			Bucket:     aws.String(backupBucket),
			CopySource: aws.String(src),
			Key:        aws.String(src),
		})
		if err != nil {
			logger.Panic(err)
		}

		logger.Infof("Backup %q to %q", src, backupBucket)
	}
}

func main() {
	lambda.Start(handleBackup)
}

The code above is slightly elided and slightly simplified, but just barely. Hard coding the backup bucket is bad juju, so if you copy this solution, I would recommend you read the backup bucket as an environment variable. I leave that as an exercise for the reader.

And just to explain what the code is doing:

It sets up the AWS SDK client and my log tooling.
It iterates through the incoming events that notify it that an object has been uploaded to a bucket in S3.
It copies source-bucket/source-object to backup-buckiet/source-bucket/source-object, performing the backup.
It logs the action for my later inspection via CloudWatch.

My actual code does a little more log setup and configuration, but that’s about it.

The Build

I have a Makefile in place that builds this program while making sure to set:

GOOS=linux
GOOS=amd64

Because sometimes I run make from an Apple M2. Then, I package it up into a zip-file.

The Deploy

I then deploy the above code using Terraform. I’m going to break down my Terraform file here so I can explain what each piece does.

Backup Bucket

The backups have to go somewhere, so I use my usual S3 bucket creation module to setup the bucket.

module "cloud-backup-bucket" {
    source = "./s3/standard"
    bucket = "backup-bucket" # not the real name
}

I will explain this module in more detail under The Setup below. For now, it’s enough to know that it sets up an aws_s3_bucket with the given bucket name.

Lambda Function

At the core of the system is the Lambda function configuration.

resource "aws_lambda_function" "s3-backup" {
    filename = "build/s3-backup.zip"
    function_name = "S3Backup"
    role = aws_iam_role.s3-backup.arn
    handler = "main"
    source_code_hash = filebase64sha256("build/s3-backup.zip")
    runtime = "go1.x"
    timeout = 100
}

This tells Terraform to send the contents of the named zip file to the Lambda function named “S3Backup”. When run, it will execute the main program using a Go runtime (though, it should be completely self-sufficient and require nothing but what’s in the binary itself).

Backup Role

One of the nice things about Lambda’s is that IAM roles are seamlessly built-in. I can grant my Lambda exactly the permissions needed to run it and nothing more. This is one of the reasons I stick with AWS even though it’s more costly. Fine-grained access controls help me sleep better.

resource "aws_iam_role" "s3-backup" {
    name_prefix = "S3Backup-"
    assume_role_policy = data.aws_iam_policy_document.lambda-assumption.json
}

data "aws_iam_policy_document" "lambda-assumption" {
    statement {
        actions = [ "sts:AssumeRole" ]
        principals {
            type = "Service"
            identifiers = [ "lambda.amazonaws.com" ]
        }
        effect = "Allow"
    }
}

This defines a role and grants the Lambda service the rights to assume it. The role configuration for the Lambda function above determines which role Lambda chooses to assign.

Backup Role Permissions

And then I have to assign that role the permissions required to do the job.

data "aws_iam_policy_document" "s3-backup-write" {
    statement {
        effect = "Allow"
        actions = [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents",
        ]
        resources = [ "arn:aws:logs:*:*:*" ]
    }

    statement {
        effect = "Allow"
        actions = [
            "s3:PutObject"
        ]
        resources = [
            module.cloud-backup-bucket.bucket-arn,
            "${module.cloud-backup-bucket.bucket-arn}/*"
        ]
    }

}

resource "aws_iam_role_policy" "s3-backup" {
    name = "S3Backup"
    role = aws_iam_role.s3-backup.id
    policy = data.aws_iam_policy_document.s3-backup-write.json
}

The first statement in the policy allows the Lambda function to log output to CloudWatch making it possible for me to debug it at runtime. The second statement grants the function the permissions required to write the backup files to the backup bucket. It does not, grant the ability to perform backups yet because it still needs read permissions on some source bucket or buckets. That is taken care of in my ./s3/standard module, which we will talk about next.

The Setup

Now that the Lambda is deployed via the configuration above and ready to perform backups, we need to configure an S3 bucket to trigger those backups. This is the final part of the setup. All of it is performed by using my ./s3/standard Terraform module, which allows any of my applications to configure backups as needed with just a few extra lines of configuration.

So, let’s say I have a bucket named “image-files” that I want backed up. Here’s the module configuration for it:

module "image-files-bucket" {
    source = "./s3/standard"
    bucket = "image-files"
    backup = {
        role-id = aws_iam_role.s3-backup.id
        lambda = aws_lambda_function.s3-backup
    }
}

With a little extra work, I could probably change that to backup-enabled = true, but this works for my simple purposes for now, so we’ll stick with it for the moment.

What does it do?

Module Inputs

At the top of my module, I have the following definitions:

variable "bucket" {
  type = string
}

variable "backup" {
  type = object({
    role-id = string
    lambda = object({
      arn = string
      function_name = string
    })
  })
  default = null
}

That defines my input. In particular, notice that lambda is defined as an object with arn and function_name properties, which allows me to pass the outputs from aws_lambda_function.s3-backup directly into the module.

Module Outputs

Also at the top are listed the outputs, which just include the bucket ARN, which is the only attribute I’ve needed from this module so far:

output "bucket-arn" {
  value = aws_s3_bucket.bucket.arn
}

Bucket Creation

All the buckets created by ./s3/standard are private to me, so:

resource "aws_s3_bucket" "bucket" {
  bucket = var.bucket
}

resource "aws_s3_bucket_ownership_controls" "bucket" {
  bucket = aws_s3_bucket.bucket.id
  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

With that in place, the bucket is created and privacy is enforced.

Backup Notifications

Now comes the stuff for triggering the backups whenever an object is created in a bucket being backed up. Most of my buckets are not backed up. As such I do not want notifications sent to the backup Lambda unless I ask for backups. This is the code that configures those notifications, but only if the backup variable is set on the module inputs:

locals {
  lambdas = var.backup != null ? [ var.backup.lambda ] : []
}

resource "aws_s3_bucket_notification" "bucket" {
  bucket = aws_s3_bucket.bucket.id

  dynamic "lambda_function" {
    for_each = local.lambdas
    content {
        lambda_function_arn = lambda_function.value.arn
        events = [ "s3:ObjectCreated:*" ]
    }
  }
}

resource "aws_lambda_permission" "cloudwatch-to-s3-backup" {
  count = var.backup != null ? 1 : 0
  statement_id = "AllowCloudWatchToCallS3Backup"
  action = "lambda:InvokeFunction"
  function_name = var.backup.lambda.function_name
  principal = "s3.amazonaws.com"
  source_arn = aws_s3_bucket.bucket.arn
}

The local block sets up a list that is used to control the dynamic block. If var.backup is null, the bucket notification resource is created, but is just left with defaults. If var.backup is not null, then a single lambda_function block is defined that will send S3 event notifications every time an object is created in the configured bucket (object creation covers any write operation to an S3 bucket).

The second resource grants the Lambda the permission required to receive any S3 events from this bucket. Without which, the bucket would try to send the events, but Lambda wouldn’t have permission to receive them. The count acts as an if-statement, enabling or disabling the block based on whether backups are configured.

If we stopped here, though, the backup Lambda would get the notification that a backup was needed, but would not have permission to read the object that needed to be backed up in order to copy it. One more resource is required.

Backup Permissions

Here is how we grant my program code the permissions required to read the object to be copied so it can be sent to the backup bucket. These permissions are paired with the write permissions defined previously allow backups to be performed, reading the source object and then writing the destination object.

data "aws_iam_policy_document" "s3-backup-read" {
  count = var.backup != null ? 1 : 0
  statement {
    effect = "Allow"
    actions = [
      "s3:GetObject",
      "s3:ListBucket",
    ]
    resources = [
      aws_s3_bucket.bucket.arn,
      "${aws_s3_bucket.bucket.arn}/*"
    ]
  }
}

resource "aws_iam_role_policy" "s3-backup-read" {
  count = var.backup != null ? 1 : 0
  name = "S3Backup-${aws_s3_bucket.bucket.id}"
  role = var.backup.role-id
  policy = data.aws_iam_policy_document.s3-backup-read[count.index].json
}

Strictly speaking, the s3:ListBucket permission probably isn’t necessary, but I included it anyway. I should probably remove it and see if that breaks anything. I don’t think it will in this case, but I don’t think it will harm anything to leave it, so…

As mentioned above, count acts as an if-statement, enabling or disabling the block based on whether backups are configured.

The Conclusion

There are a lot of little pieces to configure to get something like this setup, but none of them are very complex in and of themselves. The main complexity in using a service like AWS is knowing how fine-grained the permissions are. Whenever configuring an AWS service, you have to ask questions like:

I’m running some code, what permissions will it have and how do I grant them?
I’m sending data across services, how do I configure what data to send?
I’m sending data across services, how do I grant the receiver permission to receive the data?

Often AWS requires you to grant permissions to services receiving as well as tell it what to send. And then, to grant another set of permissions to the running code.

And to reiterate what I said above, I really don’t think Lambda makes sense for most use-cases. But in specific cases where you need to run code based upon events in the AWS ecosystem, it can be a good fit, especially if you aren’t triggering very often. If these jobs were frequent, I might worry about the expense and switch to a message queue and a job running in my cluster, but for the tasks I mentioned above, Lambda has proven to be an excellent fit.

Cheers.

Serverless S3 Backups