Fanning out S3 events without a Lambda in the middle

S3 bucket notifications allow one destination per event type prefix combination, which is why so many architectures end up with a "router" Lambda whose only job is to copy a message onto three queues. Turning on EventBridge notifications removes that function entirely: the bucket publishes to the default event bus, and each consumer owns its own rule.

Turning on the notification

One flag on the bucket, and every object-level event lands on the default bus.

resource "aws_s3_bucket" "uploads" {
  bucket = "the-cloud-engineer-uploads"
}

resource "aws_s3_bucket_notification" "uploads" {
  bucket      = aws_s3_bucket.uploads.id
  eventbridge = true
}

resource "aws_cloudwatch_event_rule" "thumbnail_requested" {
  name        = "uploads-thumbnail-requested"
  description = "Image uploads that need a thumbnail"

  event_pattern = jsonencode({
    source      = ["aws.s3"]
    detail-type = ["Object Created"]
    detail = {
      bucket = { name = [aws_s3_bucket.uploads.id] }
      object = { key = [{ prefix = "incoming/" }, { suffix = ".png" }] }
    }
  })
}

The pattern is worth reading twice. A list under object.key is an OR, so that rule matches keys under incoming/ or keys ending in .png. If you need both conditions, use $or explicitly or split the rule.

Giving each consumer its own queue

Each consumer gets a queue and a redrive policy it controls, so a slow consumer cannot back up the others.

Resources:
  ThumbnailQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: uploads-thumbnail
      VisibilityTimeout: 120
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt ThumbnailDlq.Arn
        maxReceiveCount: 5

  ThumbnailDlq:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: uploads-thumbnail-dlq
      MessageRetentionPeriod: 1209600

Verifying the wiring

Put an object, then read the queue. If the rule matched, the message body is the full EventBridge envelope.

aws s3 cp ./fixture.png s3://the-cloud-engineer-uploads/incoming/fixture.png

aws sqs receive-message \
  --queue-url "$THUMBNAIL_QUEUE_URL" \
  --max-number-of-messages 1 \
  --wait-time-seconds 20 \
  --query 'Messages[0].Body' --output text | jq '.detail.object'

Consuming the envelope

The shape is stable enough to type once and reuse. Note that size is absent for delete events, which is the detail that bites people who assume one interface covers every detail-type.

const parseObjectCreated = (body) => {
  const envelope = JSON.parse(body);

  if (envelope["detail-type"] !== "Object Created") {
    return null;
  }

  return {
    bucket: envelope.detail.bucket.name,
    key: decodeURIComponent(envelope.detail.object.key.replace(/\+/g, " ")),
    size: envelope.detail.object.size,
    etag: envelope.detail.object.etag,
  };
};

The trade-off

EventBridge adds a few tens of milliseconds compared with a direct bucket-to-SQS notification, and the default bus has no ordering guarantee you did not already lack. What you get back is a routing layer you can change without touching the bucket, and one less function to patch.