Leveraging the Fan Out Pattern to Build Scalable Systems

In today’s world, businesses need to handle massive amounts of data, and the ability to scale quickly is paramount. In this article, we will explore the Fan Out Pattern, a design pattern that allows us to scale our systems horizontally. We will also look at how we can implement this pattern using AWS services (SNS and SQS) and automate the process with Terraform.

Building Blocks

Before we dive into the details of the Fan Out Pattern, let’s take a look at the building blocks that we will use to implement it.

Amazon Simple Notification Service (SNS)

Amazon Simple Notification Service (SNS) is a fully managed pub/sub messaging service that allows you to send messages to multiple subscribers. SNS is a great choice for building scalable systems, as it allows you to scale your system horizontally by adding more subscribers to your topic.

SNS Figure 1: How SNS works: each subscriber get all messages

Each message that you send to SNS is delivered to all subscribers. SNS also supports message filtering, which allows you to filter messages based on attributes. This feature is useful when you want to send messages to a subset of subscribers.

SNS Filters Figure 2: SNS Message Filtering for payment and shipping services

You can integrate SNS with other AWS services, such as AWS Lambda, Amazon Kinesis and Amazon SQS, which we will focus on in the next section.

Amazon Simple Queue Service (SQS)

AWS SQS is a service that helps to decouple and scale distributed systems, applications, and microservices by allowing you to send, store, and receive messages between software components or services with high reliability and scalability.

SQS Figure 3: How SQS works

The main difference between AWS SQS and SNS is their purpose and messaging models. SNS is a pub/sub messaging service, while SQS is a message queue. SNS allows you to send messages to multiple subscribers, while SQS allows you to sent messages to a queue and stored until they are processed by a consumer.

Both services can be used together in a decoupled architecture to build scalable and reliable distributed systems.

What is the Fan-Out Pattern?

The fan-out pattern is a messaging pattern that allows you to distribute messages to multiple subscribers, enabling you to scale your system quickly. The pattern is also known as the publish-subscribe pattern, and it is used in many systems, including event-driven architectures.

With this pattern, you can publish a message to a single topic, which is then delivered to multiple subscribers asynchronously (we spread the load across multiple subscribers). The pattern is useful when you want to scale your system horizontally, and you don’t want to have a single point of failure.

Fanout Figure 4: Fan Out Pattern with AWS SNS and SQS

In AWS we can use the SNS and SQS services to implement the Fan Out Pattern. SNS can be used to replicate messages to multiple subscribers, while SQS can be used to store messages until they are processed by a consumer(s).

We can simplify use AWS SNS and SQS to implement the Fan Out Pattern as follows:

We publish a message to SNS.
SNS replicates the message to multiple subscribers (SQS queues).
Each subscriber can have message filters enabled.
Each message in the queue is processed by a consumer(s).

Please note, that each SQS queue can have multiple consumers, which allows us to scale our system horizontally and independently.

The Fan Out pattern with AWS SNS and SQS has several advantages, including:

Scalability - we can scale our system horizontally by adding more subscribers to our topic.
Decoupling - we can decouple our system by using SNS and SQS.
Reliability - we can use SQS to store messages until they are processed by a consumer(s).

Example implementation using Terraform

In this section, we will look at an example implementation of the Fan Out Pattern using AWS SNS and SQS. We will use Terraform to automate the process of creating the resources.

Prerequisites

To follow along with this example, you will need the following:

Terraform installed on your machine,
AWS account.

Before we start, we need to configure Terraform with AWS provider. You can find more information about how to do that in the official Terraform documentation.

Let’s create a new directory and initialize Terraform and then create a new file called providers.tf and add the following code:

# providers.tf 

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

provider "aws" {
  region = "eu-central-1"
}

The code above configures the AWS provider and sets the region to eu-central-1.

Necessary resources

To implement the fan-out pattern using AWS SNS and SQS, you will need to create an SNS topic and multiple SQS queues.

We will create the following resources:

SNS Topic,
Two SQS Queues.

Desired architecture can be seen in the following diagram:

Figure 5: Desired architecture: one SNS Topic and two SQS Queues

Let’s create a new file main.tf and define the SNS Topic and two SQS queues:

# main.tf

resource "aws_sns_topic" "fanout_topic" {
  name = "fanout-topic"
}

resource "aws_sqs_queue" "fanout_sqs_a" {
  name = "fanout-sqs-a"
}

resource "aws_sqs_queue" "fanout_sqs_b" {
  name = "fanout-sqs-b"
}

Now, we have all the necessary resources defined, but we need to connect them together. We will do that by creating a subscription for each SQS queue.

We will add the following code to main.tf file:

# main.tf

resource "aws_sns_topic_subscription" "fanout_sqs_a_subscription" {
  protocol             = "sqs"
  raw_message_delivery = true
  topic_arn            = aws_sns_topic.fanout_topic.arn
  endpoint             = aws_sqs_queue.fanout_sqs_a.arn
}

resource "aws_sns_topic_subscription" "fanout_sqs_b_subscription" {
  protocol             = "sqs"
  raw_message_delivery = true
  topic_arn            = aws_sns_topic.fanout_topic.arn
  endpoint             = aws_sqs_queue.fanout_sqs_b.arn
}

One last thing that we need to do is to add a policy to the SNS Topic that allows the SQS queues to subscribe to the topic. We’ll need to update the main.tf file with the following code:

# main.tf

data "aws_iam_policy_document" "sqs_policy" {
  statement {
    effect = "Allow"
    
    actions = ["sqs:SendMessage"]

    resources = [
      aws_sqs_queue.fanout_sqs_a.arn,
      aws_sqs_queue.fanout_sqs_b.arn,
    ]

    principals {
      type        = "Service"
      identifiers = ["sns.amazonaws.com"]
    }

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_sns_topic.fanout_topic.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "fanout_sqs_a_subscription" {
  policy    = data.aws_iam_policy_document.sqs_policy.json
  queue_url = aws_sqs_queue.fanout_sqs_a.id
}

resource "aws_sqs_queue_policy" "fanout_sqs_b_subscription" {
  policy    = data.aws_iam_policy_document.sqs_policy.json
  queue_url = aws_sqs_queue.fanout_sqs_b.id
}

Optionally we can create outputs.tf file to get additional information about the created resources:

# outputs.tf

output "fanout_topic_arn" {
  value = aws_sns_topic.fanout_topic.arn
  description = "ARN of the SNS topic"
}

output "fanout_sqs_a_url" {
  value = aws_sqs_queue.fanout_sqs_a.url
  description = "URL of the SQS queue A"
}

output "fanout_sqs_b_url" {
  value = aws_sqs_queue.fanout_sqs_b.url
  description = "URL of the SQS queue B"
}

Now, we can run terraform init and terraform apply to create the resources. As the output, we will get the ARN of the SNS Topic and URLs of the SQS queues.

Testing the implementation

To test the implementation, we will use the AWS CLI to publish a message to the SNS Topic. We will use the following command:

aws sns publish --topic-arn <SNS_TOPIC_ARN> --message "Hello World" --region eu-central-1

As the output, we will get the message ID. We can use the message ID to check if the message was delivered to the SQS queues. We will use the following command:

aws sqs receive-message --queue-url <SQS_QUEUE_URL> --region eu-central-1

and as a result, we will get the following output:

{
    "Messages": [
        {
            "MessageId": "...",
            "ReceiptHandle": "...",
            "MD5OfBody": "3e25960a79dbc69b674cd4ec67a72c62",
            "Body": "Hello world"
        }
    ]
}

After testing the implementation, we can run terraform destroy to remove all the resources.

You can find the full code in this Gist.

Conclusion

In conclusion, the Fan Out Pattern is a messaging pattern that enables distributing messages to multiple subscribers, allowing the system to scale quickly. By using AWS SNS and SQS services, we can implement this pattern, and with Terraform automation, we can create the necessary resources quickly and easily.

The Fan Out Pattern has several advantages, including scalability, decoupling, and reliability, which make it a popular choice in building distributed systems. With the right implementation, businesses can handle massive amounts of data and scale their systems horizontally, ensuring that their services are available and reliable for their customers.

Building Blocks#

Amazon Simple Notification Service (SNS)#

Amazon Simple Queue Service (SQS)#

What is the Fan-Out Pattern?#

Example implementation using Terraform#

Prerequisites#

Necessary resources#

Testing the implementation#

Conclusion#