SNS + SQS => Fanout

What is this ‘fanout’?

Fanout is a common design pattern used in cloud applications, to take advantage of parallel processing.

Parallel processing — example

Let us see how OTT providers like Youtube, Netflix, and Amazon Prime use this pattern to save time in solving their important time-consuming tasks of video transcoding, optimized for different devices and network conditions.

Videos and movies you see over the internet (a.k.a OTT), need to be transcoded in multiple formats. During playback on different devices from HD/UHD TVs to laptops to mobile phones, one of the encoded videos will be chosen based on network and device capabilities.

Data flow in video processing — Fanout

1. Video/movie producers upload their high-quality videos to the video platform.
2. Many of the video producers - use AWS backed-up technology. Typically this results in the video being uploaded into S3.
3. This video upload (file upload) event publishes the message to the SNS topic, Typically multiple processing units (specific to different video quality) listen to this event.
4. All of them will start processing immediately (in parallel) and upload the resulting video back to S3 and mark videos ready for distribution.

Ref: https://aws.amazon.com/blogs/compute/messaging-fanout-pattern-for-serverless-architectures-using-amazon-sns/

Fanout implementation with SNS & SQS

AWS Simple Queue Service (SQS) and Simple Notification Service (SNS) are important “glue” components for scalable, cloud-based applications. These are often used in event-driven architecture. Pipelining these 2 services lead to fan-out messages. Let us dive deep into how to achieve this out of the box from AWS's offering.

SNS is an AWS messaging service(one to many) used mainly to send messages to multiple subscribed endpoints. Typically we need to create an ‘SNS topic’, a logical access point that acts as a communication channel. With AWS CLI, SDK, or the web console, you can start publishing messages on this topic. The SNS topic will forward the messages to its subscribers. We can add more than 1 SQS queue as subscribers for processing.

SQS is another AWS messaging service(one to one) used in asynchronous processing, typically used to decouple different systems.

Fanout: Create an ‘SNS Topic’ + multiple ‘SQS Queues’ and subscribe these multiple queues to the same topic to replicate a message over multiple queues. The above scenario will replicate a message to all queues, and this allows for parallel asynchronous processing.

You can see => how a message published to an SNS topic is distributed to several SQS queues in parallel. Using this pattern, you can build applications that take advantage of parallel, asynchronous processing.