Member-only story
SQS Consumer Design: Achieving High Scalability while managing concurrency in Go
Recently I’ve had to redesign an async queuing system. After researching several options, I went with AWS Simple Queue Service(SQS) and Simple Notification Service(SNS). With a combination of these two services, I was able to duplicate and improve the previous (and failing) RabbitMQ set-up. The great thing about the AWS offering is that they are built from the ground up for micro-services. They abstract the vast majority of the complexity of a complicated async messaging system, from handling horizontal scaling via the message waiting period to Dead Letter Queues and a multitude of other features, right out of the box.
As per AWS suggestion on scaling, they recommend scaling horizontally across multiple services as well as within a single service by utilizing multiple threads. In this blog post, we will focus on the best way to retrieve and handle messages from SQS using Golang within a single service by utilizing multiple threads.
Throughout this example, we are going to be dealing with an AWS queue with 20,000 messages. We will measure three different techniques of managing concurrent workers in Golang, as well as a synchronous control test. We will measure the total number of items consumed over the course of 3 minutes and take the average for a messages consumed per minute (mpm).
