Amazon SQS stands for Simple Queue Service. SQS was the first service available in AWS. Amazon SQS (Simple Queue Service) is a managed message queue service offered by Amazon Web Services (AWS). It provides an HTTP API over which applications can submit items into and read items out of a queue. The queue itself is fully managed by AWS, which makes SQS an easy solution for passing messages between different parts of software systems that run in the cloud.
Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.
SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.
Amazon SQS Queue Types :
There are two types of Queue:
- Standard Queues (default)
- FIFO Queues (First-In-First-Out)
Amazon SQS offers a standard queue as the default queue type.
It allows you to have an unlimited number of transactions per second.
It guarantees that a message is delivered at least once. However, sometime, more than one copy of a message might be delivered out of order.
It provides best-effort ordering which ensures that messages are generally delivered in the same order as they are sent but it does not provide a guarantee.
2. FIFO Queue (First-In-First-Out):
The FIFO Queue complements the standard Queue.
It guarantees ordering, i.e., the order in which they are sent is also received in the same order.
The most important features of a queue are FIFO Queue and exactly-once processing, i.e., a message is delivered once and remains available until consumer processes and deletes it.
FIFO Queue does not allow duplicates to be introduced into the Queue.
It also supports message groups that allow multiple ordered message groups within a single Queue.
FIFO Queues are limited to 300 transactions per second but have all the capabilities of standard queues.
Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.
- Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component where a queue is a temporary repository for messages that are awaiting processing.
- With the help of SQS, you can send, store and receive messages between software components at any volume without losing messages.
- Using Amazon SQS, you can separate the components of an application so that they can run independently, easing message management between components.
- Any component of a distributed application can store the messages in the queue.
- Messages can contain up to 256 KB of text in any format such as json, xml, etc.
- Any component of an application can later retrieve the messages programmatically using the Amazon SQS API.
- The queue acts as a buffer between the component producing and saving data, and the component receives the data for processing. This means that the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer is only intermittently connected to the network.
- If you got two EC2 instances which are pulling the SQS Queue. You can configure the autoscaling group if a number of messages go over a certain limit. Suppose the number of messages exceeds 10, then you can add additional EC2 instance to process the job faster.
The benefits of using SQS:
Scalability Your SQS queues scale to the volume of messages you’re writing and reading. You don’t need to scale the queues; all the scaling and performance-at-scale aspects are taken care of by AWS.
Pay for what you use When using SQS, you only get charged for the messages you read and write (see the details in the Pricing section). There aren’t any recurring or base fees.
Ease of setup Since SQS is a managed service, so you don’t need to set up any infrastructure to start using SQS. You can simply use the API to read and write messages, or use the SQS <-> Lambda integration.
Options for Standard and FIFO queues When creating an SQS queue, you can choose between a standard queue and a FIFO queue out of the box. Both of these queue types can be useful for different purposes.
Automatic deduplication for FIFO queues Deduplication is important when using queues, and for FIFO queues SQS will do the work to remove any duplicate messages for you. This makes FIFO queues on SQS suitable for tasks where it’s critical to have each task done exactly once.
A separate queue for unprocessed messages This feature of SQS is useful for debugging. All messages that couldn’t be processed are sent into a “dead-letter” queue where you can inspect them. This queue has all the usual integrations enabled, so you can subscribe to it using an AWS Lambda event, for example, to send a notification when an item can’t be processed.
Disadvantages of using SQS:
High cost at scale With pay per use pricing, if the number of messages you send is very high, your SQS bill can be quite significant. Part of SQS pricing is data transfer charges, and those can add up if you send larger messages, or if you process messages from outside the main AWS region in which the queue is located. In some cases, when running at scale with millions of messages processed every day, the cost of using SQS might be higher than the cost of operating your own queue system, even including the overhead to manage your own solution.
Lack of support for broadcast messages With its “exactly once” delivery, SQS doesn’t support a way for multiple entities to retrieve the same message, making SQS not so useful for one-to-many broadcasts.
To address this, in the cases where one-to-many delivery is important, developers can use SQS alongside SNS.
Reduced control over performance When running a message queue system at scale, something you may well end up wanting to do is to fine-tune its performance to suit your needs. With SQS this isn’t an option: the service is fully managed, and you don’t get to look under the hood.
What is SQS used for?
Decoupling microservices. In a microservice architecture, messages represent one of the easiest ways to set up communication between different parts of the system. If your microservices run in AWS, and especially if those are Serverless services, SQS is a great choice for that aspect of the communication.
Sending tasks between different parts of your system. You don’t have to be running a microservices-oriented application to take advantage of SQS. You can also use it in any kind of application that needs to communicate tasks to other systems.
Distributing workloads from a central node to worker nodes. You can frequently find messaging systems in the flows of distributed large workloads like map-reduce operations. For these kinds of operations, it’s essential to be able to maintain a queue of all the tasks that need to be processed, efficiently distribute the tasks between the machines or functions doing the work, and guarantee that every part of the work is only done once.
Scheduling batch jobs. SQS is a great option for scheduling batch jobs for two reasons. First, it maintains a durable queue of all the scheduled jobs, which means you don’t need to keep track of the job status—you can rely on SQS to pass the jobs through and to handle any retries, should an execution fail and your batch system returns the message to the queue. Second, it integrates with AWS Lambda; if you’re using AWS Lambda to process the batch jobs, SQS automatically launches your Lambda functions once the data is available for them to process.
Amazon SQS limits:
Queue delay. Minimum 0 seconds, maximum 15 minutes. The built-in delay functionality in SQS queues exists to allow inserting a pause between when a message is produced and when it’s visible in the queue. If you need that delay to be longer than 15 minutes, SQS can’t offer a delay that long, so you’ll need to implement the delay in the system producing the messages before it pushes them to SQS.
In-flight messages. SQS has a maximum of 120,000 messages (standard queue) or 20,000 messages (FIFO queue) that can be in flight: in other words, messages that have been received by a consumer but not yet removed from the queue. In most well-architected systems, it would be hard to reach this number of in-flight messages unless you are in an outage scenario. Before using SQS in production, determine if this could become an issue for you, especially in edge cases and possible error states for your application.
Message attributes. Each message is allowed to have a maximum of 10 metadata attributes. If you’d like to have more metadata than that, consider including it in the message itself rather than as a field attached to the message.
Message content. Messages you submit to SQS queues can be in plain text (unformatted) or in JSON or XML format. Other formats aren’t supported.
Message retention. Minimum: 60 seconds. Maximum: 14 days. If you think you might need shorter or longer retention times, SQS might not work for you.
Maximum message size is 256KB. If you need to send larger messages, consider uploading the larger message to S3 and including a reference to the S3 object as part of the message.
Message visibility timeout. You might be wondering ‘What is SQS visibility timeout?’ The visibility timeout is a configurable time period during which SQS temporarily “hides” a message that has been received by a consumer to avoid its being processed by other consumers. After the visibility timeout expires, if the message hasn’t been removed from the queue by the consumer that received it, it becomes visible to other queue consumers. The default visibility timeout is 30 seconds, the minimum is 0 seconds and the maximum is 12 hours.
Amazon SQS alternatives
Amazon MQ is a managed queue service from AWS which is based on Apache ActiveMQ. Its architecture isn’t as flexible as SQS, and in order to use the Amazon MQ queues you need to use ActiveMQ libraries. On the other hand, being based on ActiveMQ means that Amazon MQ is compatible with any application that already uses ActiveMQ without any code changes.
Amazon Kinesis Data Streams Amazon Kinesis Data Streams is a managed AWS service that’s designed for high-volume near-real-time data intake. Kinesis Data Streams can be a good choice if you are producing large amounts of data that need to be processed in batches in near real time.
Apache Kafka Apache Kafka is an open-source stream processing system. Its main purpose, similar to AWS Kinesis, is to process large amounts of data in near real time. It has the same advantages and disadvantages as Kinesis Data Streams when compared with SQS: from one side, it is able to process very large number of messages; from the other side, it requires much more configuration and maintenance and is harder to use for producers and consumers compared to SQS.
RabbitMQ RabbitMQ is a popular open source message broker. It offers all of the same functionality as SQS and more: for example, it supports various authentication mechanisms for RabbitMQ queues, provides the option of using “lazy queues” that can handle a backlog of millions of messages, and supports queue protocols like JMS and AMQP.
Happy Learning !!