SQS

Standard Queue

• Oldest offering (over 10 years old)
• fully managed service, used to decouple applications
• Attributes
  • unlimited throughput, unlimited number of messages in queue
  • default retention of messages: 4 days, 1 minutes ~ 14 days
  • Low Latency (< 10ms on publish and receive)
  • Limitation of 256KB per message sent
• can have duplicate messages (at least once delivery)
• can have out of order messages (best effort ordering)
• message is persisted in SQS until a consumer deletes it
• work with ASG (cloud-watch-metric queue length ApproximateNumberOfMessages)
• security
  • encryption
    • HTTPS (in-flight)
    • KMS (at-rest)
  • access controls - IAM Policies
  • SQS Access Policies
    • cross account access
    • allow other services to write
• Message Visibility Timeout
  • After a message is polled by a consumer, it becomes invisible to other consumers
  • by default, 30s
  • If a message is not processed within the visibility timeout, it might be processed twice
  • A consumer could call the ChangeMessageVisibility API to get more time
• Long Polling
  • When a consumer requests messages from the queue, it can optionally “wait” for messages to arrive if there are none in the queue
  • LongPolling decreases the number of API calls made to SQS while increasing the efficiency and reducing latency of your application
  • The wait time can be between 1 sec to 20 sec
  • Long polling can be enabled at the queue level or at the API level using WaitTimeSeconds
• FIFO Queue
  • Limited throughput: 300 msg/s, batching 3000 msg/s
  • Exactly-once send capability (by removing duplicates)
  • Messages are processed in order by the consumer
• SQS with ASG
  • CloudWatch Metric - Queue Length
  • CloudWatch Alarm
  • Auto Scaling Group

SNS

• The “event producer” only sends message to one SNS topic
• As many “event receivers” (subscriptions) as we want to listen to the SNS topic notifications
• Each subscriber to the topic will get all the messages (supports message filtering)
• Up to 12,500,000 subscriptions per topic
• 100,000 topics limit
• Security
  • Encryption
  • IAM Policies
  • SNS Access Policies
• Combine with SQS to achieve fan-out & persistent
  • access policy
  • cross-region delivery: works with SQS Queues in other Regions
• FIFO Topic
  • Ordering by Message group ID
  • Deduplication using a Deduplication ID or content based deduplication
  • can have SQS Standard and FIFO queues as subscribers
  • limited throughput (same as SQS FIFO)
• Message Filtering (JSON)

Kinesis

• Makes it easy to collect, process, and analyze streaming data in real-time
• Ingest real-time data such as: Application logs, Metrics, Website clickstreams, IoT telemetry data…
• Kinesis Data Streams: capture, process, and store data streams
• Kinesis Data Firehose: load data streams into AWS data stores
• Kinesis Data Analytics: analyze streaming-data with SQL or Apache Flink
• Kinesis Video Streams: capture, process, and store video streams

Data Streams

• Retention between 1 day to 365 days
• Ability to reprocess (replay) data
• Once data is inserted in Kinesis, it can’t be deleted (immutability)
• Data that shares the same partition goes to the same shard (ordering)
• Producers: AWS SDK, Kinesis Producer Library (KPL), Kinesis Agent
• Consumers:
  • Write your own: Kinesis Client Library (KCL), AWS SDK
  • Managed: AWS Lambda, Kinesis Data Firehose, Kinesis Data Analytics,
• Capacity Modes
  • Provisioned mode
    • each shard gets 1MB/s in (or 1000 records per seconds)
    • each shard gets 2MB/s out
  • On-demand mode
    • default capacity provisioned (4MB/s or 4k records per second)
    • scales automatically based on observed throughput peak during the last 30 days
• Security
  • IAM policies
  • encryption in-flight https endpoint, at rest using KMS
  • VPC Endpoints available
  • Monitor API calls using CloudTrail

Data Firehose

• load streaming data to data-stores
• Pay for data going through Firehose
• Near Real Time
  • 60 seconds latency minimum for non full batches
  • Or minimum 1 MB of data at a time
• Supports many data formats, conversions, transformations, compression
• Supports custom data transformations using AWS Lambda
• Can send failed or all data to a backup S3 bucket

Comparison

Kinesis vs SQS ordering

Let’s assume 100 trucks, 5 kinesis shards, 1 SQS FIFO

• Kinesis Data Streams:
  • On average you’ll have 20 trucks per shard
  • Trucks will have their data ordered within each shard
  • The maximum amount of consumers in parallel we can have is 5
  • Can receive up to 5 MB/s of data
• SQS FIFO
  • You only have one SQS FIFO queue
  • You will have 100 Group ID
  • You can have up to 100 Consumers (due to the 100 Group ID)
  • You have up to 300 messages per second (or 3000 if using batching)

AmazonMQ

• SQS, SNS are “cloud-native” services: proprietary protocols from AWS
• Traditional applications running from on-premises may use open protocols such as: MQTT, AMQP, STOMP, OpenWire, WSS
• When migrating to the cloud, instead of re-engineering the application to use SQS and SNS, we can use Amazon MQ
• Amazon MQ is a managed message broker service for RabbitMQ, ActiveMQ
• Amazon MQ doesn’t “scale” as much as SQS / SNS
• Amazon MQ runs on servers, can run in Multi-AZ with failover
• Amazon MQ has both queue feature (~SQS) and topic features (~SNS)

References

• https://aws.amazon.com/kinesis/data-streams/faqs/