Amazon SQS

WIP

AWS SQS Basic

Queues

Below sourceQueue is the queue message are published to by the publisher and deadQueue is the dead letter queue (DLQ) use by the sourceQueue.

  • Types are Standard and FIFO, sourceQueue and deadQueue need to have matching types.
  • sourceQueue and deadQueue must be on the same region of the AWS Account
  • Message Retention, the max value is 14 days for any queue and the default is 4 days.

Some testing code to understand the magic.

Source Queue

  • Visibility timeout, once a message is received by a consumer it is not visible to other consumers for the period of this timeout. Useful when your consumer application scales in K8s
  • RedrivePolicy, specifies the source queue, the dead-letter queue and the conditions under which Amazon SQS moves messages from sourceQeueu to deadQueue if the consumer of the sourceQueue fails to process a message. This will contain the ARN of the deadQueue and the value of the maxReceiveCount
  • maxReceiveCount, used by the redrive policy to know how many times the message can be received before being auto-magically pushed to the deadQueue
  • Heartbeat, used when you dont know how long it will take the consumer to process the message once receieved. Specify the initial visibility timeout (for example, 2 minutes) and then—as long as your consumer still works on the message—keep extending the visibility timeout by 2 minutes every minute.
  • Delete, once processed the consumer must manually delete the message, else it will increment maxReceiveCount

Dead Letter Queue

  • Retention Period, the expiration of a message is always based on its orginal enqueue timestamp. So it starts counting from the sourceQueue. The period in the deadQueue must always be larger than the sourceQueue.
  • Helps with message failures to isolate the failed message
  • Replay Policy, this is manual, AWS suggest using a Lambda. A message could fail for a number of reasons including downstream dependancy borked or serialization failure.

References