Message Queues

AMPS includes high performance queuing built on the AMPS messaging engine and transaction log. AMPS message queues combine elements of classic message queuing with the advanced messaging features of AMPS, including content filtering, aggregation and projection, historical replay, and so on. This chapter presents an overview of queues.

AMPS message queues help you easily solve some common messaging problems:

• Ensuring that a message is only processed once.

• Distributing tasks across workers in a fair manner.

• Ensuring that a message is delivered to and processed by a worker.

• Ensuring that when a worker fails to process a message, that message is redelivered.

• Setting limits for retries in processing a message, and taking an action when those limits are exceeded (such as moving the message to a dead-letter queue).

While it's possible to create applications with these properties by using the other features of AMPS, message queues provide these functions built into the AMPS server. In addition, message queues allow you to:

• Replicate messages between AMPS instances while preserving delivery guarantees.

• Create views and aggregates based on the current contents of a queue.

• Filter messages into and out of a queue.

• Provide a single published message to multiple queues.

• Aggregate multiple topics into a single queue.

Use message queues when it is important to ensure that a message is processed once, by a single consumer. When it is important to distribute messages to a number of consumers, use the AMPS publish/subscribe delivery model rather than a message queue.

The following diagram illustrates a simple usage of a queue to distribute work across three consumers:

Overview of queue distribution

This diagram shows a simple use of AMPS queues to distribute work. In the diagram, the transaction log is configured to record a topic named Work. AMPS is also configured with a queue named WorkToDo, which is based on the underlying topic Work. The publisher publishes three messages to the topic Work, and AMPS includes those messages in the WorkToDo queue. Each message is delivered to one of the three subscribers to the WorkToDo queue. Unlike pub/sub messaging, each subscriber only receives one message, and each message is delivered to only one subscriber.

Notice that, even though AMPS provides queue semantics over the WorkToDo topic, the messages are recorded in the transaction log once, in the Work topic. Other subscribers could subscribe to the Work topic to receive the full stream of messages, or do a bookmark replay over the Work topic to recreate the message flow or audit the messages published to that topic.

Also notice that, while the Work topic (the underlying topic) must be recorded in the transaction log, there is no need to define the Work topic in the SOW section of the configuration file. Queues use the transaction log to determine which messages to enqueue, and when a message is acknowledged, the acknowledgment is also recorded in the transaction log.