Details of High Availability

AMPS High Availability, which includes multi-site replication and the transaction log, is designed to provide long uptimes and speedy recovery from disasters. Replication allows deployments to improve upon the already rock-solid stability of AMPS. Additionally, AMPS journaling provides the persisted state necessary to make sure that client recovery is fast, painless, and error free.

Guaranteed Publishing

An interruption in service while publishing messages could be disastrous if the publisher doesn't know which message was last persisted to AMPS. To prevent this from happening, AMPS has support for guaranteed publishing.

With guaranteed publishing, the AMPS client library is responsible for retaining and retransmitting the message until the server acknowledges that the message has been successfully persisted to the server and has been acknowledged as persisted by any replication destinations that are configured for synchronous replication. This means that each message always has at least one part of the system (either the client library or the AMPS server) responsible for persisting the message, and if failover occurs, that part of the system can retain and recover the message as necessary.

An important part of guaranteed publishing is to be able to uniquely identify messages. In AMPS, the unique identifier for a message is a bookmark, which is formed from a combination of a number derived from the client name and a sequence number managed by the client. A sequence number is simply an ever-increasing number assigned by a publisher to any operation that changes the state of persistent storage in AMPS (that is, publish or sow_delete commands).

The AMPS clients automatically manage sequence numbers when applications use the named methods or the Command interface. The libraries set the sequence number on each published message, ensure that the sequence number increases as appropriate, and initialize the sequence number at logon using information retrieved from the server acknowledgment of the logon command. The sequence number is also used for acknowledgments. The persisted acknowledgment returned in response to a publish command contains the sequence number of the last message persisted rather than the CommandId of the publish command message (for more details see Acknowledgement Conflation and Publish Acknowledgements).

The logon command supports a processed acknowledgment message, which will return the Sequence of the last record that AMPS has persisted. When the processed acknowledgment message is returned to the publisher, the Sequence corresponds to the last message persisted by AMPS. The publisher can then use that sequence to determine if it needs to 1) re-publish messages that were not persisted by AMPS, or 2) continue publishing messages from where it left off. Acknowledging persisted messages across logon sessions allows AMPS to guarantee publishing. The HAClient classes in the AMPS clients manage sequence numbers, including setting a meaningful initial sequence number based on the response from the logon command, automatically.

Connections should request a processed acknowledgment message with every logon command. This ensures that the Sequence returned in the acknowledgment message matches the publisher's last published message. The 60East AMPS clients do this automatically when using the named logon methods. If you are building the command yourself or using a custom client, you may need to add this request to the command yourself.

In addition to the acknowledgment messages, AMPS also keeps track of the published messages from a client based on the client's name. The client name is set during the logon command, so to set a consistent client name, it is necessary for an application to log on to AMPS. A logon is required by default in AMPS versions 5.0 and later, and optional by default in AMPS versions previous to 5.0.

All publishers must set a unique client name field when logging on to AMPS. This allows AMPS to correlate the sequence numbers of incoming publish messages to a specific client, which is required for reliable publishing, replication, and duplicate detection in the server. In the event that multiple publishers have the same client name, AMPS can no longer reliably correlate messages using the publish sequence number and client name.

When a transaction log is enabled for AMPS, it is an error for two clients to connect to an instance with the same name.

Durable Publication and Subscriptions

The AMPS client libraries include features to enable durable subscription and durable publication. In this chapter we've covered how publishing messages to a transaction log persists them. We've also covered how the transaction log can be queried (subscribed) with a bookmark for replay. Now, putting these two features together yields durable subscriptions.

A durable subscriber is one that receives all messages published to a topic (including a regular expression topic), even when the subscriber is offline. In AMPS this is accomplished through the use of the bookmark subscription on a client.

Implementation of a durable subscription in AMPS is accomplished on the client by persisting the last observed bookmark field received from a subscription. This enables a client to recover and resubscribe from the exact point in the transaction log where it left off.

A durable publisher maintains a persistent record of messages published until AMPS acknowledges that the message has been persisted. Implementation of a durable publisher in AMPS is accomplished on the client by persisting outgoing messages until AMPS sends a persisted acknowledgment that says that this message, or a later message, has been persisted. At that point, the publishers can remove the message from the persistent store. Should the publisher restart, or should AMPS fail over, the publisher can re-send messages from the persistent store. AMPS uses the sequence number in the message to discard any duplicates. This helps ensure that no messages are lost, and provides fault-tolerance for publishers.

The AMPS C++, Java, C# and Python clients each provide different implementations of persistent subscriptions and persistent publication. Please refer to the High Availability chapter of the Developer Guide for the language of your choice to see how this feature is implemented.

Heartbeat in High Availability

Use of the heartbeat feature allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This default method may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber.

The heartbeat feature of the AMPS server and the AMPS clients allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS server assumes disconnection has occurred if these regular heartbeats cease, ensuring disconnection is detected in a timely manner.

Heartbeats are initialized by the AMPS client by sending a heartbeat message to the AMPS server. To enable heartbeats in your application, refer to the High Availability chapter in the Developer Guide for your specific client language.