High Availability

The AMPS JavaScript Client provides an easy way to create highly-available applications using AMPS, via the Client class. In other clients the highly available version of the client is called HAClient, however in JavaScript the Client class has all the functions of Client, providing both regular and highly available modes. Among its regular functions, Client provides protection against network, server, and client outages.

The Client class in the JavaScript library combines functionality of both Client and HAClient classes of other AMPS client libraries.

Using High Availability (HA) functions of the Client allows applications to automatically:

  • Recover from temporary disconnects between client and server.

  • Failover from one server to another when a server becomes unavailable.

Since the Client can automatically manage failover and reconnection, 60East recommends using the HA functions for applications that need to:

  • Ensure no messages are lost or duplicated after a reconnect or failover.

  • Persist messages and bookmarks in memory for protection against client failure.

You can choose how your application uses HA features. For example, you might need automatic reconnection, but have no need to resume subscriptions or republish messages. The high availability behavior in Client is provided by implementations of defined interfaces. You can combine different implementations provided by 60East to meet your needs, and implement those interfaces to provide your own policies.

Some of these features require specific configuration settings on your AMPS instance(s). This chapter mentions these features and describes how to use them from the AMPS JavaScript client. You can find full documentation for these settings and server features in the AMPS User Guide.

Reconnection with Client

This description provides a high-level framework for understanding the components involved in failover with the Client. The components are described in more detail in the following sections.

The Client reconnect handler performs the following steps when reconnecting:

  1. Calls the ServerChooser to determine the next URI to connect to and the Authenticator to use for that connection.

    If the connection fails, calls getError() on the ServerChooser to get a description of the failure, sends an error to the error handler, and stops the reconnection process.

  2. Calls the DelayStrategy to determine how long to wait before attempting to reconnect, and waits for that period of time.

  3. Connects to the AMPS server. If the connection fails, calls reportFailure() on the ServerChooser and begins the process again.

  4. Logs on to the AMPS server. If the connection fails, calls reportFailure() on the ServerChooser and begins the process again.

  5. Calls reportSuccess() on the ServerChooser.

  6. Receives the bookmark for the last message that the server has persisted. Discards any older messages from the PublishStore.

  7. Republishes any messages in the PublishStore that have not been persisted by the server.

  8. Re-establishes subscriptions using the SubscriptionManager for the client. For bookmark subscriptions, the reconnect handler uses the BookmarkStore for the client to determine the most recent bookmark, and resubscribes with that bookmark. For subscriptions that do not use a bookmark, the SubscriptionManager simply re-enters the subscription, meaning that it is entered at the point at which the Client reconnects.

The ServerChooser, DelayStrategy, PublishStore, SubscriptionManager, and BookmarkStore are all extension points for the Client. You can adapt the failover and recovery behavior by setting a different object for the behavior you want to customize on the Client or by providing your own implementation.

Choosing Store Durability

The Client provides recovery after disconnection using Stores. As the name implies, stores hold information about the state of the client.

These stores provide the following capabilities:

  • A bookmark store tracks received messages, and is used to resume subscriptions.

  • A publish store tracks published messages, and is used to ensure that messages are persisted in AMPS.

The AMPS JavaScript client provides a memory-backed version of each store. The store interface is public, and an application can create and provide a custom store as necessary. A Client can use a memory backed store for protection, as described below:

  • Memory-backed stores provide disconnection recovery from AMPS by storing messages and bookmarks in your process' address space. This is the highest performance option for working with AMPS in a highly available manner. The trade-off with this method is there is no protection from a crash or failure of your client application. If your application is terminated prematurely or, if the application terminates at the same time as an AMPS instance failure or network outage, then messages may be lost or duplicated.

    A memory-backed store should only be used by one instance of a client at a time.

The store interface is public, and an application can create and provide a custom store as necessary. While clients provide convenience methods for creating memory-backed Client objects with the appropriate stores, you can also create and set the stores in your application code.

The Client provides convenience methods for creating clients and setting stores. You can also construct a Client and set the store implementations you choose.

In this example, we create several clients. The first client has the full set of HA features, such as: supports automatic connection failover and reconnection, uses memory stores for both bookmarks and publishes, and automatically re-subscribes in case of a failover. The second client does not set a store for publishes, which means that AMPS will not store outgoing messages, but is using the bookmark store to track incoming messages. The final client does not specify any stores, and so has no persistence for published messages or bookmark subscriptions, but takes advantage of the automatic failover and reconnection in the Client.

// Failover, Bookmark and Publish stores, re-subscription
const memoryClient = Client.createMemoryBacked('memory-backed');

// No publish store, no failover, memory-backed bookmark store
const subscriberClient = new Client('subscriber');
subscriberClient.bookmarkStore(new MemoryBookmarkStore());

// Failover behavior only.
const failoverClient = new Client('failover');
const serverChooser = new DefaultServerChooser();
serverChooser.add('ws://localhost:9000/amps/json');
serverChooser.add('ws://localhost:9100/amps/json');
serverChooser.add('ws://localhost:9200/amps/json');

failoverClient.serverChooser(serverChooser);
failoverClient.delayStrategy(new FixedDelayStrategy());

Connections and the Server Chooser

In the high availability mode, Client attempts to keep itself connected to an AMPS instance at all times, by automatically reconnecting or failing over when it detects that the client is disconnected. When you are using the Client directly, your disconnect handler usually takes care of reconnection. Client with HA classes on the other hand, can provide a disconnect handler that automatically reconnects to the current server or to the next available server.

To inform the Client of the addresses of the AMPS instances in your system, you pass a ServerChooser instance to the Client. ServerChooser acts as a smart enumerator over the servers available: Client calls ServerChooser methods to inquire about what server should be connected, and calls methods to indicate whether a given server succeeded or failed.

The AMPS JavaScript Client provides a simple implementation of ServerChooser, called DefaultServerChooser, that provides very simple logic for reconnecting. This server chooser is most suitable for basic testing, or in cases where an application should simply rotate through a list of servers. For most applications, you implement the ServerChooser interface yourself for more advanced logic, such as choosing a backup server based on your network topology, or limiting the number of times your application should try to reconnect to a given address.

To connect to AMPS, you provide a ServerChooser to Client and then call Client.connect() to create the first connection:

const memoryClient = new Client('server-chooser-demo');

// primary.amps.xyz.com is the primary AMPS instance, and
// secondary.amps.xyz.com is the secondary
const chooser = new DefaultServerChooser();

chooser.add('ws://primary.amps.xyz.com:12345/amps/fix');
chooser.add('ws://secondary.amps.xyz.com:12345/amps/fix');

memoryClient.serverChooser(chooser);

try {
  await memoryClient.connect();

  // ... later
  await client.disconnect();
}
catch (err) {
  // handle connection error
}

Client remains connected to the server until disconnect() is called. Client automatically attempts to reconnect to your server if it detects a disconnect and, if that server cannot be connected, fails over to the next server provided by the ServerChooser. In this example, the call to connect() attempts to connect and login to primary.amps.xyz.com, and resolves if that is successful. If it cannot connect, it tries secondary.amps.xyz.com, and continues trying servers from the ServerChooser until a connection is established. Likewise, if it detects a disconnection while the client is in use, then Client attempts to reconnect to the server with which it was most recently connected; if that is not possible, then it moves on to the next server provided by the ServerChooser.

The default ServerChooser simply provides the next URL in the sequence. This strategy works for many applications. If you need a different strategy, you can implement your own logic for failover by creating a class derived from ServerChooser.

Setting a Reconnect Delay and Timeout

You can control the amount of time between reconnection attempts and set a total amount of time for the Client to attempt to reconnect.

The AMPS JavaScript client includes a method for setting a delay strategy on a client, Client.delayStrategy(). This method accepts an instance of any type that provides the methods getConnectWaitDuration() and reset(), as described in the API documentation.

While you can easily implement your own delay strategy, the client also provides two delay strategies:

  • FixedDelayStrategy provides the same delay each time the Client tries to reconnect.

  • ExponentialDelayStrategy provides an exponential backoff until a connection attempt succeeds.

To use either of these classes, you simply create an instance, set the appropriate parameters, and install that instance as the delay strategy for the Client. For example, the following code sets up a reconnect delay that starts at 200ms and increases the delay by 1.5 times after each failure. The strategy allows a maximum delay between connection attempts of 5 seconds, and will not retry longer than 60 seconds.

const client = new Client('delay-strategy-demo');

client.delayStrategy(
  new ExponentialDelayStrategy({
    initialDelay: 200,
    maximumDelay: 5 * 1000,
    backoffExponent: 1.5,
    maximumRetryTime: 60 * 1000
  })
);

Implementing a Server Chooser

As described above, you provide the Client with connection strings to one or more AMPS servers using a ServerChooser. The purpose of a ServerChooser is to provide information to the Client. A ServerChooser does not manage the reconnection process, and should not call methods on the Client.

A ServerChooser has two required responsibilities to the Client:

  • Tells the Client the connection string for the server to connect to. If there are no servers, or the ServerChooser wants the connection to fail, the ServerChooser returns null.

    To provide this information, the ServerChooser implements the getCurrentUri() method.

  • Provides an Authenticator for the current connection string. This is especially important for installations where different servers require different credentials or authentication tokens must be reset after each connection attempt.

    To provide the authenticator, the ServerChooser implements the getCurrentAuthenticator() method.

The Client calls the getCurrentUri() and getCurrentAuthenticator() methods each time it needs to make a connection.

Each time a connection succeeds, the Client calls the reportSuccess() method of the ServerChooser. Each time a connection fails, the Client calls the reportFailure() method of the ServerChooser. The Client does not require the ServerChooser to take any particular action when it calls these methods. These methods are provided for the Client to do internal maintenance, logging, or record keeping. For example, a Client might keep a list of available URIs with a current failure count, and skip over URIs that have failed more than 5 consecutive times until all URIs in the list have failed more than 5 consecutive times.

When the ServerChooser returns a null from getCurrentUri(), indicating that no servers are available for connection, the Client calls the getError() method on the ServerChooser, if one is provided, and includes the string returned by getError() in the generated exception.

Heartbeats and Failure Detection

Use of the Client allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This system may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber.

The heartbeat feature of the AMPS client allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS client and server both assume disconnection has occurred if these regular heartbeats cease, ensuring disconnection is detected in a timely manner. To use the heartbeat feature, call the heartbeat() method on Client:

const memoryClient = new Client('importantStuff');

// ...

memoryClient.heartbeat(3);
await memoryClient.connect();

Method heartbeat() takes one parameter: the heartbeat interval. The heartbeat interval specifies the periodicity of heartbeat messages sent by the server: the value 3 indicates messages are sent on a three-second interval. If the client receives no messages in a six-second window (two heartbeat intervals), the connection is assumed to be dead, and the Client attempts reconnection. The optional second parameter of the heartbeat() method allows the idle period to be set to a value other than two heartbeat intervals.

Heartbeats are handled asynchronously by the AMPS client. Your application must not flood the execution queue for longer than the heartbeat interval, or the application is subject to being disconnected.

Considerations for Publishers

Publishing with the Client in the HA mode is nearly identical to regular publishing; you simply call the publish() method with your message’s topic and data. The AMPS client sends the message to AMPS, and then returns from the publish() call. For maximum performance, the client does not wait for the AMPS server to acknowledge that the message has been received.

When a Client sets a publish store, the publish store retains a copy of each outgoing message and requests that AMPS acknowledge that the message has been persisted. The AMPS server acknowledges messages back to the publisher. Acknowledgments can be delivered for multiple messages at periodic intervals (for topics recorded in the transaction log) or after each message (for topics that are not recorded in the transaction log). When an acknowledgment for a message is received, the Client removes that message from the publish store. When a connection to a server is made, the Client automatically determines which messages from the publish store (if any) the server has not processed, and replays those messages to the server once the connection is established.

For reliable publishers, the application must choose how best to handle application shutdown. For example, it is possible for the network to fail immediately after the publisher sends the message, while the message is still in transit. In this case, the publisher has sent the message, but the server has not processed it and acknowledged it. During normal operation, the Client will automatically connect and retry the message. On shutdown, however, the application must decide whether to wait for messages to be acknowledged, or whether to exit.

Publish store implementations provide an unpersistedCount() method that reports the number of messages that have not yet been acknowledged by the AMPS server. When the unpersistedCount() reaches 0, there are no unpersisted messages in the local publish store.

For the highest level of safety, an application can wait until the unpersistedCount() reaches 0, which indicates that all of the messages have been persisted to the instance that the application is connected to, and the synchronous replication destinations configured for that instance. When a synchronous replication destination goes offline, this approach will cause the publisher to wait to exit until the destination comes back online or until the destination is downgraded to asynchronous replication.

For applications that are shut down periodically for short periods of time (for example, applications that are only offline during a weekly maintenance window), another approach is to use the Client.flush() method to ensure that messages are delivered to AMPS, and then rely on the connection logic to replay messages as necessary when the application restarts.

For example, the following code flushes messages to AMPS, then warns if not all messages have been acknowledged:

const client = Client.createMemoryBacked('ha-publisher');
// ...
await client.connect();

// Publish messages
client.publish('topic', {id: 1});
client.publish('topic', {id: 2});
// ...

/**
* We think we are done, but the server may not
* have received or acknowledged all messages yet.
* Wait until the server has received all messages.
* The program could also specify a timeout in this
* command to avoid blocking forever if the network
* is down or all servers are offline.
*/
await client.flush();

/**
* Print warning to the console if messages have
* been published but not yet acknowledged as persisted
*/
if (client.publishStore().unpersistedCount() > 0) {
  console.log('All messages have been published');
  console.log('But not all have been persisted');
}

client.disconnect();

In this example, the client sends each message immediately when publish() is called. If AMPS becomes unavailable between the final publish() and the disconnect(), or one of the servers that the AMPS instance replicates to is offline, the client may not have received a persisted acknowledgment for all of the published messages. For example, if a message has not yet been persisted by all of the servers in the replication fabric that are connected with synchronous replication, AMPS will not have acknowledged the message.

Before shutting down the client, the code does two things:

  • First, the code flushes messages to the server to ensure that all messages have been delivered to AMPS.

  • Next, the code checks to see if all of the messages in the publish store have been acknowledged as persisted by AMPS. If the messages have not been acknowledged, they will remain in the publish store and will be published to AMPS, if necessary, the next time the client connects. An application may choose to wait until unpersistedCount() returns 0, or (as we do in this case) simply warn that AMPS has not confirmed that the messages are fully persisted. The behavior you choose in your application should be consistent with the high-availability guarantees your application needs to provide.

AMPS uses the name of the Client to determine the origin of messages. For the AMPS server to correctly identify duplicate messages, each instance of an application that publishes messages must use a distinct name. That name must be consistent across different runs of the application.

AMPS provides persisted acknowledgment messages for topics that do not have a transaction log enabled. However, the level of durability provided for topics with no transaction log is minimal. Learn more about transaction logs in the AMPS User Guide.

Considerations for Subscribers

Client provides two important features for applications that subscribe to one or more topics: re-subscription, and a bookmark store to track the correct point at which to resume a bookmark subscription.

Resubscription With SubscriptionManager

Any asynchronous subscription placed using a Client is automatically reinstated after a disconnect or a failover. These subscriptions are placed in an in-memory SubscriptionManager, which is created automatically when the Client.createMemoryBacked() static method is called. Alternatively, it can be created and assigned before the client is connected:

const client = new Client('subscription-manager-demo');
client.subscriptionManager(new DefaultSubscriptionManager());

Most applications will use this built-in subscription manager, but for applications that create a varying number of subscriptions, you may wish to implement SubscriptionManager to store subscriptions in a more durable place. Note that these subscriptions contain no message data, but rather simply contain the parameters of the subscription itself (for instance, the command, topic, message handler, options, and filter).

When a re-subscription occurs, the AMPS JavaScript Client re-executes the command as originally submitted, including the original topic, options, and so on. AMPS sends the subscriber any messages for the specified topic (or topic expression) that are published after the subscription is placed. For a sow_and_subscribe command, this means that the client reissues the full command, including the SOW query as well as the subscription.

Bookmark Stores

In cases where it is critical not to miss a single message, it is important to be able to resume a subscription at the exact point that a failure occurred. In this case, simply recreating a subscription isn't sufficient. Even though the subscription is recreated, the subscriber may have been disconnected at precisely the wrong time, and will not see the message.

To ensure delivery of every message from a topic or set of topics, the AMPS Client can plug in a BookmarkStore that, combined with the bookmark subscription and transaction log functionality in the AMPS server, ensures that clients receive any messages that might have been missed. The client stores the bookmark associated with each message received, and tracks whether the application has processed that message; if a disconnect occurs, the client uses the BookmarkStore to determine the correct resubscription point, and sends that bookmark to AMPS when it re-subscribes. AMPS then replays messages from its transaction log from the point after the specified bookmark, thus ensuring the client is completely up-to-date.

Client helps you to take advantage of this bookmark mechanism through the Client.bookmarkStore() method and MemoryBookmarkStore class. When a bookmark store is assigned to the client, whenever a disconnection or failover occurs, your application automatically re-subscribes to the message after the last message it processed.

To take advantage of bookmark subscriptions, do the following:

  • Ensure the topic(s) to be subscribed to are included in a transaction log. See the AMPS User Guide for information on how to specify the contents of a transaction log.

  • Before connecting, create and assign a bookmark store object to the client.

  • Use the Client.bookmarkStore().discard() method in message handlers to indicate when a message has been fully processed by the application.

The following example creates a bookmark subscription against a transaction-logged topic, and fully processes each message as soon as it is delivered:

const client = new Client('aClient');
client.subscriptionManager(new DefaultSubscriptionManager());
client.bookmarkStore(new MemoryBookmarkStore());

// ...

await client.execute(
 // The subscription command
 new Command('subscribe')
     .topic('myTopic')
     .bookmark(Client.Bookmarks.MOST_RECENT)
     .subId('MySubId'),

 // Message handler discards every message after processing
 message => {
   console.log(message.data);
   client.bookmarkStore().discard(message);
 }
);

Storing these bookmarks in the bookmark store allows the application to restart the subscription from the last message processed, in the event of either server failure or disconnect.

For optimum performance, it is critical to discard every message received from a bookmark replay once its processing is complete. If a message is never discarded, it remains in the bookmark store. During re-subscription, Client always restarts the bookmark subscription with the oldest undiscarded message, and then filters out any more recent messages that have been discarded. If an old message remains in the store, but is no longer important for the application’s functioning, then the client and the AMPS server will incur unnecessary network and CPU activity.

The command method, subId(), specifies an identifier to be used for this subscription. If the subId is not provided, Client will generate one and resolve the Client.execute() Promise with it, like most other Client functions. If you wish to resume a subscription from a previous point after the application has disconnected, the application must pass the same subscription ID as before. Passing a different subscription ID bypasses any recovery mechanisms, creating an entirely new subscription. When you use an existing subscription ID, the Client locates the last-used bookmark for that subscription in the bookmark store, and attempts to re-subscribe from that point.

  • Client.Bookmarks.NOW specifies that the subscription should begin from the moment the server receives the subscription request. This results in the same messages being delivered as if you had invoked subscribe() instead, except that the messages will be accompanied by bookmarks. This is also the behavior that results if you supply an invalid bookmark.

  • Client.Bookmarks.EPOCH specifies that the subscription should begin from the beginning of the AMPS transaction log (that is, the first entry in the oldest journal file for the transaction log).

  • Client.Bookmarks.MOST_RECENT specifies that the subscription should begin from the last-used message in the associated BookmarkStore. Alternatively, if this subscription has not been seen before, it instructs the subscription to begin with EPOCH. This is the most common value for this parameter, and is the value used in the preceding example. By using MOST_RECENT, the application automatically resumes from wherever the subscription left off, taking into account any messages that have already been processed and discarded.

When the Client re-subscribes after a disconnection and reconnection, it always uses MOST_RECENT, ensuring that the continued subscription always begins from the last message used before the disconnect, so that no messages are missed.

Conclusion

With only a few changes, most AMPS applications can take advantage of the high availability features of the Client to become more highly-available and resilient. Using the PublishStore, publishers can ensure that every message published has actually been persisted by AMPS. Using BookmarkStore, subscribers can make sure that there are no gaps or duplicates in the messages received. Client makes both kinds of applications more resilient to network and server outages and temporary issues. Though Client provides useful defaults for the PublishStore, BookmarkStore, SubscriptionManager, ServerChooser, and DelayStrategy, you can customize any or all of these to the specific needs of your application and architecture.

PreviousDelta PublishNextAMPS Programming: Working With Commands

Last updated