The HAClient class, included with the AMPS C++ client, contains a disconnect handler and other features for building highly-available applications. The HAClient includes features for managing a list of failover servers, resuming subscriptions, republishing in-flight messages, and other functionality that is commonly needed for high availability. 60East recommends using the HAClient for automatic reconnection wherever possible, as the HAClient disconnect handler has been carefully crafted to handle a wide variety of edge cases and potential failures.

If an application needs to reconnect or fail over, use an HAClient, and the AMPS client library will automatically handle failover and reconnection. You control which servers the client fails over to using an implementation of the ServerChooser interface, and control the timing of the failover using an implementation of the ReconnectDelayStrategy interface.

If you need to add custom behavior to the failover (such as logging, resetting an internal cache, refreshing credentials and so on), the ConnectionStateListener class allows your application to be notified and take action when disconnection is detected and at each stage of the reconnection process.

To extend the behavior of the AMPS client during reconnection, implement a ConnectionStateListener.

In some cases, an application does not want the AMPS client to reconnect, but instead wants to take a different action if disconnection occurs. For example, a stateless publisher that sends ephemeral data (such as telemetry or prices) may want to exit with an error if the connection is lost rather than risk falling behind and providing outdated messages. Often, in this case, a monitoring process will start another publisher if a publisher fails, and it is better for a message to be lost than to arrive late.

To cover cases where the application has unusual needs, the AMPS client library allows an application to provide custom disconnect handling.

Your application gets to specify exactly what happens when a disconnect occurs by supplying a function to client.setDisconnectHandler(), which is invoked whenever a disconnect occurs. This may be helpful for situations where a particular connection needs to do something completely different than reconnecting or failing over to another AMPS server.

The handler runs on the thread that detects the disconnect. This may be the client receive thread (for example, if the disconnect is detected due to heartbeating) or an application thread (for example, if the disconnect is detected when sending a command to AMPS).

The example below` shows a disconnect handler that will exit the application when a disconnect is detected:

class MyApp
{
    string _uri;
    Client _client;

public:
    MyApp(const string& uri) : _uri(uri), _client("myapp")
    {
        _uri = uri;

        /* setDisconnectHandler() method is called to supply a function for use when AMPS
         * detects a disconnect. At any time, this function may be called by AMPS to
         * indicate that the client has disconnected from the server, and to allow your
         * application to choose what to do about it.
         *
         * An application that intends to reconnect would use an HAClient and
         * the provided disconnect handler. In this case, however, we want
         * the application to exit if the connection is ever lost.
         *
         * There are various options for providing the function: in this case,
         * because we want to call a member function on an instance of this
         * class, we use std::bind.
         */

        auto handle = bind(&MyApp::FailOnDisconnect, this, placeholders::_1);
        _client.setDisconnectHandler(AMPS::DisconnectHandler(handle));
        _client.connect(uri);
        _client.logon();
        _client.executeAsync(Command("subscribe")
                                .setTopic("orders"),
                              bind(&MyApp::ShowMessage,
                                    this,
                                    placeholders::_1));
    }
    void ShowMessage(const Message& m)
    {
        /* display order data to the user */
        ...
    }
    /* Our disconnect handler’s implementation begins here.
     *
     * If we wanted the application to reconnect, resubmit the subscription, and so
     * on, we would use the HAClient (and the provided disconnect handler).
     *
     * In this case, we want to exit with an error if the connection ever fails,
     * so we replace the disconnect handler with a function that does
     * exactly that.
     */
    void FailOnDisconnect(Client& client)
    {
        /* simple: exit if the client is ever disconnected */

        ::exit(EXIT_FAILURE);
    }
};

The AMPS client includes a heartbeat feature to help applications detect disconnection from the server within a predictable amount of time. Without using a heartbeat, an application must rely on the operating system to notify the application when a disconnect occurs. For applications that are simply receiving messages, it can be impossible to tell whether a socket is disconnected or whether there are simply no incoming messages for the client.

When you set a heartbeat, the AMPS client sends a heartbeat message to the AMPS server at a regular interval, and waits a specified amount of time for the response. If the operating system reports an error on send, or if there is no activity received from the server within the specified amount of time, the AMPS client considers the server to be disconnected. Likewise, the server will ensure that traffic is sent to the client at the specified interval, using heartbeat messages when no other traffic is being sent to the client. If, after sending a heartbeat message, no traffic from the client arrives within a period twice the specified interval, the server will consider the client to be disconnected or nonresponsive.

The AMPS client processes heartbeat messages on the client receive thread, which is the thread used for asynchronous message processing. If your application uses asynchronous message processing and occupies the thread for longer than the heartbeat interval, the client may fail to respond to heartbeat messages in a timely manner and may be disconnected by the server.

Every distributed system will experience occasional disconnections between one or more nodes. The reliability of the overall system depends on an application’s ability to efficiently detect and recover from these disconnections. Using the AMPS C/C++ client’s disconnect handling, you can build powerful applications that are resilient in the face of connection failures and spurious disconnects.