Troubleshooting Disconnected Clients

One common symptom of problems in an AMPS application is that AMPS disconnects clients unexpectedly. AMPS disconnects clients in the following situations:

• When transaction logging is configured for the instance and a client with a duplicate name logs on

• When heartbeating is enabled and the client misses a heartbeat

• When a slow client falls behind by more than the configured threshold

• When the entitlement cache for an instance is reset

• When the administration console disconnects a client

• When the transport is disabled

This section presents techniques to help you identify why clients are disconnected and correct any problems that may exist.

Locating the Reason for Disconnection

To discover the reason that a client was disconnected, use the following command to find the client name in the logs:

amps-grep client_name log_file

The results of this can provide information as to why the client was disconnected. AMPS logs a reason for the disconnection if the disconnection was the result of an internal action by AMPS.

If the client chose to disconnect, or the disconnection is the result of network instability, the disconnection is logged but no further information is available to AMPS.

The reasons that AMPS will log for a client disconnection are shown below:

Reason	Explanation
connection closed	AMPS has detected that the socket has been closed.
entitlement reset	Entitlements have been reset for this user (or the full transport).
name in use	A client session (as identified by the client name) has reconnected, and the redundant connection has been closed.
heartbeat	The connection failed to provide a heartbeat within the period of time set by the connection, so AMPS has closed the connection.
slow client	The connection was closed due to the client capacity settings being exceeded.
unknown command	The client sent an unknown or malformed command, so AMPS closed the connection.
auth	The connection failed to authenticate.
entitlement	The connection successfully authenticated, but the user does not have permission to log in.

These reasons are included in the 07-0013 log message that records that the connection has closed.

Duplicate Client Name Disconnection

When a client is disconnected due to another client with the same name logging on, the messages produced might look like:

2019-11-20T16:26:59.6408410-08:00 [5] warning: 02-0025 A client logon with an 'in use' client name for the same user id forced a disconnect of client: client[my-name] with user id:

To resolve this issue, ensure that clients use unique names when connecting to instances that configure a transaction log.

Missed Heartbeat Disconnection

When AMPS disconnects a client due to the client failing to heartbeat, the log messages produced look like the following:

2019-11-20T16:35:23.9185690-08:00 [6] error: 07-0042 AMPS heartbeat manager is disconnecting an unresponsive client: no-heartbeat-client

This error most often arises from severe network congestion, a deadlock or similar problem in the application that is preventing the AMPS client library from producing heartbeats, or a problem in AMPS that prevents AMPS from servicing heartbeat requests.

Slow Client Disconnection

The following shows sample log entries for slow client disconnection. If a client named sleepy-client was disconnected for being a slow client, the relevant entries in the transaction log might look like:

2019-11-20T15:33:06.8496430-08:00 [7] warning: 70-0011 client[sleepy-client] slow consumption detected, offline messages.
2019-11-20T15:33:06.8498130-08:00 [7] error: 70-0004 client[sleepy-client] is not consuming messages, disconnecting slow client

Notice that there may be a considerable period of time between the client being offlined and the client being disconnected.

There are several approaches to solving the problem:

• Reduce the number of messages returned. Clients most often fall behind when a SOW query or a replay from the transaction log returns a large number of messages. If possible, use content filtering to return a more precise set of messages.

• Improve the rate at which the client handles messages. If the client message handler takes a relatively long time to process the message, moving message processing onto a different thread or streamlining the processing may improve the speed of the client and allow the client to keep up.

• Adjust the client offlining threshold. You can also increase the capacity of messages that AMPS will buffer for clients connected to a specific transport, as described in Slow Client Management.

Admin Console Client Disconnection

Disconnection from the admin console provides no additional information, and produces a log message like the following:

2019-11-20T15:33:06.8502350-08:00 [4] info: 07-0013 client[sleepy-client] disconnected.

Admin Console Transport Disabled

A transport being disabled through the admin console produces messages like the following:

2019-11-20T16:04:00.9548130-08:00 [10] info: 07-0047 Transport[json-tcp] being disabled.
2019-11-20T16:04:00.9550150-08:00 [4] info: 07-0013 client[amps-json-tcp-18] disconnected.