Slow Client Management and Capacity Limits
AMPS provides the ability to manage memory consumption for clients to prevent slow clients, or clients that require large amounts of state, to disrupt service to the instance.
Sometimes, AMPS can publish messages faster than an individual client can consume messages, particularly in applications where the pattern of messages includes "bursts" of messages. Clients that are unable to consume messages faster or equal to the rate messages are being sent to them are "slow clients". By default, AMPS queues messages for a slow client in memory to grant the slow client the opportunity to catch up. However, scenarios may arise where a client can be over-subscribed to the point that the client cannot consume messages as fast as messages are being sent to it. In particular, this can happen with the results of a large SOW query, where AMPS generates all of the messages for the query much faster than the network can transmit the messages.
Some features, such as conflated subscriptions, aggregated subscriptions and pagination require AMPS to buffer messages in memory for extended periods of time. Without a way to set limits on memory consumption, subscribers using these features could cause AMPS to exceed available memory and reduce performance or exit.
Memory capacity limits, typically called slow client management, are one of the ways that AMPS prevents slow clients, or clients that consume large amounts of memory, from disrupting service to other clients connected to the instance. 60East recommends enabling slow client management for instances that serve high message volume or are mission critical.
There are two methods that AMPS uses for managing slow clients to minimize the effect of slow clients on the AMPS instance:
Client Offlining - When client offlining occurs, AMPS buffers the messages for that client to disk. This relieves pressure on memory, while allowing the client to continue processing messages.
Disconnection - When disconnection occurs, AMPS closes the client connection, which immediately ends any subscriptions, in-progress
sowqueries, or other commands from that client. AMPS also removes any offlined messages for that client.
AMPS provides resource pool protection, to protect the capacity of the instance as a whole, and client-level protection, to identify unresponsive clients.
Resource Pool Policies
AMPS uses resource pools for memory and disk consumption for clients. When the memory limit is exceeded, AMPS chooses a client to be offlined. When the disk limit is exceeded, AMPS chooses a client to be disconnected.
When choosing which client will be offlined or disconnected, AMPS identifies the client that uses the largest amount of resources (memory and/or disk). That client will be offlined or disconnected. The memory consumption calculated for a client includes both buffered messages and memory used to support features such as conflated subscriptions and aggregated subscriptions.
AMPS allows you to use a global resource pool for the entire instance, a resource pool for each transport, or any combination of the two approaches. By default, AMPS configures a global resource pool that is shared across all transports. When an individual transport specifies a different setting for a resource pool, that transport receives an individual resource pool. For example, you might set high resource limits for a particular transport that serves a mission-critical application, allowing connections from that application to consume more resources than connections for less important applications.
The following table shows resource pool options for slow client management:
MessageMemoryLimit
The total amount of memory to allocate to messages before offlining clients.
Default: 10% of total host memory or 10% of the amount of host memory AMPS is allowed to consume (as reported by ulimit -m ), whichever is lowest.
MessageDiskLimit
The total amount of disk space to allocate to messages before disconnecting clients.
Default: 1GB or the amount specified in the MessageMemoryLimit, whichever is highest.
MessageDiskPath
The path to use to write offline files.
Default: /var/tmp
Individual Client Policies
AMPS also allows you to set policies that apply to individual clients. These policies are applied to clients independently of the instance level policies. For example, a client that exceeds the capacity limit for an individual client will be disconnected, even if the instance overall has enough capacity to hold messages for the client.
As with the Resource Pool Policies, Transports can either use instance-level settings or create settings specific to that transport.
The following table shows the client level options for slow client management:
ClientMessageAgeLimit
The maximum amount of time for the client to lag behind. If a message for the client has been held longer than this time, the client will be disconnected. This parameter is an AMPS time interval (for example, 30s for 30 seconds, or 1h for 1 hour).
Notice that this policy applies to all messages and all connections.
If you have applications that will consume large result sets (SOW queries) over low-bandwidth network connections, consider creating a separate transport with the age limit set higher to allow those operations to complete.
Default: No age limit
ClientMaxCapacity
The amount of available capacity a single client can consume. Before a client is offlined, this limit applies to the MessageMemoryLimit. After a client is offlined, this limit applies to the MessageDiskLimit. This parameter is a percentage of the total.
Default: 50% (previous versions defaulted to 100%)
Client offlining can require careful configuration, particularly in situations where applications retrieve large result sets from SOW queries when the application starts up. More information on tuning slow client offlining for AMPS is available in Slow Client Offlining for Large Result Sets section in the Operations Best Practices section.
Last updated