Conflated Subscriptions
Last updated
Last updated
AMPS provides the ability for the server to conflate messages to a subscription. When a subscription requests conflation, the server will retain messages for that subscription for a certain period of time, the conflation interval, and provide the latest update to that message once a message has been retained for that interval. In effect, AMPS guarantees that a subscription will receive no more than one update for a given message per conflation interval.
Conflated subscriptions provide a way to reduce the bandwidth and processing for a subscriber in cases where a subscriber needs periodic updates with the current state of a message, rather than the complete message stream. AMPS provides per-subscription conflation for cases where only a small number of subscribers require conflation, or if conflation is required only in unusual cases. If multiple subscribers will have the same conflation needs, consider using .
For example, imagine an application that monitors selected stocks and displays the current prices on a large screen, which refreshes every few seconds. This application may use the same topics as a trading desk, but has very different needs for data freshness and completeness. Since updates to each symbol will only be displayed every few seconds, the application only needs point-in-time updates of the prices, rather than the full stream of price changes. To meet this need, the application could specify that the subscription conflates price updates by tickerId with a conflation interval of two seconds. For each distinct value of the tickerId field, AMPS will retain messages for two seconds. If another message with the same tickerId is processed for the subscription during the conflation interval, that message completely replaces the previous message. At the end of the two second conflation interval, the message is delivered to the application. This lets the application receive an up-to-date price at most every two seconds, without having to process a large number of updates that will never be displayed. This approach also ensures that the price is never more than two seconds out of date, which means that each time the screen is refreshed, the price is current.
As mentioned in the example above, if the subscription uses tickerId for conflation and the following sequence of messages arrive during a conflation interval:
AMPS delivers only the last message for that tickerId:
Notice that when a subscription is conflated, AMPS does not guarantee that messages are delivered precisely in the order in which they arrived in AMPS, since the latest update is delivered based on the conflation interval.
When the timestamp option is used with conflated subscriptions, AMPS provides the timestamp for the first message conflated.
Conflated subscriptions reduce the bandwidth for a subscription, and may reduce the processing resources required for a subscription. However, rather than immediately delivering messages, AMPS retains messages in memory for the conflation interval. This can increase the memory required for the subscription.
AMPS contains other features for conflating messages and reducing bandwidth. Conflated subscriptions are most appropriate when:
Network bandwidth is at a premium, and you would like AMPS to spend slightly more processing time and potentially more memory to reduce the bandwidth needs of the application.
Each subscription has different conflation needs. For example, if each subscription has a dramatically different conflation interval, or needs to conflate by different fields. If most subscribers will use a similar conflation interval and use the same fields for conflation, using a Conflated Topic can provide equivalent results with lower overhead.
The conflation needs are relatively predictable and consistent for the subscription. If you need the application to conflate messages only when processing is slow or there are bursts of message traffic, client-side conflation provides that ability and may be a better choice than a conflated subscription. See the developer guide for your programming language of choice for details.
The considerations above are general guidance to help you consider options and choose a conflation strategy.
To request conflation on a subscription, set the following options on the subscription:
conflation=n
Specifies whether to conflate this subscription. The value provided can be a time interval, auto or none.
When present and set to a value other than none, enables conflation for the subscription.
Can also be set to auto, which requests that AMPS attempt to determine an appropriate conflation interval based on client consumption.
Recognizes the same time specifiers used in the AMPS configuration file (for example, 100ms or 1s or 1m).
Defaults to none.
conflation_key=[keys]
When conflation is enabled, specifies the fields to use to determine message uniqueness. The format of this option is a comma-delimited list of XPath identifiers within brackets.
For example, to conflate based on the value of the /tickerId and /customerId within a message. the value of this option would be:
[/tickerId,/customerId]
Defaults to the SOW key fields for SOW topics.
No default for non-SOW topics. This option is required for non-SOW topics.
This option is not valid with the oof option unless the keys provided are identical to the keys for the topic.
This option can only be used when conflation is also specified.
For example, to request a 10 second conflation interval with messages conflated on the [/orderId] field, you would use the following options string:
You can also combine approaches as necessary. For example, if most of your subscriptions require a 3 second conflation interval by tickerId, while a few subscriptions require a 15 second interval, you could create a with a 3 second interval. Those subscriptions that require a 15 second interval could subscribe with that interval. This provides both sets of subscriptions with the intervals that they need.