LogoLogo
AMPS Server Documentation 5.3.4
AMPS Server Documentation 5.3.4
  • Welcome to AMPS 5.3.4
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
      • Production Configuration
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Configuring a Transaction Log
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Advanced Queue Configuration
      • Queue Subscriptions Compared to Bookmark Replays
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Configuring Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to a File
      • Logging to a Compressed File
      • Logging to Syslog
      • Logging to the Console
      • Looking up Errors with ampserr
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Automating AMPS with Actions
    • Replicating Messages Between Instances
      • Replication Basics
      • Configuring Replication
      • Replication Configuration Validation
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
      • Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Optional Functions
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Optional SOW Key Generator
        • Chaining Key Generator
      • Optional Authentication/Entitlements Modules
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • Simple Access Entitlements Module
      • Optional Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • AMPS Statistics
    • File Format Versions
  • AMPS Configuration Guide
    • AMPS Configuration Basics
      • Getting Started With AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working With Configuration Files
      • Including External Files
    • Instance Level Configuration
    • Admin Server and Statistics
    • Modules
    • Message Types
    • Transports
    • Logging
    • State of the World (SOW)
      • SOW/Topic
      • SOW/*Queue
      • SOW/ConflatedTopic
      • SOW/View
    • Replication
      • Replication Validation
    • Transaction Log
    • Authentication
    • Entitlement
    • Actions
      • Configuration for Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Protocols
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu
      • disks
      • memory
      • name
      • network
    • AMPS Instance Statistics
      • api
      • clients
      • config.xml
      • config_path
      • conflated_topics
      • cpu
      • cwd
      • description
      • environment
      • lifetimes
      • logging
      • memory
      • message_types
      • name
      • name_hash
      • pid
      • processors
      • queues
      • queries
      • replication
      • sow
      • statistics
      • subscriptions
      • timestamp
      • transaction_log
      • transports
      • tuning
      • uptime
      • user_id
      • version
      • views
  • AMPS Command Reference
    • Commands to AMPS
      • logon
      • Publishing
        • publish
        • delta_publish
      • Subscribing to and Querying Topics
        • subscribe
        • sow
        • sow_and_subscribe
        • unsubscribe
        • delta_subscribe
        • sow_and_delta_subscribe
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat
      • flush
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end : Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
  • Out-of-Focus Reason Codes
  • Usage
  • Example
  • Client-Side Filtering in a sow_and_subscribe Command
  • AMPS Filtering in a sow_and_subscribe Command
  • OOF Processing in a sow_and_subscribe Command
Export as PDF
  1. AMPS User Guide

Out-of-Focus Messages (OOF)

One of the more difficult problems in messaging is knowing when a record that previously matched a subscription has been updated so that the record no longer matches the subscription. AMPS solves this problem by providing an out-of-focus, or OOF, message to let subscribers know that a record they have previously received no longer matches the subscription. The OOF messages help subscribers easily maintain state and remove records that are no longer relevant.

OOF notification is optional. A subscriber must explicitly request that AMPS provide out-of-focus messages for a subscription.

When OOF notification has been requested, AMPS produces an oof message for any record that has previously been received by the subscription at the point at which:

  • The record is deleted

  • The record expires

  • The record no longer matches the filter criteria

  • The record is no longer within the pagination window (for paginated subscriptions), or

  • The subscriber is no longer entitled to view the new state of the record.

AMPS produces an oof message for each record that no longer matches the subscription. The oof message is sent as part of processing the update that caused the record to no longer match. Each oof message contains information the subscriber can use to identify the record that has gone out of focus and the reason that the record is now out of focus.

Since AMPS must maintain the current state of a record to know when to produce an oof message, these messages are only supported for SOW topics, conflated topics, and views. The oof option is not supported for subscriptions that do not include a SOW query or bookmark replays.

When AMPS returns an OOF message, the data contained in the body of the message represents the updated state of the message (except as described below). This will allow the client to make a determination as to how to handle the data, be it to remove the data from the client view or to change their query to broaden the filter thresholds. This enables a client to take a different action depending on why the message no longer matches. For example, an application may present a different icon for an order that moves to a status of completed than it would present for an order that moves to a status of cancelled.

When a delta_publish message causes the SOW record to go out of focus, AMPS returns the merged record.

When there is no updated message to send, AMPS sends the state of the record before the change that produced the oof. This can occur when the message had been deleted, when the message has expired, or when an update causes the client to no longer have permission to receive the record.

For a conflated view or a subscription that uses conflation, the data included in the oof message will be the last data that the subscriber received. When both an update to a message and a change that would cause the message to go out of focus happen in the same conflation interval, the subscriber receives an oof notification with the previously-received state of the message. Likewise, if a change that causes a message to go out of focus and a change that causes the message to come back into focus occur within the same conflation interval, the subscriber receives the state of the message at the end of the conflation interval. The subscriber does not receive an indication that the message had gone out of focus during the conflation interval and then come back into focus.

Out-of-Focus Reason Codes

An out of focus message returns the reason that the message was produced in the reason field of the message header.

Reason Field
Description
Message Contents

deleted

The message was deleted from the topic.

previous message

expired

The message was removed from the topic due to expiration.

previous message

match

The message no longer matches the content filter or has moved outside of the record set requested.

updated message that no longer matches subscription criteria

entitlement

The message has changed such that the user is not entitled to see the updated message.

previous message

Usage

Consider the following scenario where AMPS is configured with the following SOW key for the buyer topic:

<SOW>
    <Topic>
        <Name>buyer</Name>
        <MessageType>xml</MessageType>
        <Key>/buyer/id</Key>
    </Topic>
</SOW>

When the following message is published, it is persisted in the SOW topic:

<buyer>
    <id>100</id>
    <loc>NY</loc>
</buyer>

A client issues a sow_and_subscribe request for the topic buyer with the filter /buyer/loc="NY" and the oof option set on the request. The client will be sent the message as part of the SOW query result.

Subsequently, the following message is published to update the loc tag to LN:

<buyer>
    <id>100</id>
    <loc>LN</loc>
</buyer>

The original message in the SOW cache is updated. The client does not receive the second publish message, because that message does not match the filter (/buyer/loc="NY"). This is problematic. The client has a message that is no longer in the SOW cache and that no longer matches the current state of the record. Since the oof option was set on the subscription, however, the AMPS engine sends an oof message to let these clients know that the message that they hold is no longer in the SOW cache. The following is an example of what's returned.

The header of the message will contain the following fields to help the application identify the reason for the oof message and which message no longer matches:

Field
Value

Command

oof

Topic

buyer

Reason

match

SowKey

6387219447538349146

The message data will contain the updated message, as shown following.

<buyer>
    <id>100</id>
    <loc>LN</loc>
</buyer>

Had the message been deleted, the message data for the OOF notification would contain deleted message, and the reason would be deleted.

An easy way to think about the situations where AMPS sends an OOF notification is to consider what would happen if the client re-issued the original sow request after the above message was published. The /client/loc="NY" expression no longer matches the message in the SOW cache and as a result, this message would not be returned.

Example

To help reinforce the concept of OOF messages, and how OOF messaging can be used in AMPS, consider a scenario where there is a GUI application whose requirement is to display all open orders of a client. There are several possible solutions to ensure that the GUI client data is constantly updated as information changes, some of which are examined below; however, the goal of this section is to build up a sow_and_subscribe message to demonstrate the power that OOF notifications add to AMPS.

Client-Side Filtering in a sow_and_subscribe Command

First, consider an approach that sends a sow_and_subscribe message on the topic orders using the filter /Client="Adam".

AMPS completes the sow portion of this call by sending all matching messages from the orders SOW topic. AMPS then places a subscription whereby all future messages that match the filter get sent to the subscribing GUI client.

As the messages come in, the GUI client will be responsible for determining the state of the order. It does this by examining the State field and determining if the state is equal to “Open” or not, and then updating the GUI based on the information returned.

AMPS Filtering in a sow_and_subscribe Command

The next step is to add an additional ’AND’ clause to the filter. In this scenario we can let AMPS do the filtering work that was previously handled on the client. This is accomplished by modifying our original sow_and_subscribe to use the following filter:

/Client = "Adam" AND /State = "Open"

Similar to the above case, this sow_and_subscribe will first send all messages from the orders SOW topic that have a Client field matching "Adam" and a State field matching "Open". Once all of the SOW topic messages have been sent to the client, the subscription will ensure that all future messages matching the filter will be sent to the client.

There is a less obvious issue with this approach to maintaining the client state. The problem with this solution is that, while it initially will yield all open orders for client "Adam", this scenario is unable to stay in sync with the server. For example, when the order for Adam is filled, the State changes to State=Filled. This means that, inside AMPS, the order on the client will no longer match the initial filter criteria. The client will continue to display and maintain these out-of-sync records. Since the client is not subscribed to messages with a State of “Filled,” the GUI client would never be updated to reflect this change.

OOF Processing in a sow_and_subscribe Command

The final solution is to implement the same sow_and_subscribe query which was used in the first scenario. This time, we use the filter requests only for the State that we're interested in, but we add the oof option to the command so the subscriber receives OOF messages.

/Client = "Adam" AND /State = "Open"
options: oof

AMPS will respond immediately with the query results, exactly as it does with a sow_and_subscribe command that does not use the oof option.

This approach provides the following advantage: for all future messages in which the same Open order is updated, such that its status is no longer Open, AMPS will send the client an OOF message specifying that the record which previously matched the filter criteria has fallen out of focus. AMPS will not send any further information about the message unless another incoming AMPS message causes that message to come back into focus.

In the following diagram, the Publisher publishes a message stating that Adam’s order for MSFT has been fulfilled. When AMPS processes this message, it will notify the GUI client with an oof message that the original record no longer matches the filter criteria. The oof message will include a Reason field with it in the message header, defining the reason for the message to lose focus. In this case the Reason field will state match since the record no longer matches the filter.

AMPS will also send oof messages when a message is deleted or has expired from the SOW topic.

We see the power of the oof message when a client application wants to have a local cache that is a subset of the SOW. This is best managed by first issuing a query filter sow_and_subscribe which populates the GUI, and enabling the oof option. AMPS informs our application when those records which originally matched no longer do, at which time the program can remove them.

PreviousBatching Query ResultsNextState of the World Message Enrichment

Last updated 11 months ago

This approach puts the burden of work on the GUI and, in a high volume environment, has the potential to make the client GUI unresponsive due to the potential load that this filtering can place on a CPU. If a client GUI becomes unresponsive, AMPS will queue the messages to ensure that the client is given the opportunity to catch up. The specifics of how AMPS handles slow clients is covered in the section discussing .

Slow Client Management
sow_and_subscribe example
/State filter in a sow_and_subscribe
sow_and_subscribe with oof enabled
oof message