LogoLogo
AMPS Server Documentation 5.3.4
AMPS Server Documentation 5.3.4
  • Welcome to AMPS 5.3.4
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
      • Production Configuration
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Configuring a Transaction Log
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Advanced Queue Configuration
      • Queue Subscriptions Compared to Bookmark Replays
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Configuring Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to a File
      • Logging to a Compressed File
      • Logging to Syslog
      • Logging to the Console
      • Looking up Errors with ampserr
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Automating AMPS with Actions
    • Replicating Messages Between Instances
      • Replication Basics
      • Configuring Replication
      • Replication Configuration Validation
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
      • Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Optional Functions
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Optional SOW Key Generator
        • Chaining Key Generator
      • Optional Authentication/Entitlements Modules
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • Simple Access Entitlements Module
      • Optional Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • AMPS Statistics
    • File Format Versions
  • AMPS Configuration Guide
    • AMPS Configuration Basics
      • Getting Started With AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working With Configuration Files
      • Including External Files
    • Instance Level Configuration
    • Admin Server and Statistics
    • Modules
    • Message Types
    • Transports
    • Logging
    • State of the World (SOW)
      • SOW/Topic
      • SOW/*Queue
      • SOW/ConflatedTopic
      • SOW/View
    • Replication
      • Replication Validation
    • Transaction Log
    • Authentication
    • Entitlement
    • Actions
      • Configuration for Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Protocols
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu
      • disks
      • memory
      • name
      • network
    • AMPS Instance Statistics
      • api
      • clients
      • config.xml
      • config_path
      • conflated_topics
      • cpu
      • cwd
      • description
      • environment
      • lifetimes
      • logging
      • memory
      • message_types
      • name
      • name_hash
      • pid
      • processors
      • queues
      • queries
      • replication
      • sow
      • statistics
      • subscriptions
      • timestamp
      • transaction_log
      • transports
      • tuning
      • uptime
      • user_id
      • version
      • views
  • AMPS Command Reference
    • Commands to AMPS
      • logon
      • Publishing
        • publish
        • delta_publish
      • Subscribing to and Querying Topics
        • subscribe
        • sow
        • sow_and_subscribe
        • unsubscribe
        • delta_subscribe
        • sow_and_delta_subscribe
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat
      • flush
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end : Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
  • Using Multiple Underlying Topics
  • Priority Queues
  • Synchronizing Work with Barrier Messages
  • Limiting Currently Deliverable Messages
Export as PDF
  1. AMPS User Guide
  2. Message Queues

Advanced Queue Configuration

This section describes more advanced configuration settings for AMPS message queues.

Using Multiple Underlying Topics

AMPS queues can contain messages from any number of underlying topics. This provides a flexible delivery model, and allows applications to populate multiple queues with a single publish to AMPS, which simplifies publisher code, reduces bandwidth, and ensures that the message is provided to all queues from the same point in the message stream.

To create a queue that includes messages from underlying topics, you provide a regular expression that matches the set of topic names that contain messages for the queue. You also provide a DefaultPublishTarget that specifies the topic name for AMPS to use when a message is published directly to the queue topic.

For example, you might configure a set of topics as follows:

<SOW>
  ...
  <Queue>
     <Name>ORDERS_ANALYTICS</Name>
     <MessageType>json</MessageType>
     <UnderlyingTopic>^ORDERS$|^ORDERS_ANALYTICS_DIRECT$</UnderlyingTopic>
     <DefaultPublishTarget>ORDERS_ANALYTICS_DIRECT</DefaultPublishTarget>
  </Queue>
  <Queue>
     <Name>ORDERS_RISK</Name>
     <MessageType>json</MessageType>
     <UnderlyingTopic>^ORDERS$|^ORDERS_RISK_DIRECT$</UnderlyingTopic>
     <DefaultPublishTarget>ORDERS_RISK_DIRECT</DefaultPublishTarget>
  </Queue>
  ...
</SOW>

In this case, when a message is published to the ORDERS topic, both the ORDERS_ANALYTICS and the ORDERS_RISK queues deliver the message. However, a publisher can also publish directly to each queue by publishing a message to the _DIRECT topic for that queue. Furthermore, any publish to the name of the queue will be routed to the appropriate _DIRECT topic.

The following table demonstrates how messages are provided to topics with this configuration:

Publish To
Results

ORDERS

Both ORDERS_ANALYTICS and ORDERS_RISK enqueue the message, since ORDERS matches the UnderlyingTopic of both queues.

ORDERS_ANALYTICS

The message is published to the DefaultPublishTarget of ORDERS_ANALYTICS, which is ORDERS_ANALYTICS_DIRECT.

The message is then enqueued to ORDERS_ANALYTICS, since ORDERS_ANALYTICS_DIRECT matches the UnderlyingTopic of ORDERS_ANALYTICS.

ORDERS_RISK

The message is published to the DefaultPublishTarget of ORDERS_RISK, which is ORDERS_RISK_DIRECT.

The message is then enqueued to ORDERS_RISK, since ORDERS_RISK_DIRECT matches the UnderlyingTopic ORDERS_RISK.

ORDERS_ANALYTICS_DIRECT

The message is published to ORDERS_ANALYTICS_DIRECT, and is then enqueued to ORDERS_ANALYTICS.

ORDERS_RISK_DIRECT

The message is published to ORDERS_RISK_DIRECT, and is then enqueued to ORDERS_RISK.

Priority Queues

AMPS includes the ability to specify that messages from a queue are delivered in order of priority rather than being delivered strictly in publication order. To enable prioritization on a Queue, add the Priority tag to the queue configuration and specify the field or expression to use to set the priority. When a queue definition specifies Priority, AMPS orders delivery based on a descending sort of the value calculated for the Priority rather than in descending order of age.

For example, to create a queue that delivers messages in order of the product of the price and quantity specified in the message, you could use the following queue definition:

<AMPSConfig>
    ...
    <!-- Other configuration, including
         recording  that OrderPriority
         and Orders are recorded in the
         transaction log.
      -->

    <SOW>
        ...
        <Queue>
            <Name>OrderPriority</Name>
            <MessageType>json</MessageType>
            <Semantics>at-least-once</Semantics>
            <UnderlyingTopic>Orders</UnderlyingTopic>
            <Priority>/price * /qty</Priority>
        </Queue>
    </SOW>

    ...
</AMPSConfig>

Synchronizing Work with Barrier Messages

AMPS queues provide the ability to synchronize work across a set of subscribers by sending the same message to all subscribers simultaneously, when the queue is fully processed up to the point of the message.

These messages are referred to as barrier messages, since they are roughly equivalent to the concept of a barrier in multithreaded programming.

Barrier messages simplify coordination between current subscribers to a queue. They allow you to easily coordinate operations (such as end of day processing), update reference data at the precise point in the message stream that the update should take effect, send a shutdown message when all available work is processed, and so on. Since barrier messages are integrated with queue delivery, there is no need to write extra code to correlate the barrier message with the correct point in the queue, to manage multiple subscriptions, or to write code that guarantees delivery to all current subscribers.

To specify which messages should be treated as barrier messages, the queue configuration includes a BarrierExpression element that contains a filter. All messages that match that filter are considered barrier expressions.

When a message matches the BarrierExpression, AMPS delivers the message in the following way:

  • AMPS does not deliver the message until all previous messages in the queue are acknowledged.

  • AMPS does not deliver messages that are after the barrier message until the barrier message is sent to subscribers.

  • When all previous messages in the queue are acknowledged, AMPS delivers the barrier message to all current subscribers to the queue.

  • AMPS immediately removes the message from the queue, without requiring acknowledgment from any subscriber.

A barrier message sent to a subscriber does not count toward that subscriber's backlog, since the barrier message is immediately acknowledged. When sending the barrier message, AMPS does not consider the current backlog of the subscriber (so, subscribers that request max_backlog=0 or which use a regular expression topic and have their backlog filled by messages from other topics will still receive the barrier message).

AMPS does, however, apply content filters and entitlement filters for each subscription when delivering the message. If the content filter or entitlement filter for a subscription does not match the barrier message, that subscription will not receive the message. The fact that this subscription does not receive the actual barrier message does not affect the behavior of the barrier itself. A subscription that does not match the message will still see message delivery pause until the barrier is released, but will not receive the actual barrier message.

Each AMPS instance manages the delivery of barrier messages for the subscribers connected to that instance, when all messages prior to the barrier message in the local queue have been acknowledged.

Limiting Currently Deliverable Messages

In some cases, it may be important to limit the maximum number of messages that AMPS considers to be part of the currently active, deliverable part of the queue. This can be advantageous in cases where memory is limited, or in cases where it is expected that a queue is only processed infrequently (for example, a reconciliation process that runs at the end of each day).

The TargetQueueDepth option on a queue allows you to set the target depth for the number of messages that AMPS will consider to be deliverable at a given time.

If the number of messages for the queue exceeds the TargetQueueDepth on an instance, messages beyond the specified depth are considered to be inactive (though still part of the queue). Those messages are not deliverable to subscribers on this instance. They do not appear in a sow query of the queue, and are not considered when a view over the queue is calculated. A sow_delete that uses a filter will only apply to the messages in the queue that are currently deliverable. As messages are acknowledged from the queue, AMPS will add messages to the active part of the queue until the active part of the queue once again reaches the TargetQueueDepth. For messages that are not currently deliverable, AMPS will not read those messages from the transaction log and will not maintain queue state for those messages. This also means that, if no messages within the TargetQueueDepth match a given subscription, no messages will be delivered to that subscription even if later messages in the transaction log match the subscription.

Although AMPS does not consider messages that are currently inactive to be deliverable, AMPS will preserve queue delivery guarantees in the event that a message that is not currently deliverable is acknowledged before it enters the queue and becomes deliverable. To do this, AMPS keeps a list of acknowledgments that have been received for messages that are not in the queue. When a message that would be added to the active part of the queue has already been acknowledged, AMPS does not add that message to the queue. Applications that use TargetQueueDepth to limit memory growth should avoid a pattern where messages are acknowledged before they enter the queue to reduce memory consumption for acknowledgments that cannot yet be processed.

Likewise, if a transfer request from another instance arrives for a message that is in the transaction log, but is beyond the limit set by the TargetQueueDepth, AMPS will process messages further into the queue until it reaches the message to be transferred. At that point, AMPS will transfer the message and stop processing further messages until either:

  • The current depth drops below the TargetQueueDepth, or

  • AMPS once again receives a request for a message that is in the transaction log, but beyond the current active messages in the queue.

A TargetQueueDepth cannot be set for a queue that specifies a Priority or a BarrierExpression.

PreviousHandling Unprocessed MessagesNextQueue Subscriptions Compared to Bookmark Replays

Last updated 1 year ago

Priority expressions use the same grammar as other expressions in AMPS, as described in . The results of the Priority expression are interpreted as an unsigned long. That is, the result should be a positive integer that can be represented in 64 bits. Non-numeric values or NULL are treated as NaN, and have the lowest priority.

AMPS Expressions