LogoLogo
AMPS Server Documentation 5.3.4
AMPS Server Documentation 5.3.4
  • Welcome to AMPS 5.3.4
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
      • Production Configuration
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Configuring a Transaction Log
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Advanced Queue Configuration
      • Queue Subscriptions Compared to Bookmark Replays
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Configuring Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to a File
      • Logging to a Compressed File
      • Logging to Syslog
      • Logging to the Console
      • Looking up Errors with ampserr
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Automating AMPS with Actions
    • Replicating Messages Between Instances
      • Replication Basics
      • Configuring Replication
      • Replication Configuration Validation
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
      • Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Optional Functions
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Optional SOW Key Generator
        • Chaining Key Generator
      • Optional Authentication/Entitlements Modules
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • Simple Access Entitlements Module
      • Optional Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • AMPS Statistics
    • File Format Versions
  • AMPS Configuration Guide
    • AMPS Configuration Basics
      • Getting Started With AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working With Configuration Files
      • Including External Files
    • Instance Level Configuration
    • Admin Server and Statistics
    • Modules
    • Message Types
    • Transports
    • Logging
    • State of the World (SOW)
      • SOW/Topic
      • SOW/*Queue
      • SOW/ConflatedTopic
      • SOW/View
    • Replication
      • Replication Validation
    • Transaction Log
    • Authentication
    • Entitlement
    • Actions
      • Configuration for Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Protocols
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu
      • disks
      • memory
      • name
      • network
    • AMPS Instance Statistics
      • api
      • clients
      • config.xml
      • config_path
      • conflated_topics
      • cpu
      • cwd
      • description
      • environment
      • lifetimes
      • logging
      • memory
      • message_types
      • name
      • name_hash
      • pid
      • processors
      • queues
      • queries
      • replication
      • sow
      • statistics
      • subscriptions
      • timestamp
      • transaction_log
      • transports
      • tuning
      • uptime
      • user_id
      • version
      • views
  • AMPS Command Reference
    • Commands to AMPS
      • logon
      • Publishing
        • publish
        • delta_publish
      • Subscribing to and Querying Topics
        • subscribe
        • sow
        • sow_and_subscribe
        • unsubscribe
        • delta_subscribe
        • sow_and_delta_subscribe
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat
      • flush
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end : Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
  • When to Use a Pattern in a Topic
  • Limitations When Storing Multiple Logical Topics in a Physical Topic
  • Configuration File Precedence
  • Logical Topics and the Transaction Log
  • Entitlements for Logical Topics
Export as PDF
  1. AMPS User Guide
  2. State of the World (SOW) Topics

Storing Multiple Logical Topics in One Physical Topic

AMPS provides a way to easily define a set of related SOW topics by specifying a Pattern element in the Topic configuration. When this element is present, AMPS creates a container SOW topic that can include a number of SOW topics in one physical file. A publish to a topic name that matches the Pattern will be treated as an individual SOW topic within the container topic that defines the pattern. The definition of each individual topic (for example, the Key values defined, the hash indexes defined, and so on) is defined by the container Topic, and is the same for every individual topic within the container SOW topic.

Using this approach creates a single physical topic (that is, the container is a single in-memory SOW topic and, when the topic is persisted, a single file) that contains records for any number of individual topic names. The topics within the container maintain the last value of each individual record within each of the topics. Publishers and subscribers can use these topics as though the topics were each configured individually as a Topic in the AMPS configuration file (with some minor behavioral differences resulting from all of the topics and messages being stored in the same data structure, as described in the following sections).

Although AMPS treats every topic within the container SOW topic as a distinct topic for the purposes of publishing and subscribing, AMPS manages those topics as records within a single SOW object. When the overall SOW topic is persisted, every message for an individual topic is stored within the same file. Likewise, the overall SOW topic is treated as a single topic in memory (including for monitoring and statistics purposes). In cases where an application has a large number of topics and each topic has a small number of messages (typically, in cases where each topic has only a single message), using a Pattern can use considerably less memory than individual Topic entries for the same number of topics and messages.

Legacy Protocol Note

Querying or subscribing to topics that use the Pattern element is supported for connections using the amps protocol. Legacy protocols do not include support for this feature.

60East recommends using a Pattern for a topic in situations where an existing system uses topic names rather than content filtering, and it is not practical to adjust the system. For example, when migrating a legacy system that distinguishes orders for different customers using different topic names rather than using the content of the message (such as using topic names /orders/customerA and /orders/customerB rather than including a customer field on the message), creating a SOW topic using a Pattern of ^/orders/ might be the most straightforward way to adapt the system to AMPS.

For a small number of topics, or cases where an individual topic would have a large number of entries, 60East recommends using individual topics rather than specifying a Pattern.

When to Use a Pattern in a Topic

The Pattern element allows you to define a large number of SOW topics that will hold a small number of records (typically, only one record per topic) while minimizing the memory and storage overhead for each topic. This can be especially helpful when migrating a system that uses topic-based routing to AMPS, since you can easily create a large number of topics (hundreds or thousands) without having to explicitly specify each one in the AMPS configuration file in cases where it is important to query the last value of each topic.

Consider using the Pattern element in cases where:

  • You need to be able to query the current value of a record (or topic). If you do not need to query current values, there is no need to define topics in the SOW at all (consider using ad hoc topics instead).

  • The information that determines whether a given message is unique is not contained in the message itself. If that information is already present in the data, it is more efficient to use a Topic with the unique property configured as a Key.

  • The messages have the same structure and are the same logical type of message. Messages that are different types, or that have different structures, would typically be represented in different topics.

  • Your application requires a large number of topics, or you do not know the topics in advance, such that it is impractical to define the topics using individual Topic declarations.

  • Each unique topic will have a small number of messages (ideally, only one message per topic).

  • Your application does not require historical point in time query, or enrichment on the messages.

  • All of the topics to be managed together have the same general set of permissions. AMPS does not support applying different entitlements to individual topics within an overall SOW topic (some limited workarounds are available through content filters).

If any of the above considerations are not true, consider using a set of Topic declarations rather than using the Pattern element in a single Topic.

Container topics are most commonly used when adapting a system that did not support content filtering (content-based routing) to an AMPS-based application in cases where the message data itself does not contain enough information to support content-based routing. Applications designed for AMPS most frequently use a Topic and content filtering rather specifying a Pattern for a Topic and providing routing information in the topic name.

Limitations When Storing Multiple Logical Topics in a Physical Topic

For most purposes, topics that use a Pattern work just like any other topic defined using the Topic directive. However, there are some differences in behavior, as outlined below:

  • When an application issues a sow or sow_and_subscribe that uses a regular expression for the topic name, messages from topics within a topic that uses Pattern are delivered between a single group_begin and group_end pair. Messages from any topic name within the topic may be delivered in any order within the query results. Each message will indicate which topic within the topic it originated from.

  • A topic that uses Pattern cannot be the underlying topic for a view.

  • A topic that uses Pattern can be the underlying topic for a conflated topic, but the conflated topic must be configured to use such a topic.

  • All of the topic names within the topic must have the same permissions.

Configuration File Precedence

AMPS allows you to define a standalone topic, view, queue, or conflated topic with a Name that matches the Pattern of the Topic. To do so, however, that definition must appear in the configuration file before the definition of the topic that uses the Pattern. The topic, view, queue, or conflated topic will be configured as though the topic that defined the Pattern is not present.

For example, the following SOW configuration creates a Topic named /orders/specialHandling and a Topic with a Pattern that matches ^/orders/. The /orders/specialHandling topic adds preprocessing, and could also, in principle, have different permissions than the topic names that are matched by the Pattern.

<SOW>
  <Topic>
     <Name>/orders/specialHandling</Name>
     <MessageType>json</MessageType>
     <Key>/orderId</Key>
     <Preprocessing>
       <Field>COALESCE(/orderId,
                  CONCAT(/customerName, /customerSerialNumber)) as /orderId</Field>
     </Preprocessing>
     <FileName>./sow/%n.sow</FileName>
  </Topic>

  <!-- Regular expression topic since this definition
       has a Pattern directive. -->

  <Topic>
     <Name>RegexOrders</Name>
     <Pattern>^/orders/</Pattern>
     <MessageType>json</MessageType>
     <Key>/orderId</Key>
     <FileName>./sow/%n.sow</FileName>
  </Topic>
</SOW>

With these definitions, a publish to the following topic names would produce the following results:

Message Published to Topic

Results

/orders/specialHandling

Matches Topic definition. Stored in the Topic.

The Preprocessing directive runs and creates the /orderId from the /customerName and /customerSerialNumber if there is no /orderId already present.

/orders/RHAT

Matches the regular expression topic definition, stored in the regular expression topic.

/orders/specialHandling/oops

Matches the regular expression topic definition, stored in the regular expression topic.

Notice that a Topic definition is an exact match on the topic name, not a pattern match.

/customer/orders/timothy_someone

Does not match either the Topic or the regular expression topic.

Not included in the SOW.

Logical Topics and the Transaction Log

The transaction log can be used to record publishes to a physical topic that contains multiple logical topics.

To do this, the transaction log specification must contain a Topic directive that matches the physical topic Name. This will capture all of the logical topics in the transaction log. Notice that only the physical topic Name is consiered in this case.

Entitlements for Logical Topics

AMPS considers permissions for all of the logical topics within the physical SOW topic to be identical. When checking permissions for the topic with an entitlement module, AMPS requests that the module provide permissions for the Pattern specified in the topic. Any topic name included in the container will use the permissions, entitlement filter, and entitlement select list specified by the module for that Pattern.

If it becomes necessary to restrict access to individual topics within the physical topic, there are two approaches that you can take:

  1. Create a new topic with a Pattern that specifies the topics that require different permissions, and apply the permissions to that topic.

  2. Provide an entitlement filter that uses the TOPIC_NAME() function to restrict access to specific topic names; for example, TOPIC_NAME() IN ('/orders/RHAT', '/orders/MSFT', '/orders/IBM'). Using this method is less efficient than providing permissions for those topics (either as standalone topics, or for a regular expression topic containing exactly those three topics), but this approach can be a good option in cases where subscribers typically subscribe only to the topics they are entitled to, different subscribers have substantially different sets of entitlements, or there are no logical or convenient groupings that can be used to separate the topics into several regular expression topic declarations.

PreviousSetting Per-Message LifetimeNextQuerying the State of the World (SOW)

Last updated 3 months ago