LogoLogo
AMPS Server Documentation 5.3.4
AMPS Server Documentation 5.3.4
  • Welcome to AMPS 5.3.4
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
      • Production Configuration
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Configuring a Transaction Log
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Advanced Queue Configuration
      • Queue Subscriptions Compared to Bookmark Replays
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Configuring Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to a File
      • Logging to a Compressed File
      • Logging to Syslog
      • Logging to the Console
      • Looking up Errors with ampserr
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Automating AMPS with Actions
    • Replicating Messages Between Instances
      • Replication Basics
      • Configuring Replication
      • Replication Configuration Validation
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
      • Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Optional Functions
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Optional SOW Key Generator
        • Chaining Key Generator
      • Optional Authentication/Entitlements Modules
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • Simple Access Entitlements Module
      • Optional Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • AMPS Statistics
    • File Format Versions
  • AMPS Configuration Guide
    • AMPS Configuration Basics
      • Getting Started With AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working With Configuration Files
      • Including External Files
    • Instance Level Configuration
    • Admin Server and Statistics
    • Modules
    • Message Types
    • Transports
    • Logging
    • State of the World (SOW)
      • SOW/Topic
      • SOW/*Queue
      • SOW/ConflatedTopic
      • SOW/View
    • Replication
      • Replication Validation
    • Transaction Log
    • Authentication
    • Entitlement
    • Actions
      • Configuration for Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Protocols
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu
      • disks
      • memory
      • name
      • network
    • AMPS Instance Statistics
      • api
      • clients
      • config.xml
      • config_path
      • conflated_topics
      • cpu
      • cwd
      • description
      • environment
      • lifetimes
      • logging
      • memory
      • message_types
      • name
      • name_hash
      • pid
      • processors
      • queues
      • queries
      • replication
      • sow
      • statistics
      • subscriptions
      • timestamp
      • transaction_log
      • transports
      • tuning
      • uptime
      • user_id
      • version
      • views
  • AMPS Command Reference
    • Commands to AMPS
      • logon
      • Publishing
        • publish
        • delta_publish
      • Subscribing to and Querying Topics
        • subscribe
        • sow
        • sow_and_subscribe
        • unsubscribe
        • delta_subscribe
        • sow_and_delta_subscribe
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat
      • flush
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end : Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
  • Topics, Publish and Subscribe
  • Topic Configuration for Basic Pub/Sub
  • Regular Expression Subscriptions
  • Content Filters
  • Further Reading
Export as PDF
  1. Introduction to AMPS

AMPS Basics: Subscribe and Publish to Topics

PreviousGalvanometer and RESTful StatisticsNextState of the World (SOW): The Message Database

Last updated 4 months ago

Topics, Publish and Subscribe

The simplest way to use AMPS is as a low-latency publish and subscribe messaging system. Publish and subscribe messaging is at the heart of AMPS, and all of the other features of AMPS build on this foundation.

In a publish and subscribe messaging system, publishers send messages without necessarily knowing which subscribers will receive the message. This decouples publishers from subscribers for maximum flexibility.

While publishers do not need specific knowledge about the subscribers, publishers are responsible for adding information to the message so that subscribers know which messages are of interest. In publish and subscribe messaging systems, including AMPS, publishers send messages to a specific topic. The topic most often indicates the type of message, and is a way for the subscriber to locate the messages of interest. For example, in an order processing system, a publisher might publish messages to an ORDERS topic. Subscribers that need to receive orders then subscribe to the ORDERS topic, and receive messages that are sent to that topic.

Each message in AMPS is published to a specific topic. The publisher chooses the topic when the message is published, and subscribers can receive messages from that topic.

Unlike many messaging systems, AMPS provides an additional layer of selectivity for subscribers. Rather than receiving every message from a given topic, an AMPS subscriber can use content filtering to receive only the messages that the subscriber needs to process. Content filtering provides several advantages. First, being more selective about the messages delivered to the subscriber makes better use of bandwidth between AMPS and the subscriber, since the subscriber only receives messages that are of interest. Subscriber code is easier to write and more efficient because the subscriber is guaranteed that the messages received have the values requested. Further, because the subscriber chooses which messages to receive, content filtering makes publishers and subscribers less tightly-coupled. A publisher does not need to know what fields are important to a subscriber, or whether a field that was previously unused is now important.

The diagram below shows the basic concept of publish and subscribe messaging:

In the diagram above, there is a Publisher sending AMPS a message to the ORDERS topic. The message being sent contains information on Ticker IBM with a Price of 125. Both of these fields are contained within the message payload itself (i.e., the message content). AMPS routes the message to Subscriber 1 because it is subscribing to all messages on the ORDERS topic. Similarly, AMPS routes the message to Subscriber 2 because it is subscribed to any messages having the Ticker equal to IBM. Subscriber 3 is looking for a different Ticker value and is not sent the message.

Topic Configuration for Basic Pub/Sub

Unlike many messaging systems, AMPS does not require any configuration for simple publish and subscribe. For this basic functionality, there is no need to declare topics in advance. Since there is no need to declare these topics, topics that provide basic publish and subscribe are often referred to as ad hoc topics in AMPS.

It is valid for a publisher to publish to any topic, whether or not that topic has been previously configured. Likewise, it is valid for a subscriber to subscribe to any topic, whether or not a message has been previously published to that topic or whether the topic appears in a configuration file. Features that rely on persisted messages, however, are not available without configuration for a topic.

Every topic in AMPS has a specific message type. Publishers and subscribers don't need to explicitly set the message type when publishing to or subscribing to a topic. Each connection to AMPS specifies the message type to be used for that connection -- either implicitly, by connecting to a port that provides a specific message type, or explicitly (when connecting to a port that can provide multiple message types).

AMPS allows topics that use different message types to have the same name, but considers them to be different topics. Messages published to an XML topic named quotes will not be delivered to subscriptions to a JSON topic named quotes.

Regular Expression Subscriptions

AMPS allows subscribers to provide a regular expression that defines a set of topics rather than a literal topic name. This further decouples publishers and subscribers.

It's important to remember that each message is published to a specific topic. Regular expressions are only applicable to subscriptions: a publisher should not use regular expressions in the topic when publishing messages.

Spark: Basic Publish and Subscribe Example

Here's how to use spark to send and receive a message from AMPS. The example assumes that you're using the sample configuration file produced by the AMPS server, and that you are running spark on the same system that AMPS is running on.

First, start a subscriber:

  1. Open a terminal in your Linux environment.

  2. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to start a subscription:

    $ $AMPS_DIR/bin/spark subscribe -server localhost:9007 \
                                     -type json -topic test

    This command starts a subscription to the JSON topic test.

  3. spark will connect to AMPS, logon using default credentials (the current username and an empty password) and enter the subscription. Unless there are errors, the command will produce no output until a message arrives.

  4. Leave this terminal running. When you publish a message to the test topic, spark will print the message in this terminal.

Next, publish a message to the same topic:

  1. Open a new terminal in your Linux environment.

  2. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to send a single message to AMPS:

    $ echo '{"note":"Crank it up!"}' | \
      $AMPS_DIR/bin/spark publish -server localhost:9007 \
      -type json -topic test
  3. As with the subscriber sample, spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the publish command, spark reads the message from standard input and publishes the message to the JSON topic test. The command produces output similar to the following line (the rate calculation will likely be different):

    total messages published: 1 (333.33/s)
  4. When the publisher sends the message, the subscriber should receive the message and produce the following output:

    {"note":"Crank it up!"}

Congratulations! You've just sent your first message through AMPS.

Although this example is simple and relies on default behavior, the sample demonstrates some important AMPS concepts:

  1. As mentioned earlier, there is no need to preconfigure simple publish/subscribe topics. Since the default configuration file doesn't specify any settings for the JSON topic test, AMPS knows to treat the topic as a simple pub/sub topic. Also, since the spark commands specify the JSON message type when connecting to the server, the topic is a JSON topic.

  2. For simple publish and subscribe topics, AMPS delivers the message verbatim to the subscriber. AMPS doesn't interpret or normalize the message. In fact, AMPS doesn't even parse the message unless there's a need to. With this configuration and this subscription, there's no need for AMPS to parse the message, so no parsing happens.

  3. The spark program connects to AMPS and logs on to AMPS before sending any commands. All AMPS installations include authentication and entitlement. By default, AMPS loads an authentication and entitlement policy (implemented as an AMPS module) that requires a logon, but accepts any username and password as credentials. This policy is intended for evaluating, testing and development purposes. More information on securing an AMPS instance is available in the User Guide. For this example, the important point is to be aware that an AMPS instance always has a security policy and that policy was at work even in this simple example. The default behavior for spark works with the default policy for AMPS.

Content Filters

In the Basic Sub-Pub example, the subscriber requested all messages on the JSON test topic. AMPS includes expressive, flexible and extensible content filtering that allows subscribers to specify exactly the messages that they want to receive. Content filtering is one of the most useful features of AMPS. When subscribers use content filtering, publishers can be completely independent of subscribers. The publisher does not need to know which parts of a message are important to subscribers. Subscribers can precisely declare the content that they are interested in, so they only receive relevant messages. Publishers do not need to be updated when subscribers add additional criteria or when new subscribers come online.

You can think of content filtering as adding a WHERE clause to the subscription. Like a WHERE clause, AMPS returns only matching messages.

AMPS content filters use a combination of XPath identifiers to locate a value within a message and SQL-92 operators for comparing those values. For example, given a JSON message like:

{"note":"Hello, World!"}

The following content filters would match the message:

/note = 'Hello, World!'

This filter uses the equality operator, =, to compare the /note field in the message with an exact match for the string.

/note LIKE '(?i)world'

The LIKE operator uses Perl Compatible Regular Expressions to match a field. In this case, the regular expression matches any string that contains world, using case-insensitive matching.

/note BEGINS WITH 'Hello'

The AMPS BEGINS WITH operator matches any string that begins with the exact sequence of characters provided.

Spark: Subscription with Content Filter

Here's how to use spark to subscribe using a content filter. The example assumes that you're using the sample configuration file produced by the AMPS server and that you are running spark on the same system that AMPS is running on.

First, start a subscriber:

  1. Open a terminal in your Linux environment.

  2. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to start a subscription:

    $ $AMPS_DIR/bin/spark subscribe -server localhost:9007  \
                   -type json -topic test                   \
                   -filter "/note LIKE '(?i)sample'"

    This command starts a subscription to the JSON topic test. This subscription will only return messages where the filter matches.

  3. spark will connect to AMPS, logon using default credentials (the current username and an empty password) and enter the subscription. Unless there are errors, the command will produce no output until a message arrives.

  4. Leave this terminal running. When you publish a message to the test topic that matches the filter, spark will print the message in this terminal.

Next, publish messages to the subscriber:

  1. Open a new terminal in your Linux environment.

  2. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to publish a message to AMPS. This message matches the filter:

    $ echo '{"note":"Filter sample!"}' | \
      $AMPS_DIR/bin/spark publish -server localhost:9007 \
      -type json -topic test
  3. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to publish a message to AMPS. This message does not match the filter:

    $ echo '{"note":"Not a match. Sorry."}' | \
      $AMPS_DIR/bin/spark publish -server localhost:9007 \
      -type json -topic test
  4. Each time you run spark, it automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With each publish command, spark reads the message from standard input and publishes the message to the JSON topic test. Each of the commands above produces output similar to the following line (the rate calculation will likely be different):

    total messages published: 1 (333.33/s)
  5. When the publisher sends a message that matches the filter, the subscriber should receive the message and produce the following output:

    {"note":"Filter sample!"}

Further Reading

AMPS provides high-performance publish and subscribe messaging that requires minimal configuration and provides high performance, flexible publishing and message routing.

The AMPS client libraries include samples of basic publish and subscribe functionality. See the client library distribution for those samples.

Notice that some libraries are distributed as pre-built binaries through package management systems. 60East also offers full distributions including documentation and source from the 60East website.

If you've installed a pre-built library using a package manager, visit the 60East website to download the full distribution that contains the samples.

The sections on and have full details on the expression language used in AMPS.

See in the AMPS User Guide for a more complete discussion of subscribe and publish.

AMPS User Guide
AMPS Expressions
AMPS Functions
Subscribe and Publish
Basic Subscribe and Publish Functionality