1 of 100

AMPS Server Documentation 5.3.4

Welcome to AMPS 5.3.4

Welcome to the AMPS Server Documentation! This set of documentation contains detailed information on the AMPS server itself for version 5.3.4.

If you are looking for developer docs for client libraries, or previous versions of the AMPS server documentation, see the documentation page on the 60East web site.

Here are some suggested starting points:

You can also visit the AMPS Server FAQ site for frequently asked questions, and the AMPS developer pages for resources on developing applications with AMPS.

Introduction to AMPS

Welcome to the Advanced Message Processing System (AMPS) from 60East Technologies! AMPS is designed to help you quickly and easily develop and deploy data-intensive applications, with demanding requirements, for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems.

AMPS isn't a traditional database or messaging product. This guide presents a brief introduction to help you understand the capabilities of AMPS and how AMPS operates.

AMPS is widely used for applications such as:

Tradeplant operations (including backtesting and historical analysis)
Risk calculations
Elastic worker farms
View servers
Message flow integration and "shock absorbers"

AMPS combines a set of capabilities that cut across traditional boundaries between applications that work with data.

AMPS is built around a fast messaging engine that supports both publish and subscribe (fan-out) and queued (competitive consumption) message delivery with full content filtering.

AMPS also provides an integrated database that applications can use as a current value cache, key/value document store, and fully queryable database -- or all of these at once. With this database, AMPS includes a built-in aggregation and analytics engine for near-real time analysis of streaming data, including aggregation across multiple topics or message formats.

Integrated message logging provides the ability to record and replay streams of messages with full fidelity.

AMPS is designed from the ground up for enterprise deployment at scale. AMPS provides an extensive set of high-availability features, including integrated replication and automatic failover and recovery for applications. Detailed monitoring and statistics are included from a RESTful interface for ease of data collection and integration with enterprise monitoring and management systems.

Authentication and entitlement capability applies to every operation in AMPS, for fine-grained control over permissions to meet enterprise policy and regulatory requirements. Access to data can be controlled at a topic level, at a message level (content-based security), or at the level of individual fields within a message (limiting the fields a given user has access to view).

60East developed AMPS to serve the needs of some of the most demanding data-intensive applications on the planet. The feature set and capabilities have been engineered for the highest levels of performance, designed for ease of use, and proven in production applications worldwide.

Getting to Know AMPS

AMPS is designed to be a developer friendly product. 60East recommends reading about AMPS with a running instance of AMPS and your development environment of choice available. Although 60East makes every effort to clearly describe how AMPS works, there is no substitute for seeing exactly how a running instance behaves (not to mention the advantages of being able to try out ideas or do quick prototyping while you read).

The table below lists the main parts of the AMPS documentation:

Title

Description

Overview of AMPS functionality.

This is a good place to start if you are new to AMPS or if you are familiar with older versions of AMPS.

Detailed description of AMPS functionality, including guidance and best practices.

If you have detailed questions about how AMPS works, refer to the AMPS User Guide.

Guide to AMPS configuration.

This guide shows the configuration file syntax and accepted values. The guide also includes useful examples of configuring commonly-used options.

Guide to the RESTful monitoring interface and the AMPS statistics database.

Use this guide when creating a monitoring strategy or when collecting statistics about an instance.

Description of the commands sent from an AMPS client to the AMPS server and responses from the server.

Client Language Developer Guides

Guide to using a client library to work with AMPS.

This guide uses the spark command line utility for basic examples for simplicity, although a production installation would use an application to perform these functions.

Overview of AMPS

AMPS Concepts

This section describes the overall AMPS approach and the features AMPS provides.

The AMPS messaging system is designed around a few simple principles:

Parallelize work and minimize waits and blocking to take full advantage of modern multisocket, multicore systems.
Eliminate redundant or unused work by only performing tasks that are necessary to provide the functionality requested by a given operation.
Reduce or eliminate cross-system coordination by solving the full range of data delivery and storage problems commonly faced by data-intensive applications.
Provide a small, flexible set of commands for ease of use.
Provide multiple delivery paradigms supporting both publish-subscribe delivery (many to many) and message queues (single consumption of a message) as well as the ability to query the state of a topic at a point in time.
Stay application-focused to provide exactly the capabilities that are heavily used in demanding high-performance applications.
Stay hardware aware and build for the future by engineering for next-generation commodity hardware and designing AMPS to fully exploit non-uniform memory access (NUMA), flash-based storage, and high-bandwidth networking.

These concepts are the foundation of how AMPS works and are helpful for understanding how to best use AMPS.

To best take advantage of AMPS, applications typically use the built-in features of AMPS rather than their traditional equivalents.

For example, rather than keeping a separate, independent record of each message published to AMPS for audit purposes, applications most often use one of the data persistence features in AMPS. This speeds development and simplifies deployment by eliminating integration effort, and also solves potential correctness issues which could be caused if messages in persistent storage become inconsistent with the messages provided through the messaging system. With AMPS, the messaging system itself can contain a fully-queryable and replayable record of the system.

As another example, AMPS provides integrated replication rather than relying on an external process. AMPS replication is aware of the format and semantics of the transaction log, the configuration of the instance, and the commands sent by publishers. This integration allows AMPS to very efficiently provide a full-fidelity message stream and to provide "self-healing" for an instance to catch up when it has been offline. Further, the message store used for replication (the AMPS transaction log) is also used for durable subscriptions and message replay. Designing and implementing these features together reduces complexity, storage requirements, and overhead to enable both capabilities. Within the AMPS server, the implementation uses a sophisticated parallelized algorithm for storage and replay that reduces overall latency and prevents slow consumers or replication destinations from affecting faster consumers. The overall result is to simplify configuration and application development, provide strong consistency and reliability guarantees, and provide the highest possible level of performance.

As a final example, rather than requiring a complex topic structure, requiring applications to oversubscribe and discard messages, AMPS provides both topic filtering and content filtering. AMPS includes an expressive filter grammar to provide precise selection of messages of interest to an individual subscriber. AMPS provides this capability to fully decouple publishers and subscribers. With AMPS, there's no need to maintain and administer a granular topic structure. Precise filtering and routing improves both network and processor utilization by providing only actionable messages to a subscriber. Likewise, for many applications, there is no need for a publisher to be aware of the processing performed by the subscriber or by AMPS itself.

The examples above highlight just a few of the capabilities AMPS provides and how the AMPS approach simplifies development, administration, and operations while providing reliability and performance benefits over conventional systems.

Feature Highlights

Some of the highlights of AMPS features include:

Topic based publish and subscribe, including full support for regular expressions to specify topic names.
Content filtering based on XPath identifiers (to specify the fields of a message) and SQL-92 (to form a predicate), with added support for Perl-Compatible Regular Expressions (PCRE2).
Message queues including content filtering for both publishers and subscribers, configurable strategies for delivery fairness, and truly distributed queues that can efficiently enforce queue semantics and delivery guarantees across a replicated network of AMPS instances.
Content-aware messaging support for a wide range of message types, including standard formats such as JSON, FIX, MessagePack, XML, Google Protocol Buffers, and BSON. AMPS also supports simple key/value pairs in FIX format (called NVFIX to emphasize that the format uses name/value pairs rather than FIX tags), and a high-performance binary protocol called BFlat. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing types to easily combine messages of different types in a single payload.
An integrated database and record-aware current value storage (called State of the World, or SOW), with optional historical query capability.
Historical replay of message streams, including the ability to preserve the total message ordering across independent topics.
Integrated replication and high availability, including automatic resynchronization for instances that fail over.
Aggregation and Complex Event Processing (CEP), including the ability to aggregate information across different message streams and message streams of different formats.
Advanced messaging capabilities such as atomic query-and-subscribe, incremental (delta) updates, and out-of-focus notifications that tell a subscription when a record no longer matches.
Built in statistics and monitoring, with data provided via a standard RESTful interface.
Integrated authentication and entitlement across all AMPS features.
Actions for automating AMPS functionality, including both routine maintenance tasks and dataflow-aware processing (such as alerting in response to slowdowns or invalid data).
Client development kits for popular programming languages such as Java, C#/.NET, C++, Python, JavaScript, and Go.
Extensibility API in the AMPS server for adding message types, extending the functions available to the AMPS query language, adding new actions, integrating with enterprise authentication and entitlement systems, and more.

This guide provides an overview of the most commonly used functionality of AMPS, but it is not intended to cover all of the features of AMPS or provide an exhaustive discussion of any individual feature. As mentioned earlier, the AMPS User Guide provides full details about AMPS features.

Getting Started With AMPS

Crank Up the AMPS

This chapter is for users who are new to AMPS and want to quickly get a simple instance of AMPS running. This chapter will describe how to install AMPS on a Linux system, describe the layout of the AMPS distribution, and use the included spark command line AMPS client to send and receive a simple message. If you are on a Windows system without easy access to a Linux installation, a section at the end of the chapter includes information on configuring a Linux virtual machine.

This section covers the following topics:

Installing AMPS

On the 60East website at http://www.crankuptheamps.com/evaluate the current release of AMPS is available for evaluation download.

To get started, download the Linux installation to a directory on your Linux system.

Installing AMPS is simply a matter of unpacking the distribution. The distribution contains the complete set of libraries and dependencies needed to run the AMPS server on a typical Linux server distribution. No additional software or packages are necessary for the server itself.

To install AMPS, unpack the distribution in the directory where you want the binaries and libraries to be stored. For the remainder of this guide, the installation directory will be referred to as $AMPSDIR as if an environment variable with that name was set to the correct path.

Within $AMPSDIR are the following sub-directories:

Starting AMPS

The AMPS engine binary is named ampServer and is found in $AMPSDIR/bin. Start the AMPS engine with a single command line argument that includes a valid path to an AMPS configuration file. You use the configuration file to enable and configure the AMPS features that your application will use. This guide discusses the most commonly used configuration options for each feature. The full set of options is described in the AMPS Configuration Guide.

The AMPS server generates a minimal sample configuration file with the --sample-config option. You can save the sample configuration file to $AMPSDIR/amps_config.xml with the following command line:

$AMPSDIR/bin/ampServer --sample-config > $AMPSDIR/amps_config.xml

The sample configuration file generated by AMPS includes a very minimal configuration. The client language distributions include a sample configuration file that sets up AMPS to work with the samples provided with that client, and the AMPS Configuration Guide contains a full description of the configuration items with sample configuration snippets.

The server sample configuration only provides configuration for using AMPS to subscribe to and publish to ad hoc topics. The sample configuration file does not include any persistence for AMPS messages.

The file enables the instance monitoring interface (the "Galvanometer"), including the ability to query and subscribe to topics using a websocket connection.

A production configuration would likely provide persistent event and error logging to a file to allow an operations team to troubleshoot the instance and would typically persist monitoring statistics to a file. Such a configuration would likely enable additional message delivery features for certain topics and would also include configuration for high-availability and disaster recovery. The configuration would typically configure AMPS actions to perform routine maintenance.

AMPS uses the current working directory for storing files (logs and persistence) for any relative paths specified in the configuration. While this is important for real deployments, the sample configuration used in this chapter does not persist anything, so you can safely start AMPS from any working directory using this configuration.

On older processor architectures (and in some emulated environments) ampServer will start the ampServer-compat binary. The ampServer-compat binary avoids using hardware instructions that are not available on these systems.

You can also set the AMPS_PLATFORM_COMPAT environment variable to force ampServer to start the ampServer-compat binary. 60East recommends using this option only on systems that do not support the hardware instructions used in the standard binary. The ampServer-compat binary will not perform as well as ampServer, since it uses fewer hardware optimizations.

Once you have a configuration file saved to $AMPSDIR/amps_config.xml you can start AMPS with that file as follows:

$AMPSDIR/bin/ampServer $AMPSDIR/amps_config.xml

If your first start-up is successful, you should see AMPS display a simple message similar to the following to let you know that your instance has started correctly.

AMPS A.B.C.D.973814.e1a57f7 - Copyright (c) 2006-202X 60East Technologies Inc.
(Built: XXXX-YY-ZZT00:26:45Z)

The version numbers and dates will be appropriate for the version that you've started.

If you see this, congratulations! You have successfully cranked up the AMPS!

Command Line Options

The AMPS server binary supports the following command line options:

Option

Effect

--verify-config

Parse and verify the specified configuration file, then exit.

--sample-config

Produce a minimal AMPS config.xml file to standard output, then exit.

--dump-config

Process the specified configuration file, resolving any Include directives and expanding environment variables. Dump the resulting file to standard output.

--version

Print the AMPS version string, then exit.

--help

Print usage information for the command line options accepted by the ampServer program, then exit.

--daemon

Run AMPS as a daemon process.

-D<variable>=<value>

Set the specified environment variable to the specified value when running the AMPS process. AMPS accepts any number of -D options.

For example, to set the variable AMPS_PATH to /mnt/fast/AMPS use the command line option -DAMPS_PATH=/mnt/fast/AMPS

JSON Messages - A Quick Primer

AMPS includes support for a wide variety of message types, as well as the ability to develop custom message types and to send binary payloads. This section focuses on JSON as the main message type used for samples in this guide. We use JSON for the guide because the format is simple, easily readable, and already in use in many environments.

JSON format is a simple, standardized message format. JSON has two basic constructs:

Objects that consist of key / value pairs
Arrays of values

JSON supports hierarchical construction: the value for a key can be a single value, an array of values, or another set of key/value pairs. For example, the following JSON message includes two nested sets of key value pairs. Notice that a key only needs to be unique within each set of values -- the name value for the ship does not conflict with the name value for the character.

{
    "id" : 73,
    "character" : {
        "name" : "Han Solo",
        "occupation" : "smuggler",
        "ship" : {
            "name"  : "Millennium Falcon",
            "speed" : ".5 past light speed",
            "cargo" : [ "widgets", "baskets", "spice"]
        }
    }
}

Many AMPS applications use JSON as the payload. In addition, the amps protocol used to send commands to AMPS represents commands in a simplified subset of JSON. For example, a publish command might look like:

{"c":"publish","t":"test-topic"}{ "id" : 1, "message" : "Hello, World!" }

The command to AMPS, using the amps protocol, can be treated as a JSON document which contains the header information for AMPS -- in this case, a publish to the topic test-topic. The header is followed by the message body, the payload of the command.

While the amps protocol is implemented as a subset of JSON, you can use any message type with the amps protocol. The header for the command will still be JSON, while the body can be in the message type of your choice, as in the sample below, which publishes to an XML topic:

{ "c":"publish","t":"xml-topic"}<example><id>1</id><message>Hello, world!</message></example>

The AMPS client libraries create and parse AMPS headers. For example, the publish method in the AMPS client libraries creates the appropriate header for a publish command based on the provided parameters.

Your applications use the Message and Command interfaces of the AMPS client libraries to work with the AMPS headers. There is no need for your application to parse or serialize the AMPS headers directly.

The AMPS client libraries handle creating and parsing AMPS headers. They do not parse or interpret the payload data on a received Message, instead the payload is returned as a sequence of bytes (or as a string).

There's one exception to this: the JavaScript client can optionally deserialize JSON messages into objects.

spark: the AMPS command-line client

Interacting with AMPS Using Spark

AMPS provides the spark utility as a command line interface to interacting with an AMPS server. spark provides many of the capabilities of the AMPS client libraries through this interface. The utility lets you execute AMPS commands from the command line. spark is a Java application, and requires Java runtime environment version 1.7 or later on the system.

spark is most commonly used for ad hoc testing or simple maintenance tasks. For more complicated tasks or more sophisticated maintenance, 60East recommends using one of the client libraries (such as the AMPS Python Client).

To test spark with the sample configuration, run the following command:

$ $AMPSDIR/bin/spark ping -server localhost:9007 -type json

This command tests connectivity to the AMPS server running at port 9007 on the local system. It confirms that the server is listening on that port using the default protocol for AMPS and accepts JSON messages on that port. The command should produce output like the following:

Successfully connected to tcp://username@localhost:9007/amps/json

You can read more about spark and other useful tools for troubleshooting AMPS in the Utilities chapter of the AMPS User Guide.

It's important to keep in mind that spark only provides basic functionality -- that is, operations that don't require any particular application logic or special handling. This guide uses spark for examples where possible, but some features of AMPS (for example, setting certain headers on messages published to AMPS) are only available through the AMPS client libraries.

Evaluating AMPS on Windows or MacOS

The AMPS server runs on 64-bit Linux operating systems. If you do not have access to a Linux system or a recent version of Windows, 60East recommends creating a Linux virtual machine to host the instance of AMPS. This is a convenient option for development systems and allows you to easily experiment with different AMPS configurations on a dedicated system.

This section provides general information for creating a virtual machine image for use as a local development or evaluation environment.

This section assumes that you are familiar with Linux and the virtualization program you will be working with. It focuses on information specific to AMPS.

Using Windows Subsystem for Linux 2

If your development system is running a recent version of Windows, then Windows Subsystem for Linux 2 is a good option for developing with AMPS. Getting AMPS running is simply a matter of starting a Linux shell, downloading AMPS, and following the directions for Linux.

Notice that Windows Subsystem for Linux 2 does not provide access to some of the functionality that the AMPS server expects: in particular, the AMPS NUMA subsystem may not be able to determine the physical processor layout and may report warnings on startup. Nevertheless, this can be a very good option for doing AMPS evaluation and development on a Windows system.

Creating a Virtual Machine Image

When creating the virtual machine image, 60East recommends the following parameters:

x64 processor
At least 4GB of memory allocated to the virtual machine
Minimum of 120GB drive space (most will be consumed by the operating system image)
At least 2 virtual processors

AMPS itself can run with less memory, processor, and disk capacity than recommended here. However, these settings will typically provide reasonable performance and enough capacity to do basic development work.

Virtual Box Settings

When installing AMPS on Virtual Box, 60East strongly recommends setting the network hardware emulation to use the Paravirtualized network adapter (virtio-net). For recent versions of Linux, performance is dramatically improved (even over the loopback interface) when using this setting.

Choosing a Linux Distribution

AMPS runs well on any Linux distribution that meets the basic requirements. The Ubuntu Linux distribution is a good choice, and is frequently used by both customers and the 60East developers as a development workstation environment. Visit https://www.ubuntu.com/download to download the latest released version of Ubuntu.

Whichever distribution you choose, 60East recommends that you download the .iso file and use that file to install the operating system.

Installing the Linux Distribution

AMPS itself doesn't require anything beyond a basic operating system distribution. For the best experience while you are evaluating and getting to know AMPS, 60East recommends that you choose a profile optimized for software development or desktop use.

Select the following additional packages if your distribution does not already install them:

Python 2.6/2.7 or 3. The utility scripts in the AMPS distribution require Python.
Java runtime environment (1.7 or more recent). The spark command line AMPS client is written in Java, and requires a JRE. This guide assumes that you have a JRE available, and presents examples using spark.
g++, gdb, and your IDE of choice if you will be developing C++ applications with AMPS.
A web browser such as Firefox or Google Chrome

Filesystem Considerations

AMPS is designed and tested to use a Linux-based filesystem such as ext4 or filesystems that provide full native Linux filesystem semantics (for example, using nfs mounted filesystems for testing or archival purposes).

Mounting another type of filesystem (for example, an NTFS volume) in a VM, container, or WSL 2 may cause failures or unexpected results, since that approach may not provide all of the filesystem operations that AMPS uses.

When using a container, VM, or WSL2, make sure that AMPS and the files that AMPS creates are hosted on filesystems that support Linux file operations, in particular, that a process running under the Linux environment can memory map files hosted on that filesystem.

Next Steps

Once you have created the virtual machine image and installed your Linux distribution of choice, you can install and start AMPS as described in Installing AMPS.

Galvanometer and RESTful Statistics

When the Admin interface is configured (as it is in the sample configuration), you can get information about the state of the AMPS instance using either the Galvanometer monitoring tool or the RESTful interface to the AMPS statistics.

The Galvanometer is a Javascript-based application that runs in your browser and provides a visualization of the data provided by the RESTful interface. The Galvanometer also includes a lightweight read-only AMPS client application, based on the Javascript client library.

The RESTFul admin interface is a lightweight view of the statistics database that AMPS maintains.

These two interfaces are available at the following URIs:

In the URIs above, <host> is the host the AMPS instance is running on and <port> is the administration port configured in the configuration file (this is 8085 in the sample configuration).

Monitoring applications typically collect information from the RESTful statistics interface. Interactive or ad hoc monitoring can use either the Galvanometer, or the interface offered by the monitoring application in use locally once the statistics are collected.

AMPS Basics: Subscribe and Publish to Topics

The simplest way to use AMPS is as a low-latency publish and subscribe messaging system. Publish and subscribe messaging is at the heart of AMPS, and all of the other features of AMPS build on this foundation.

In a publish and subscribe messaging system, publishers send messages without necessarily knowing which subscribers will receive the message. This decouples publishers from subscribers for maximum flexibility.

While publishers do not need specific knowledge about the subscribers, publishers are responsible for adding information to the message so that subscribers know which messages are of interest. In publish and subscribe messaging systems, including AMPS, publishers send messages to a specific topic. The topic most often indicates the type of message, and is a way for the subscriber to locate the messages of interest. For example, in an order processing system, a publisher might publish messages to an ORDERS topic. Subscribers that need to receive orders then subscribe to the ORDERS topic, and receive messages that are sent to that topic.

Each message in AMPS is published to a specific topic. The publisher chooses the topic when the message is published, and subscribers can receive messages from that topic.

Unlike many messaging systems, AMPS provides an additional layer of selectivity for subscribers. Rather than receiving every message from a given topic, an AMPS subscriber can use content filtering to receive only the messages that the subscriber needs to process. Content filtering provides several advantages. First, being more selective about the messages delivered to the subscriber makes better use of bandwidth between AMPS and the subscriber, since the subscriber only receives messages that are of interest. Subscriber code is easier to write and more efficient because the subscriber is guaranteed that the messages received have the values requested. Further, because the subscriber chooses which messages to receive, content filtering makes publishers and subscribers less tightly-coupled. A publisher does not need to know what fields are important to a subscriber, or whether a field that was previously unused is now important.

The diagram below shows the basic concept of publish and subscribe messaging:

In the diagram above, there is a Publisher sending AMPS a message to the ORDERS topic. The message being sent contains information on Ticker IBM with a Price of 125. Both of these fields are contained within the message payload itself (i.e., the message content). AMPS routes the message to Subscriber 1 because it is subscribing to all messages on the ORDERS topic. Similarly, AMPS routes the message to Subscriber 2 because it is subscribed to any messages having the Ticker equal to IBM. Subscriber 3 is looking for a different Ticker value and is not sent the message.

Topic Configuration for Basic Pub/Sub

Unlike many messaging systems, AMPS does not require any configuration for simple publish and subscribe. For this basic functionality, there is no need to declare topics in advance. Since there is no need to declare these topics, topics that provide basic publish and subscribe are often referred to as ad hoc topics in AMPS.

It is valid for a publisher to publish to any topic, whether or not that topic has been previously configured. Likewise, it is valid for a subscriber to subscribe to any topic, whether or not a message has been previously published to that topic or whether the topic appears in a configuration file. Features that rely on persisted messages, however, are not available without configuration for a topic.

Every topic in AMPS has a specific message type. Publishers and subscribers don't need to explicitly set the message type when publishing to or subscribing to a topic. Each connection to AMPS specifies the message type to be used for that connection -- either implicitly, by connecting to a port that provides a specific message type, or explicitly (when connecting to a port that can provide multiple message types).

AMPS allows topics that use different message types to have the same name, but considers them to be different topics. Messages published to an XML topic named quotes will not be delivered to subscriptions to a JSON topic named quotes.

Regular Expression Subscriptions

AMPS allows subscribers to provide a regular expression that defines a set of topics rather than a literal topic name. This further decouples publishers and subscribers.

It's important to remember that each message is published to a specific topic. Regular expressions are only applicable to subscriptions: a publisher should not use regular expressions in the topic when publishing messages.

Here's how to use spark to send and receive a message from AMPS. The example assumes that you're using the sample configuration file produced by the AMPS server, and that you are running spark on the same system that AMPS is running on.

First, start a subscriber:

Open a terminal in your Linux environment.
Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to start a subscription:
This command starts a subscription to the JSON topic test.
spark will connect to AMPS, logon using default credentials (the current username and an empty password) and enter the subscription. Unless there are errors, the command will produce no output until a message arrives.
Leave this terminal running. When you publish a message to the test topic, spark will print the message in this terminal.

Next, publish a message to the same topic:

Open a new terminal in your Linux environment.
Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to send a single message to AMPS:
As with the subscriber sample, spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the publish command, spark reads the message from standard input and publishes the message to the JSON topic test. The command produces output similar to the following line (the rate calculation will likely be different):
When the publisher sends the message, the subscriber should receive the message and produce the following output:

Congratulations! You've just sent your first message through AMPS.

Although this example is simple and relies on default behavior, the sample demonstrates some important AMPS concepts:

As mentioned earlier, there is no need to preconfigure simple publish/subscribe topics. Since the default configuration file doesn't specify any settings for the JSON topic test, AMPS knows to treat the topic as a simple pub/sub topic. Also, since the spark commands specify the JSON message type when connecting to the server, the topic is a JSON topic.
For simple publish and subscribe topics, AMPS delivers the message verbatim to the subscriber. AMPS doesn't interpret or normalize the message. In fact, AMPS doesn't even parse the message unless there's a need to. With this configuration and this subscription, there's no need for AMPS to parse the message, so no parsing happens.
The spark program connects to AMPS and logs on to AMPS before sending any commands. All AMPS installations include authentication and entitlement. By default, AMPS loads an authentication and entitlement policy (implemented as an AMPS module) that requires a logon, but accepts any username and password as credentials. This policy is intended for evaluating, testing and development purposes. More information on securing an AMPS instance is available in the User Guide. For this example, the important point is to be aware that an AMPS instance always has a security policy and that policy was at work even in this simple example. The default behavior for spark works with the default policy for AMPS.

Content Filters

In the Basic Sub-Pub example, the subscriber requested all messages on the JSON test topic. AMPS includes expressive, flexible and extensible content filtering that allows subscribers to specify exactly the messages that they want to receive. Content filtering is one of the most useful features of AMPS. When subscribers use content filtering, publishers can be completely independent of subscribers. The publisher does not need to know which parts of a message are important to subscribers. Subscribers can precisely declare the content that they are interested in, so they only receive relevant messages. Publishers do not need to be updated when subscribers add additional criteria or when new subscribers come online.

You can think of content filtering as adding a WHERE clause to the subscription. Like a WHERE clause, AMPS returns only matching messages.

AMPS content filters use a combination of XPath identifiers to locate a value within a message and SQL-92 operators for comparing those values. For example, given a JSON message like:

The following content filters would match the message:

This filter uses the equality operator, =, to compare the /note field in the message with an exact match for the string.

The LIKE operator uses Perl Compatible Regular Expressions to match a field. In this case, the regular expression matches any string that contains world, using case-insensitive matching.

The AMPS BEGINS WITH operator matches any string that begins with the exact sequence of characters provided.

Spark: Subscription with Content Filter

Here's how to use spark to subscribe using a content filter. The example assumes that you're using the sample configuration file produced by the AMPS server and that you are running spark on the same system that AMPS is running on.

First, start a subscriber:

Open a terminal in your Linux environment.
Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to start a subscription:
This command starts a subscription to the JSON topic test. This subscription will only return messages where the filter matches.
spark will connect to AMPS, logon using default credentials (the current username and an empty password) and enter the subscription. Unless there are errors, the command will produce no output until a message arrives.
Leave this terminal running. When you publish a message to the test topic that matches the filter, spark will print the message in this terminal.

Next, publish messages to the subscriber:

Open a new terminal in your Linux environment.
Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to publish a message to AMPS. This message matches the filter:
Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to publish a message to AMPS. This message does not match the filter:
Each time you run spark, it automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With each publish command, spark reads the message from standard input and publishes the message to the JSON topic test. Each of the commands above produces output similar to the following line (the rate calculation will likely be different):
When the publisher sends a message that matches the filter, the subscriber should receive the message and produce the following output:

State of the World (SOW): The Message Database

One of the core features of AMPS is the ability to persist the most recent update for each distinct message published to a topic. To enable this for a topic, you add the topic to the SOW.

You can think of the SOW as a database that maintains a specific set of topics, equivalent to tables. Each distinct message published to that topic is equivalent to updating a row in the table. AMPS allows applications to query the table for the current state of the topic.

SOW topics also provide full support for pub/sub messaging. Applications can use a combination of queries and subscriptions as necessary. AMPS also includes a set of commands that perform an atomic query and subscribe, allowing an application to query a SOW topic and register for updates to the topic in a single operation, without risk of missing messages or receiving duplicates.

The most common uses of SOW topics include:

Quickly loading initial state for an application. For example, an application that tracks open orders can quickly retrieve a snapshot of all of the orders that are currently open, without having to wait for updates to the orders to be published.
Queryable snapshots of data flows. For example, an application that monitors telemetry data may need to quickly determine if any telemetry source has not provided an update within a given period of time. With a SOW topic, the application can run a simple query over the current state of the topic.
NoSQL document stores. SOW topics are frequently used as high-performance key/value stores: an application can choose to explicitly provide a key and store a document in the SOW. Documents can be efficiently retrieved by key, queried over the full content of the document, or any combination. As mentioned above, a consumer can retrieve the document and be automatically notified when the content of the document changes.

SOW topics are also the foundation of many of the more advanced capabilities of AMPS, including out-of-focus tracking, aggregation, and delta messaging. These are described later in this chapter.

When Should I Store a Topic in the SOW?

Storing a topic in the State of the World is most useful when your application needs to use the current state of the data being tracked. Storing a topic in the State of the World can be especially useful if your application would benefit from automatically receiving updates as soon as they are made (described in more detail in the Atomic Query and Subscribe topic).

Below you will find common uses of a SOW topic, which include examples of practical use cases:

An application needs the current state of a record, but does not need to recreate the message flow that created that record:
An order fulfillment system presents a view of all currently pending orders when the application starts up.
An application needs the current state of a record or set of records, even when the topic is high-volume or quickly changing:
A warehouse management application locates the current inventory level for a product.
A taxi dispatch company locates taxis currently within 10 blocks of an event.
An application wants to be able to publish incremental updates to a record:
A customer updates her shipping address. All pending orders for the customer are automatically updated without affecting any other information in the order, and processors working with the orders are notified of the change.
An application wants to receive only the changed fields of a record:
A mobile application displays the status of an order as the order progresses through the stages of validation: the application receives only the identifier for the record and the changed fields.
An application needs the AMPS server to calculate values based on the current values of a record or set of records:
A management console constantly calculates the real-time value of pending orders. The console uses a view, calculated based on data saved in a topic in the SOW.
An application wants to store application state for quick retrieval:
An order processing system publishes statistics on each step of the process: a separate process monitors and aggregates those statistics. The SOW also maintains historical state for the topic so the monitor can easily recreate a snapshot of the state at a point in time and compare day over day status.

Of course, the examples above are just a small sample of the ways the AMPS SOW can be used.

How Does the SOW Work?

AMPS SOW topics persist the most recent update for each message, in the same way that a relational database stores the current state of each record. For performance, AMPS SOW topics store the full content of the message verbatim rather than storing a deserialized or "shredded" version of the message.

Each distinct record in a SOW topic is identified by a SOW key. AMPS treats the SOW key for a SOW topic the same way a relational database uses the primary key for a table: each distinct SOW key value is a unique message.

There are several ways to create a SOW key for a message. Each topic defines one of the following strategies:

Most applications specify that AMPS will calculate a SOW key based on the content of the message. The configuration of the topic specifies the field, or fields, to be used for the key.
A topic can also be configured to require that a publisher provide a SOW key for each message when publishing the message to AMPS. This is less commonly used than determining the key based on the message content, however, since this strategy does not require any explicit configuration, AMPS will default to this strategy for identifying messages if no other strategy is specified.
AMPS also supports the ability for custom SOW key generation logic to be defined in an AMPS module, which will be invoked to generate the SOW key for each message.

Although the SOW key is derived from the content of the message in many cases, the SOW key itself is metadata, distinct from the content of the message. Each record in a SOW topic has a distinct SOW key, which is stored with the record.

For example, the diagram below shows how AMPS computes the SOW key for a topic named ORDERS with a key definition of /orderId. For each publish to the topic, AMPS uses the value of the key fields (in this case, simply /orderId) to compute a SowKey, then uses that SowKey to insert or update the appropriate record.

Configuration

To create a SOW topic, you configure the topic in the SOW section of the AMPS configuration file.

At a minimum, SOW topics require a Name, and the MessageType of the messages to store in the SOW. If the SOW will be persistent, a FileName is required. Most often, SOW topics use AMPS to generate the SOW Key, and one or more Key definition elements are required to specify the fields that AMPS will use for the SOW Key.

For example, the following configuration file fragment specifies a SOW topic named test-sow. The topic stores JSON-format messages, and uses the /id field of incoming messages to that topic to uniquely identify messages. Records in this topic will be both maintained in memory and persisted to a file in the ./sow/ directory, so the contents of the topic will be retained across restarts of the AMPS instance. Notice that the file name specification uses the special format character %n as a placeholder for the topic name and message type.

<SOW>
   <Topic>
        <Name>test-sow</Name>
        <MessageType>json</MessageType>
        <FileName>./sow/%n.sow</FileName>
        <Key>/id</Key>
    </Topic>
</SOW>

The AMPS User Guide and AMPS Configuration Guide contain full details on configuring a SOW topic.

The practical examples later in this section use the configuration above.

Queries

At any point in time, applications can issue SOW queries to retrieve all of the messages that match a given topic and content filter. When a query is executed, AMPS will test each message in the SOW against the content filter specified and all messages matching the filter will be returned to the client. The topic can be a literal topic name or a regular expression pattern. For more information on issuing queries, please see Querying the State of the World (SOW) in the AMPS User Guide.

A SOW query is atomic. Updates that occur while the query is running, or while a client is receiving results, are not returned as part of the query.

Spark: Basic SOW Query Example

Here's how to use spark to query the current state of an AMPS SOW topic.

This example assumes that:

You have configured a topic named test-sow in the AMPS server of message type JSON.
The test-sow topic uses the /id field of the message to calculate the key for the topic.

To retrieve the current state of the topic, an application issues the sow command. Unlike a subscription, which stays active until it is explicitly stopped (or the application disconnects), the sow command provides results for a specific point in time. Once the results are returned, the command is over.

First, publish a message or two to the test-sow topic:

Open a new terminal in your Linux environment.

Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to send a single message to AMPS:

$ echo '{"id":1,"note":"Crank it up with a SOW!"}' | \
  $AMPS_DIR/bin/spark publish -server localhost:9007 \
  -type json -topic test-sow

spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the publish command, spark reads the message from the standard input and publishes the message to the JSON topic test-sow. The command produces output similar to the following line (the rate calculation will likely be different:
```
total messages published: 1 (333.33/s)
```
When the publisher sends the message, AMPS parses the message to determine the value of the Key fields in the message, and then either inserts the message for that key, or overwrites the existing message with that key.
You can publish any number of messages this way. Each distinct id value will create a distinct record in the topic.

Next, retrieve the current contents of the topic:

Open a new terminal in your Linux environment.
Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to retrieve the contents of the topic:
```
$ $AMPS_DIR/bin/spark sow -server localhost:9007 \
  -type json -topic test-sow
```
spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). spark then sends the sow command to AMPS. This command requests the current contents of the test-sow topic. Since the command is finished once the query is complete, spark will exit when the query results are complete.
spark shows the current contents of the topic. Notice that the output is strictly the message data, separated by newline characters. spark does not show any of the metadata for a message.

Atomic Query and Subscribe

When a topic is recorded in the SOW, an application can request the current state of the topic and simultaneously subscribe to updates from the topic. In this case, AMPS first delivers all of the messages that match the query and then provides any update to a record that matches the query. AMPS guarantees that no updates are missed or duplicated between the query and the subscription. As with a simple query, AMPS will test each message currently in the SOW against the content filter specified and all messages matching the filter will be returned to the client. When the query begins, AMPS enters a subscription with the provided filter. After the query completes, AMPS delivers messages from the subscription. In the event that a record is updated while the query is running, AMPS saves the update and delivers it immediately after the query completes.

As with a simple SOW query, the topic can be a literal topic name or a regular expression pattern. For more information on issuing queries, please see Querying the State of the World in the AMPS User Guide.

Here's how to use spark to query the current state of an AMPS SOW topic and subscribe to updates.

This example assumes that:

You have configured a topic named test-sow in the AMPS server of message type JSON.
The test-sow topic uses the /id field of the message to calculate the key for the topic.

To retrieve the current state of the topic and subscribe, an application issues the sow_and_subscribe command. Since the command includes a subscription, the command stays active until it is explicitly stopped (or the application disconnects).

First, publish a message or two to the test-sow topic:

Open a new terminal in your Linux environment.

Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to send a single message to AMPS:

$ echo '{"id":1,"note":"Crank it up with a SOW!"}' | \
  $AMPS_DIR/bin/spark publish -server localhost:9007 \
  -type json -topic test-sow

spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the publish command, spark reads the message from the standard input and publishes the message to the JSON topic test-sow. The command produces output similar to the following line (the rate calculation will likely be different:
```
total messages published: 1 (333.33/s)
```
When the publisher sends the message, AMPS parses the message to determine the value of the Key fields in the message, and then either inserts the message for that key, or overwrites the existing message with that key.
You can publish any number of messages this way. Each distinct id value will create a distinct record in the topic.

Next, retrieve the current contents of the topic:

Open a new terminal in your Linux environment.
Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to retrieve the contents of the topic:
```
$ $AMPS_DIR/bin/spark sow_and_subscribe -server localhost:9007 \
  -type json -topic test-sow
```
spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). spark then sends the sow_and_subscribe command to AMPS. This command requests the current contents of the test-sow topic and creates a subscription to the topic.
spark shows the current contents of the topic. Notice that the output is strictly the message data, separated by newline characters. spark does not show any of the metadata for a message.
spark remains running after the query completes, waiting for new publishes to arrive.

Publish more messages (or updates to the existing messages) to the topic. In the terminal you opened to publish the first messages:

Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to send a message to AMPS:

$ echo '{"id":1,"note":"Crank it up with a SOW!"}' | \
  $AMPS_DIR/bin/spark publish -server localhost:9007 \
  -type json -topic test-sow

Notice that the subscription receives the message.

If you close the subscriber and re-run it, you will see that the second time the subscriber runs, it receives the updated messages in the query and, again, waits for changes to arrive.

Advanced Messaging and the SOW

A SOW topic is the basis for many of the advanced messaging features in AMPS. While not all of these features are discussed in detail in this introduction, many features of AMPS are made possible because AMPS can retain the current state of each unique message.

The advanced messaging features that the SOW enables include:

Views and aggregations over topics (including joins between topics)
Publishing incremental updates to a message (called delta publishing in AMPS)
Receiving incremental updates to a message (called delta subscription in AMPS)
Determining when a message no longer matches a filter (called out-of-focus notification in AMPS)
Providing a snapshot of an update to a rapidly changing record at regular intervals, rather than providing every update (called conflation in AMPS)

These features can greatly simplify the processing an application needs to perform, making it easier to develop applications and increasing application performance. However, for a messaging system to provide these features, whenever a message arrives, the messaging system must have access to both the current message and the previous, saved state of the message. SOW topics provide that access for AMPS, and enable the advanced messaging features.

Record and Replay Messages with the AMPS Transaction Log

AMPS provides the ability to record topics and replay those topics at a later time. This capability is called the transaction log.

The AMPS transaction log fully supports topic and content filtering. You configure the transaction log to keep a journal of incoming messages for one or more topics, and then you can replay those messages, in order, from any point in time. With the (optional) high-availability features in the AMPS client libraries, this also provides a way to ensure that in case of failure, an application can resume the subscription without missing messages or receiving duplicate messages.

The AMPS transaction log is most often used for:

Fully Resumable Subscriptions - With the transaction log, you can ensure that an application receives all messages of interest, even in the event of a failure.
Backtesting and Audit - The transaction log allows you to replay the precise messages published, in order across all topics in the instance, at a configurable maximum rate. You can use this feature to easily audit the flow of messages, perform backtesting, or replay a sequence of events.
Capacity Planning and Stress Testing - Since the transaction log allows you to set the maximum replay rate to be a multiple of the original publish rate, you can use the transaction log to measure the load on a system at various rates, and measure the capacity of the system and the ability of your application to correctly handle increased volumes.

The transaction log is also the source of messages for:

The AMPS transaction log can typically record messages at the maximum throughput of the underlying storage device.

60East recommends storing the transaction log on a device that supports fast sequential writes, and ensuring that the device has the speed and capacity necessary to support the expected throughput. (The Operation and Deployment chapter of the AMPS User Guide includes guidance on capacity planning.)

How Does the Transaction Log Work?

The AMPS transaction log records messages that are published to the topics specified in the configuration file. Every publish is stored, in the order in which the AMPS instance processed the message.

For ease of maintenance, the AMPS server writes multiple sequential files, called journal files, for the transaction log rather than writing a single large file. The journals contain the full content of each message, as well as information on the topic, the publisher, the time at which the message was processed, and so on. The AMPS configuration sets the maximum size of a journal file. When a file reaches that size, AMPS begins writing to the next file.

When AMPS records a message into the transaction log, it assigns each message a bookmark. The bookmark identifies a single message, that is, a specific point in the transaction log of the local instance.

The AMPS server does not modify the contents of journal files. Once a message is written to a journal file, it is part of the transaction log and is considered to be immutable.

Since journal files form part of the persistent state of the server, those files should not be modified or removed while the AMPS process is running except by the AMPS process itself. The AMPS server provides a set of maintenance actions for managing journal files (see the AMPS User Guide for details).

Configuration

To create a Transaction Log, you add the TransactionLog configuration element to your AMPS configuration file. You then specify a location for AMPS to create journal files, and specify the topics that you want recorded in the file.

The following configuration is the minimum configuration to create a transaction log and record a single topic:

<TransactionLog>
    <JournalDirectory>./journals</JournalDirectory>
    <Topic>
       <Name>some-topic</Name>
       <MessageType>json</MessageType>
    </Topic>
 </TransactionLog>

The configuration above writes journal files to the journals directory underneath the AMPS server's current working directory. The configuration records a single topic, some-topic, of message type json to the transaction log. The Name option of the Topic configuration element can be either a literal topic name, or a regular expression that matches the names of a set of topics to be recorded.

Although this configuration works perfectly well, AMPS provides a number of additional options that are useful for managing transaction logs in production. AMPS also provides a set of administrative actions for setting the archival and retention policy for journal files.

A more complete configuration might include options along the following lines:

<TransactionLog>
    <JournalDirectory>./journals</JournalDirectory>
    <JournalArchiveDirectory>/mnt/high-capacity/journals</JournalArchiveDirectory>
    <JournalSize>100MB</JournalSize>
    <Topic>
       <Name>^/orders</Name>
       <MessageType>json</MessageType>
    </Topic>
    <Topic>
       <Name>^/status/customer</Name>
       <MessageType>fix</MessageType>
    </Topic>
    <Topic>
       <Name>/audit/events</Name>
       <MessageType>binary</MessageType>
    </Topic>
</TransactionLog>

<Actions>
   <Action>
     <On>
         <Module>amps-action-on-schedule</Module>
         <Options>
           <Every>21:30</Every>
           <Name>Daily Journal Maintenance Plan</Name>
         </Options>
     </On>
     <Do>
         <Module>amps-action-do-archive-journals</Module>
         <Options>
           <Age>3d</Age>
         </Options>
     </Do>
     <Do>
         <Module>amps-action-do-remove-journals</Module>
         <Options>
           <Age>7d</Age>
         </Options>
     </Do>
   </Action>
</Actions>

In this configuration, journals are created in the journals directory underneath the AMPS server's current working directory, as before. This configuration records two sets of topics and one individual topic. Taking these in the order in which they appear in the configuration file, this instance of AMPS will record:

Any topic that begins with /orders and is of message type JSON.
Any topic that begins with /status/customer and is of message type FIX.
The topic /audit/events of message type binary.

The sample above also includes a basic journal maintenance configuration. Configuring journal maintenance is strongly recommended for any instance of AMPS that will be running on a regular basis.

For this AMPS installation, the size of the journal files has been reduced from the default 1GB size to a 100MB size. This typically indicates that the instance stores less than 1GB of messages during a day, so the default journal size would include multiple days worth of messages.

This configuration specifies a two-step maintenance process:

After 3 days, journal files will be archived to the /mnt/high-capacity/journals directory -- the directory specified in the JournalArchiveDirectory parameter of the transaction log. These journal files remain a part of the transaction log but are moved to a different location (typically on a different device with higher capacity).
After 7 days, journal files will be deleted.

AMPS will run this maintenance plan every day at 21:30 (9:30 PM) local time.

When journal files are moved to the archive directory, they continue to be part of the transaction log, but they do not have to be on the same device as the JournalDirectory. Most often, a production installation of AMPS will keep journal files that are very active on fast storage and keep a longer period of history on storage that is higher capacity and lower cost. Since these devices typically also have lower throughput, these devices are best for files that must still be retained but are infrequently used.

Full details on these options are available in the AMPS User Guide and the AMPS Configuration Guide.

Message Queues

AMPS includes high performance queuing built on the AMPS messaging engine and the transaction log. AMPS queues combine elements of classic message queuing with the advanced messaging features of AMPS, including content filtering, aggregation and projection, and so on.

AMPS queues help you easily solve some common messaging problems:

Ensuring that a message is only processed once.
Distributing tasks across workers in a fair manner.
Ensuring that a message that has been delivered is processed.
Ensuring that when a worker fails to process a message, that message is re-delivered.

These uses of messaging require different behavior than the scenarios discussed in the section on Subscribing and Publishing to Topics. For basic subscribe and publish, each message is delivered to any number of subscribers. With queues, each message is fully processed by only one subscriber.

While it's possible to create applications with these properties by using the other features of AMPS, message queues provide these functions built into the AMPS server for additional performance, simple administration, and ease of development.

AMPS queues also allow you to:

Replicate messages between AMPS instances while preserving delivery guarantees.
Create views and aggregates based on the current contents of a queue.
Filter messages with specific content into specific queues.
Provide a subscriber only messages that contain specific content.
Provide a single published message to multiple queues.
Aggregate multiple topics into a single queue.
Provide content aware entitlement for security.
Provide prioritization of messages within a queue, so higher-priority messages are processed first.
Provide a synchronization point that guarantees that all messages prior to that point have been processed before messages after that point are delivered.

How Do Queues Work?

When an application needs to receive messages, there is little difference between subscribing to a queue and subscribing to a sub/pub topic. Both delivery models use the subscribe command, and both delivery models can provide a filter to specify messages of interest. Both types of topics provide the same message objects in the AMPS Client interfaces.

Once a message is received from a queue, however, the application must let AMPS know when the message is successfully processed. This acknowledgment lets AMPS know that the application is finished with the message, and has capacity to receive another message. In addition, if the queue is configured to retry messages if an application fails to process the message (at-least-once delivery), acknowledging the message indicates to AMPS that the message has been processed successfully and can be removed from the queue.

Within AMPS, the server maintains an in-memory list of all of the messages currently available for delivery in a given queue and a list of all of the messages awaiting acknowledgment from subscribers. The messages themselves are stored in the transaction log for the instance. When a message has been successfully processed, the acknowledgment for that message is also stored in the transaction log.

There is no separate storage required for a queue, since messages are recorded in the transaction log. Likewise, even when a message is removed from the queue, AMPS maintains a persisted record of that message and the acknowledgment in the transaction log. Given that the transaction log contains a full record of messages and acknowledgments, AMPS queues are persistent across server restarts, and can be replicated to other instances. (For details on replicating queues, see the AMPS User Guide.)

Keeping the delivery state -- that is, the queue itself -- independent of the topic in the transaction log has several other advantages. Since the set of messages in the queue is maintained separately from the physical storage for those messages, a queue in AMPS can hold messages from any number of underlying topics. Content filtering can be applied to the queue to selectively add messages to the queue: in fact, the same topic can easily be split into independent queues using content filtering. Messages to a single topic can also be included in multiple, independent queues (for example, one queue for immediate processing, and another queue for end-of-day auditing and reconciliation).

AMPS includes the ability for a given consumer to declare the capacity of that consumer, using the max_backlog option on a queue subscription. This option declares the number of messages that the consumer is willing to have delivered at a given time. Using this option can improve throughput, since AMPS can ensure that a consumer is never idle waiting for a new message. This also helps AMPS to balance message delivery across consumers in the most efficient manner, as measured by the current available capacity of each consumer. For example, a consumer on a small VM might take 200 milliseconds, on average, to process a message, and might declare a max_backlog of 2. A consumer running on a larger physical server, in contrast, might take 50 ms on average to process a message, and might therefore declare a max_backlog of 8 or more. The maximum allowed backlog for a subscriber is configured for each queue, so that queues that hold large units of work can set a smaller maximum value than queues that provide smaller tasks.

When Should I Use Queues?

Queues are intended to guarantee delivery of each message to a single consumer that processes the messages. Use queues when the problem you are solving requires a message to be processed once. When you need to distribute messages to a large number of consumers, use the AMPS pub/sub delivery model.

For example, a queue is a natural fit for a system that allocates work, such as a system that runs software builds or that executes financial transactions. A system that provides notifications to a large number of systems (for example, a system that distributes bids to sellers or a system that communicates status to a user interface) is a more natural fit for the pub/sub delivery model.

Queues are often used to solve problems like:

Guaranteeing that a given set of work is distributed fairly across a set of workers, while each unit of work is only performed once:
A system that performs CPU-intensive calculations needs to ensure that any time a request comes into the system, it is serviced by the next available worker.
A distributed compute grid has workers that vary widely in capacity. Each worker declares its capacity to AMPS. Workers with more capacity free receive work before workers with less free capacity, improving overall throughput for the compute grid.
Guaranteeing that a specific message is fully processed once, regardless of the number of subscribers:
A system that processes refunds enters the refund orders into a queue. Each message is delivered to one, and only one, worker. If the worker successfully processes the message, the worker removes the message from the queue. If the worker fails, AMPS automatically delivers the message to another worker, ensuring the message is processed.

The AMPS client libraries, starting in version 5.0, are queue-aware and contain features to make it easier to work with queues and create the application behavior that you need. See the Developer Guide for the client library of your choice for details on how to use these features.

For further details on message queues and how they function, the chapter on Message Queues in the AMPS User Guide presents a more complete discussion.

Configuration

As described above, both the topic that holds the messages for the queue and the queue topic itself must be recorded in the AMPS transaction log.

In addition, the queue itself must be declared in the SOW element of the AMPS configuration.

For example, the configuration below records the topics Work and WorkToDo in the AMPS transaction log:

<TransactionLog>
  <JournalDirectory>/fast-storage/journals</JournalDirectory>
  <Topic>
      <Name>Work</Name>
      <MessageType>json</MessageType>
  </Topic>
  <Topic>
      <Name>WorkToDo</Name>
      <MessageType>json</MessageType>
  </Topic>
</TransactionLog>

With these topics added to the transaction log, we can configure a WorkToDo queue that provides queuing for the messages in the Work topic.

<SOW>
  <Queue>

     <Name>WorkToDo</Name>
     <MessageType>json</MessageType>
     <Semantics>at-least-once</Semantics>
     <UnderlyingTopic>Work</UnderlyingTopic>

     <!-- recommended: set a MaxPerSubscriptionBacklog
          to allow more efficient delivery -->

     <MaxPerSubscriptionBacklog>10</MaxPerSubscriptionBacklog>


     <!-- recommended: set lifetime limit
          and delivery limits -->

     <Expiration>12h</Expiration>
     <MaxDeliveries>5</MaxDeliveries>
     <LeasePeriod>10m</LeasePeriod>

   </Queue>
</SOW>

This declares a queue named WorkToDo that provides queuing for the messages in the Work topic. By setting the semantics to at-least-once, the topic is configured to redeliver a queue message to another subscriber in the event that a subscriber fails to process that message successfully.

This example also provides an example of recommended configuration options to help manage message lifetime in cases where a message cannot be processed successfully, or where no consumers are available to process the message.

For this queue, we provide the following options:

Expiration specifies that a message will be available for, at most, 12 hours from the time it enters the queue.
MaxDeliveries specifies that a given message can be delivered from the queue at most 5 times: after that, the message will be considered to be unable to be processed and removed from the queue.
LeasePeriod specifies that a consumer has 10 minutes from the time the message is sent to acknowledge the message or the message will be automatically returned to the queue.

AMPS also provides a mechanism for publishing expired messages to a dead-letter queue, as well as a wide variety of options for controlling delivery.

Scenario and Feature Reference

AMPS offers a wide array of messaging features to solve a variety of messaging scenarios. This section presents some basic mappings between common messaging scenarios and the AMPS features that support those scenarios. Of course, this list is just a sampling of the types of applications that use AMPS.

Scenario

AMPS Feature(s)

Simple, low-latency publish and subscribe (many to many messaging) with no need to persist messages.

Ad hoc Publish and Subscribe

Publish and subscribe with a replayable audit trail.

Transaction Log and Bookmark Subscription

Snapshot of the current state of a set of messages (for example, graphing the elapsed time for all pending orders).

State of the World (SOW)

Creating a view server that aggregates information about a high-velocity data feed for reporting.

State of the World (SOW)

Views and Aggregation

Transaction Log and Replication

Snapshot of the current state of a set of messages followed by updates to those messages (for example, showing the current status of a set of orders when a UI starts and then showing real-time updates to those messages).

State of the World (SOW)

SOW and Subscribe from client application

Ensuring that a given message is processed once, by a single subscriber (for example, a workload distribution system).

Message Queues (Queues use the Transaction Log)

Replaying messages from a point in time.

Transaction Log and Bookmark Subscription

Transforming messages as they are published to AMPS.

State of the World (SOW) and Enrichment

Producing aggregate data for a stream of messages.

State of the World (SOW) and Views or Aggregated Subscriptions

Coordinating work across a set of independent workers who are each assigned discrete tasks.

Message Queues

Dividing work among a set of workers who each update a portion of a record.

State of the World (SOW) and Delta Publish

Providing highly available messaging with multiple servers providing failover.

Transaction Log and Replication

The scenarios above describe just a few of the more common scenarios in which AMPS is used. For messaging scenarios that aren't described above, contact 60East at http://support.crankuptheamps.com/ for advice and guidance.

Recovery Strategies

The AMPS server and the AMPS client libraries provide various options for recovering and resuming subscriptions. Use this cross-reference to choose the recovery strategy that best matches the needs of your application.

Getting Support

Technical Support and Assistance

At 60East, the most important part of what we do is helping people deploy systems that utilize AMPS and supporting them in maintaining the ongoing and intended operation of those systems. Considering that AMPS is often used in essential systems that push the limits of hardware, network and storage capacity, we know that support is essential to help you build, deploy and maintain the kinds of cutting-edge applications that we built AMPS to handle.

During the evaluation and development stages, we encourage you to share details with us about what you are building. This way, we can provide assistance with the design and architecture process. Once your application goes into production, use 60East support to help diagnose and correct issues that fall outside of the normal operation of the application.

The level of support you have available is dependent on your support agreement. For an outline of your specific support policies, please see your 60East Technologies License Agreement. Support contracts can be purchased through your 60East Technologies account representative.

Support Steps

You can save time if you complete the following steps before you contact 60East Technologies Support:

Check the documentation
Isolate the problem
If you require Support Services, please isolate the problem to the smallest test case possible. Capture erroneous output into a text file along with the commands used to generate the errors.
Collect your information
- Your product version number.
- Your operating system and its kernel version number.
- The expected behavior, observed behavior and all input used to reproduce the problem.
- Submit your request.

The AMPS version number used when reporting your product version number follows a format listed below. The version number is composed of the following:

Contacting 60East Technologies Support

Support is offered through the United States:

Other support options (such as support via phone, dedicated engineers, and so on) may be available, depending on the terms of your support agreement.

Advanced Topics

Next Steps

Learning More

Now that you understand the basics of how AMPS works, you have two potential paths forward in your usage of the product:

The following sections provide more information about each of these paths and also briefly describe some use cases for AMPS.

Operation and Deployment

When preparing to deploy AMPS, consult the Deployment Checklist whitepaper, available on the 60East website.

Application Development

Each language-specific Development Guide explains how to install, configure, and develop applications that use AMPS. In order to develop applications using an AMPS client, you must understand the basic concepts of AMPS, such as topics, subscriptions, messages and SOW.

AMPS Evaluation Guide

Thank you for choosing the Advanced Message Processing System (AMPS) from 60East for evaluation.

This guide provides information to help you evaluate AMPS for your application.

This guide covers the following topics:

Introduction

Welcome to 60East Technologies AMPS

Thank you for choosing to evaluate the Advanced Message Processing System (AMPS) from 60East Technologies!

AMPS is designed to make it easy to develop and deploy data-intensive applications with demanding requirements for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems.

AMPS is more than a publish and subscribe system. It is a feature-rich platform that enables you to easily build data intensive applications that provide previously unattainable low latency and high performance. AMPS combines a set of capabilities that cut across traditional component divisions. 60East designed the capabilities based on the needs of some of the most demanding data-intensive applications on the planet, and engineered the capabilities to work together seamlessly and provide the kind of performance and latency that those applications demand.

AMPS isn't a traditional database or messaging product. This guide presents a brief introduction to AMPS and contains information on evaluating AMPS.

Documentation Roadmap

This guide is designed to be use alongside other parts of the AMPS documentation.

Guide

Purpose

Overview of AMPS features and capabilities, intended as a starting point for learning AMPS.

Description of AMPS server-side features.

Guide to AMPS configuration.

The AMPS client distributions also contain guides discussing the programming model for applications and how to use the client libraries effectively.

In addition, 60East maintains an FAQ on the 60East support site at:

Site

Purpose

Frequently asked questions about the AMPS product.

For evaluation purposes, 60East recommends starting with the Introduction to AMPS for an overview of the features of AMPS, including a cross-reference as to which features are most commonly used together in particular application scenarios.

AMPS Software Requirements

The AMPS server is supported on the following platforms:

Linux 64-bit (2.6 kernel or later) on x86_64 compatible processors

While 2.6 is the minimum kernel version supported, AMPS will select the most efficient mechanisms available to it and thus reaps greater benefit from more recent kernel and CPU versions.

The AMPS distribution contains all of the supporting libraries and dependencies needed to run on a typical Linux server installation: no further software is required.

Some utilities provided with the AMPS server have additional dependencies. These utilities are not required to run the server, but can make it easier to troubleshoot and test on the system that hosts the AMPS instance:

spark, a basic command line client that supports a subset of AMPS functionality, requires Java 1.7 or later.
The utilities for inspecting AMPS files (amps_sow_dump, amps_clients_ack_dump, and so on) require a Python installation.
amps-grep requires a Python installation.
amps-sqlite3 requires a Python installation and the sqlite3 package for your distribution (often, but not always, installed by default).

Obtaining Evaluation Licenses and the AMPS Server

For existing customers, evaluation and development licenses are typically covered in the existing licensing agreement. Contact the team that manages the license agreement or 60East for details.

For new customers, you can register for your evaluation, obtain an evaluation license, and receive instructions for downloading AMPS from the Evaluate AMPS page of the 60East Website.

The registration process covers the terms of the evaluation license. You can use the support website, as described in Obtaining Evaluation Support, for any questions that arise during your evaluation, including both licensing questions and technical questions.

Obtaining Evaluation Support

For existing customers, an outline of your specific support benefits and policies is available in your 60East Technologies License Agreement. Support contracts can be purchased through your 60East Technologies account representative. Existing customers will also typically already have a non-disclosure agreement in place, allowing development teams to discuss the details of their applications with 60East for the purposes of troubleshooting issues or answering questions about AMPS.

For new customers, contact support (as described in the following sections) for any issues that emerge during your evaluation. Support for evaluation purposes is typically available for new customers without a support contract. If your company requires a non-disclosure agreement with 60East before discussing technical information, please contact support to begin the process of creating and executing an agreement.

Support Steps

You can save time if you complete the following steps before you contact 60East Technologies Support:

Check the documentation
The problem may already be solved and documented in the User Guide or Configuration Guide for the product. 60East Technologies also provides answers to frequently asked support questions on the support web site at http://support.crankuptheamps.com.
Isolate the problem
If you require Support Services, please isolate the problem to the smallest test case possible. Capture erroneous output into a text file along with the commands used to generate the errors.
Collect your information
- Your product version number.
- Your operating system and its kernel version number.
- The expected behavior, observed behavior and all input used to reproduce the problem.
- Submit your request.
- If you have a minidump file, be sure to include that in your email to crash@crankuptheamps.com.

The AMPS version number used when reporting your product version number follows a format listed below. The version number is composed of the following:

MAJOR.MINOR.FEATURE.HOTFIX.TIMESTAMP.TAG

AMPS Versioning and Certification

Each AMPS version number component has the following breakdown:

Component

Description

Minimum Verification

MAJOR

Increments when there are any backward-incompatible changes in functionality, file formats, client network formats or configuration; or when deprecated functionality is removed.

May introduce major new functionality or include internal improvements that introduce major behavioral changes.

Megacert

MINOR

Increments when functionality is added in a backwards-compatible way, or when functionality is deprecated.

May include internal improvements, including internal improvements that introduce minor behavioral changes or changes to network formats used only by the AMPS server (such as replication).

Megacert

FEATURE

Increments for previews of new features.

May introduce behavioral changes to fix incorrect behavior, enable new functionality or to enhance performance.

May include internal enhancements that do not introduce behavioral changes.

Note: A feature level of 0 indicates a long-term stable release. A feature level above zero indicates the current feature level (a preview of the next long-term stable release).

Kilocert

HOTFIX

A release for a critical defect impacting a customer. A hotfix release is designed to be 100% compatible with the release it fixes (that is, a release with same MAJOR.MINOR.FEATURE version).

May introduce behavioral changes to fix incorrect behavior. May document previously undocumented features or extend surface area to improve usability for existing features.

Cert

TIMESTAMP

Proprietary build timestamp.

(does not affect verification level)

TAG

Identifier that corresponds to precise code used in the release.

(does not affect verification level)

The certification levels are defined in the following table. Notice that, in all cases, 60East will certify at a higher level if time permits or if a change involves a critical part of AMPS (such as replication or internal utility classes that are widely used).

Certification Level

Description

Time to Certify

Megacert

Performance and long-haul testing.

Full regression suite and stress-testing suite, including replication testing and application scenario tests.

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

less than 2 weeks

Kilocert

Full regression suite and stress-testing suite, including replication testing and application scenario tests.

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

less than 1 week

Cert

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

Replication testing suite if release affects replication code.

4 hours

Contacting 60East Technologies Support

Please contact 60East Technologies Support Services according to the terms of 60East Technologies License Agreement (whether that is an existing license or evaluation license).

Support is offered through the United States:

Web:

E-mail:

Support:

Other support options may be available for existing customers, depending on the terms of the support agreement.

Evaluation and Development with AMPS

AMPS runs on any 64-bit Linux system. For best performance in a development environment, 60East recommends that the system have a minimum of 4GB of memory available.

For basic functional evaluation and development, AMPS runs well in a virtual machine, in a container, or in a WSL2 shell on Windows, as well as on a Linux host.

Product Overview

AMPS is designed to help you quickly and easily develop and deploy data-intensive applications with demanding requirements for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems.

AMPS is based on an incredibly fast messaging engine that supports multiple messaging paradigms, as well as providing persistent current value caching (effectively, an integrated database), content filtering and continuous query, historical replay, aggregation and analytics, message enrichment, focus tracking, partial updates and change tracking, and more.

Furthermore, AMPS is designed and engineered specifically for next generation computing environments. The architecture, design and implementation of AMPS allows the exploitation of parallelism inherent in emerging multi-socket, multi-core commodity systems and the low-latency, high-bandwidth of 10Gb Ethernet and faster networks. AMPS is designed to detect and take advantage of the capabilities of the hardware of the system on which it runs.

AMPS was designed to improve performance and reduce latency in real-world messaging deployments by focusing on the entire lifetime of a message from the message's origin to the time at which a subscriber takes action on the message. AMPS considers the full message lifetime, rather than just the "in flight" time, and allows you to optimize your applications to conserve network bandwidth and subscriber CPU utilization -- typically the first elements of a system to reach the saturation point in real messaging systems.

Understanding AMPS Features and Scenarios

For example, to distribute work across a set of independent processors, you would use AMPS message queues, whereas if your application requires a content-aware last value cache, you would use a Topic in the AMPS State of the World.

Evaluation Process Outline

60East provides access to technical support during the evaluation process.

To prepare to evaluate AMPS, 60East recommends the following process:

Engage 60East support with a description of the evaluation goals, and to put in place any agreements necessary to make evaluation go more smoothly (such as mutual non-disclosure agreements).
Define the detailed goals of the feasibility phase of the evaluation. Typically, these break down into:
- Functional Capability - This represents what the evaluation project needs to be able to do. (For example: accurately receive NFVIX messages and deliver them to the appropriate subscriber or subscribers while maintaining the ability to replay 30 days of history at any point.)
- Performance Goals - This represents the service level for the evaluation. (For example: Maximum latency to reach client processing of 250ms for current messages, no more than 1s to first message for beginning a replay at an arbitrary depth in history.)
- Capacity Goals - This represents the total volume of work being evaluated. (For example: The system needs to reach capability and performance goals while processing 10,000 messages per second ingestion.)
Develop initial design and testing plan. During this process, teams use the AMPS documentation to understand how to use AMPS to meet the goals of the evaluation. Teams engage 60East support as necessary to resolve any questions that emerge or get advice on tradeoffs and options to achieve the evaluation goals.
Once a design and testing plan is complete, review the design and testing plan with the 60East engineering team, adjusting as necessary.
Implement the design and tests. If questions or issues emerge, consult with 60East support to resolve the issues.
Test and review the results with 60East.
Evaluate the deployment and maintenance phase of the evaluation. Typically, this involves:
- Operations and Performance at Scale - Evaluate the application in a production-like environment at scales at or near production volumes.
- Maintenance Goals - Develop and test the maintenance, support, and upgrade plan as described in the 60East deployment checklist.
Follow up on any open issues and complete the evaluation.

Tips on Measuring Performance

One of the most common questions during evaluation of AMPS is how best to measure and quantify the overall performance of the application that uses AMPS.

There are several factors that are included in any meaningful discussion of performance:

Performance of the Underlying Hardware

AMPS is designed to use the underlying hardware as efficiently as possible, and does not suffer from artificial bottlenecks that limit performance.

The implications of this, though, are that the performance available from an installation of AMPS depends on the capacity of the underlying hardware.

In particular, pay attention to:

Storage device speed and bandwidth (for applications that persist data)
Memory speed
Network speed and capacity

Often, you can come up with the theoretical maximum performance of a system based on the underlying hardware. For example, storage that can only write 80MB/s would be unsuitable for a system that needs to retain messages that arrive at a sustained rate of 100MB/s.

Likewise, a system with 64GB of memory would see reduced performance for lookups on a 128GB data set, so benchmarking an application that retains 128GB of active data on a system with 64GB of memory will produce very different results than the same benchmark run on a system with 256GB of memory.

Operating System Performance

Most Linux distributions and installations are, by default, tuned for interactive desktop usage. This is convenient when developing applications, but can produce reduced performance as compared with a well-tuned server.

Realistic Data Complexity and Volumes

AMPS is designed for high-throughput, low latency messaging. This means that AMPS typically performs better with a realistic workload than with a very small number of messages. It is typically not useful to run a performance test with a small number of messages and then attempt to extrapolate the performance at scale.

As an example, imagine a test that deploys a Docker container from scratch, starts AMPS, sends and receives a single message, and then shuts down the container and uses the elapsed time from the start of the test to the time that the container shuts down as the "single message throughput time". That number will be orders of magnitude slower than the actual time that it takes for AMPS to deliver the message: most of the time in the test is consumed by overhead unrelated to delivering an individual message.

Although it would be unlikely that anyone would create a test with as much overhead as the scenario above, it is not uncommon to have hidden overhead in a test. Likewise, there are often "economies of scale" that the system (including AMPS) can take advantage of production-level usage that is not available at unrealistically low messaging rates.

A realistic test should avoid measuring overhead that would not be present in a production environment. If the requirement of the application is to have latency within a certain threshold when AMPS is processing messages at a sustained rate from a dozen publishers, the results of a test will be more accurate the more closely the test approximates that scenario.

In particular, as much as possible, build your tests to:

Have similar use of connections as the production application. If a given application will have multiple subscribers in production, do not use a single subscriber in performance testing and assume that parallel processing offers no benefit.
Have similar message volumes as the production application. Do not assume that you can use a rate of 100 messages per second to predict latency or processing time of an application that will need to process 1000 or 10000 messages per second.
Have similar message sizes as the production application. Do not assume that a 1MB message size in test will have the same performance characteristics as a 250KB message (or a 5MB message) in production.

Compare Equivalent Work

When benchmarking different implementation ideas, compare equivalent work. In some cases, having the AMPS server do additional work does not add noticeable latency due to the efficiencies (and parallel processing) in AMPS. In other cases, having the server do additional work may add more latency. In either case, accurately measuring throughput and latency must measure the cost of doing equivalent work in the application.

For example, if your application will use AMPS delta subscriptions (that is, have AMPS automatically calculate the differences between an update to a message and the current state of the message), rather than comparing throughput for a subscription that uses that option to a subscription that does not use that option based solely on when messages arrive at the client, compare the differences between having AMPS calculate the difference versus having the application calculate the difference, and evaluate this difference based on the total throughput numbers for a realistic number of subscribers.

Use AMPS Capabilities

AMPS is carefully designed to include functionality that reduces end-to-end throughput in the system, and to provide server-side capability where performing those functions on the server improves overall performance.

When evaluating performance, take advantage of those capabilities to get an accurate measure of how an application would perform in a production environment.

For example, if your application needs to append a calculated field to every published message, use message enrichment (or the AMPS delta publish functionality) rather than a process that extracts, rewrites, and updates the full message. Likewise, if your application will only process a subset of messages to the topic, use AMPS content filtering to ensure that AMPS only provides actionable messages rather than oversubscribing and discarding messages in your application.

If you have questions on whether your application is using the built-in capabilities of AMPS in the most effective way possible, contact 60East support for an engineer to review your design.

Next Steps

Once you have done a basic evaluation of AMPS, there are two typical paths forward in usage of the product:

On one path, you may want to learn how to configure, deploy, and administer an instance of AMPS. For this path, see the AMPS User Guide, which provides complete information for system administrators who are responsible for the deployment, availability and management of data to other users.
Alternatively, you may need to develop an application to work with AMPS, using one of the Developer Guides for Java, Python, C++, or C#. For this path, download one of the client distributions from the AMPS developer page at https://www.crankuptheamps.com/develop/. The client distributions include a set of examples and an AMPS server configuration that works with the examples.

The following sections provide more information about each of these paths and also briefly describes some use cases for AMPS.

Operation and Deployment

In preparing to deploy your instance of AMPS, you must size your host environment according to multiple dimensions: memory, storage, CPU, and network. The Operation and Deployment chapter in the AMPS User Guide provides guidelines and best practices for configuring the host environment. The chapter also specifies recommended settings for running AMPS on a Linux operating system.

Application Development

You will also need an installed and running AMPS server to use the product. Typically, a team will use a server licensed for evaluation during the initial stages of development, then transition to a full license as the evaluation completes and the team prepares to deploy the application.

AMPS User Guide

Welcome to the Advanced Message Processing System (AMPS) from 60East Technologies.

AMPS is a feature-rich message processing system that delivers previously unattainable low-latency and high-throughput performance to users. AMPS provides both publish-and-subscribe messaging and high-performance message queuing. AMPS also provides current value caching / message database functionality, analysis and aggregation.

Introduction

Thank you for choosing the Advanced Message Processing System (AMPS) from 60East Technologies. AMPS is a feature-rich message processing system that delivers previously unattainable low-latency and high-throughput performance to users. AMPS provides both publish-and-subscribe messaging and high-performance message queuing.

AMPS is designed to help you quickly and easily develop and deploy data-intensive applications with demanding requirements for low latency and high performance.

AMPS combines aspects of a traditional message bus, message queue, database, view server, analytics and event processing engine. The features that AMPS provides are designed to be easy to use, to work well together, and to provide high performance.

Documentation Resource Overview

The 60East documentation is intended to be used with a working (development) environment of AMPS available so that you can quickly explore the concepts discussed.

60East recommends starting with the Introduction to AMPS to become familiar with AMPS, and then reading the sections of the AMPS User Guide and AMPS Configuration Guide for the features that your application will use.

The Introduction to AMPS provides an overall introduction to AMPS, including information on setting up a development environment, the basic concepts and features of AMPS, and general advice on which features combine effectively for specific scenarios. The 60East documentation is intended to be used with a working (development) environment of AMPS available so that you can quickly explore the concepts discussed.
The AMPS Evaluation Guide provides advice on evaluating AMPS, included a suggested evaluation process, tips on monitoring and measuring performance in an evaluation environment, and information on how to effectively partner with 60East on an evaluation of AMPS.
The AMPS User Guide -- this guide -- provides a complete overview of the features of AMPS and how to deploy and administer an instance of AMPS.
The AMPS Configuration Guide describes the AMPS configuration file and the options available to specify the behavior of an instance of AMPS.
The Deployment Checklist is a short document providing recommendations for deploying AMPS into a shared environment, whether that environment will be used for production, test, or development.

These guides cover the general features of AMPS. This site provides additional guides, such as guides for developing applications with AMPS, a guide to the statistics available for monitoring, and so on.

Resources for Developers

For developers, becoming familiar with the Developer Guide for the AMPS Client library that you will be using is also recommended. The developer page on the 60East web site contains reference material and links to download client libraries. Full source code (including example applications) is available for all client libraries. For many client libraries, 60East also includes pre-built binaries and makes binary distributions available through popular package management sites. Notice, however, that the pre-built distributions do not contain documentation, source code, or examples.

For developers, 60East also provides an AMPS Command Reference that describes the commands to the AMPS server and responses from the AMPS server. Once you are familiar with the features you will use, as described in the user guide, the Developer Guide for your client library of choice and the AMPS Command Reference provide details on how an application communicates with the AMPS server.

Product Overview

AMPS, the Advanced Message Processing System, is built around an incredibly fast messaging engine that supports both publish-subscribe messaging and queuing. AMPS combines the capabilities necessary for scalable high-throughput, low-latency messaging in realtime deployments such as in financial services. AMPS goes beyond basic messaging to include advanced features such as high availability, historical replay, aggregation and analytics, content filtering and continuous query, last value caching, focus tracking, and more.

AMPS does more than just route and deliver messages. AMPS was designed to lower the latency in real-world messaging deployments by focusing on the entire lifetime of a message from the message's origin to the time at which a subscriber takes action on the message. AMPS considers the full message lifetime, rather than just the "in flight" time, and allows you to optimize your applications to conserve network bandwidth and subscriber CPU utilization -- typically the first elements of a system to reach the saturation point in real messaging systems.

AMPS offers both topic and content based subscription semantics, which makes it different than most other messaging platforms. Some of the highlights of AMPS include:

Topic and content based publish and subscribe
Message queuing, including content-based filtering and configurable strategies for delivery fairness
Client development kits for popular programming languages such as Java, C#, C++, C, Python, and JavaScript
Built-in support for FIX, NVFIX, JSON, BSON, MessagePack, BFlat, Google Protocol Buffer and XML messages. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing message types.
State of the World queries
Historical State of the World queries
Easy to use command interface
Full Perl compatible regular expression matching
Content filters with SQL92 WHERE clause semantics
Built-in latency statistics and client status monitoring
Advanced subscription management, including delta publish and subscriptions and out-of-focus notifications
Basic CEP capabilities for real-time computation and analysis
Aggregation within topics and joins between topics, including joins between different message types
Replication for high availability
Fully queryable transaction log
Message replay functionality
Fully-integrated authentication and entitlement system, including content-based entitlement for fine-grained control
Optional encryption (SSL) between client and server
Extensibility API for adding message types, user-defined functions, user-specified actions, authentication, and entitlement functionality

Requirements

Software Requirements

The AMPS server is supported on the following platforms:

Linux 64-bit (2.6 kernel or later) on x86 compatible processors

While 2.6 is the minimum kernel version supported, AMPS will select the most efficient mechanisms available to it and as a result, reaps greater benefit from more recent kernel and CPU versions.

The AMPS distribution contains all of the supporting libraries and dependencies needed to run on a typical Linux server installation: no further software is required.

spark, a basic command line client that supports a subset of AMPS functionality, requires Java 1.7 or later.
The utilities for inspecting AMPS files (amps_sow_dump, amps_clients_ack_dump, and so on) require a Python installation.
amps-grep requires a Python installation.

Organization of this Guide

This manual is divided into the following parts:

Part One presents introductory material and a brief overview of AMPS
Part Two explains the features of AMPS, including information on the following features:
- Subscribe and Publish, the basic building blocks of AMPS applications
- The expression language and functions used to take advantage of the content-aware features of AMPS are covered in AMPS Expressions and AMPS Functions
- Record and Replay Messages using the AMPS transaction log
- Competitive message consumption with Message Queues
- The Message Types that AMPS supports for content-aware processing
- Current value caching and database functions using State of the World (SOW) topics
State of the World topics enable many of the other advanced features in AMPS, such as:
This section also contains detailed chapters on specific topics, such as the AMPS filter language. Both application developers and administrators should become familiar with this section.
Part Three discusses AMPS deployment and operations, including:
This section is most useful for those with a focus on AMPS operations, although the information presented here is helpful for developers who want to design high-performance, high-availability applications that are easy to deploy and maintain.
Additional chapters provide reference information:
- Optionally-Loaded Modules describes special-purpose modules that are included in the AMPS distribution but are not loaded by default
- File Format Versions lists the file formats used by each AMPS version

Documentation Conventions

This manual is an introduction to the 60East Technologies AMPS product. It assumes that you have a working knowledge of Linux and uses the following conventions.

Construct

Usage

Text

Standard document text

Code

Inline code fragment

Variable

Variables within commands or configuration

Parameter (required)

Required parameters in parameter tables

Optional

Optional parameters in parameter tables

The AMPS documentation also includes the following types of notes:

Inside boxes with this icon, you will find usage tips or extra information.

Inside boxes with this icon, you will find information that's important to keep in mind when working with AMPS. These are typically recommendations that should generally be followed, but may not be applicable in special cases.

Inside boxes with this icon, you will find important information and guidelines that require special consideration or caution when using AMPS to ensure the proper functioning of the system and to avoid any potential issues or risks.

Inside boxes with this icon, you will find usage warnings or information that is critical for ensuring that AMPS functions correctly.

Additionally, here are the constructs used for displaying content filters, XML, code, command line, and script fragments.

(expr1 = 1) OR (expr2 = 2) OR (expr3 = 3) OR (expr4 = 4) OR (expr5 = 5) OR (expr6 = 6) OR (expr7 = 7) OR (expr8 = 8)

Command lines will be formatted as in the following example:

$ find . -name *.java

Technical Support

For an outline of your specific support policies, please see your 60East Technologies License Agreement. Support contracts can be purchased through your 60East Technologies account representative.

Support Steps

You can save time if you complete the following steps before you contact 60East Technologies Support:

Check the documentation
The problem may already be solved and documented in the User Guide or Configuration Reference Guide for the product. 60East Technologies also provides answers to frequently asked support questions on the support website at: http://support.crankuptheamps.com.
Isolate the problem
If you require Support Services, please isolate the problem to the smallest test case possible. Capture erroneous output into a text file along with the commands used to generate the errors.
Collect your information
- Your product version number.
- Your operating system and its kernel version number.
- The expected behavior, observed behavior and all input used to reproduce the problem.
- Submit your request.
- If you have a minidump file, be sure to include that in your email to crash@crankuptheamps.com.

The AMPS version number used when reporting your product version number follows a format listed below. The version number is composed of the following:

MAJOR.MINOR.FEATURE.HOTFIX.TIMESTAMP.TAG

AMPS Versioning and Certification

Each AMPS version number component has the following breakdown:

Component

Description

Minimum Verification

MAJOR

Increments when there are any backward-incompatible changes in functionality, file formats, client network formats or configuration; or when deprecated functionality is removed.

May introduce major new functionality or include internal improvements that introduce major behavioral changes.

Megacert

MINOR

Increments when functionality is added in a backwards-compatible way, or when functionality is deprecated.

May include internal improvements, including internal improvements that introduce minor behavioral changes or changes to network formats used only by the AMPS server (such as replication).

Megacert

FEATURE

Increments for previews of new features.

May introduce behavioral changes to fix incorrect behavior, enable new functionality or to enhance performance.

May include internal enhancements that do not introduce behavioral changes.

Note: A feature level of 0 indicates a long-term stable release. A feature level above zero indicates the current feature level (a preview of the next long-term stable release).

Kilocert

HOTFIX

A release for a critical defect impacting a customer. A hotfix release is designed to be 100% compatible with the release it fixes (that is, a release with same MAJOR.MINOR.FEATURE version).

May introduce behavioral changes to fix incorrect behavior. May document previously undocumented features or extend surface area to improve usability for existing features.

Cert

TIMESTAMP

Proprietary build timestamp.

(does not affect verification level)

TAG

Identifier that corresponds to precise code used in the release.

(does not affect verification level)

Certification Level

Description

Time to Certify

Megacert

Performance and long-haul testing.

Full regression suite and stress-testing suite, including replication testing and application scenario tests.

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

less than 2 weeks

Kilocert

Full regression suite and stress-testing suite, including replication testing and application scenario tests.

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

less than 1 week

Cert

Full unit testing suite, including new unit tests to verify correct behavior of bugfixes in this release.

Replication testing suite if release affects replication code.

4 hours

Contacting 60East Technologies Support

Please contact 60East Technologies Support Services according to the terms of your 60East Technologies License Agreement.

Support is offered through the United States:

Web:

E-mail:

Support:

Other support options (such as support via phone), may be available depending on the terms of your support agreement.

Installing and Starting AMPS

This section describes how to install and start AMPS. It describes the file structure of the AMPS distribution and how to configure a simple AMPS instance.

The Getting Started with AMPS section in the Introduction to AMPS covers setting up a basic development environment. This section includes a reference to the AMPS server command line options and provides information to help create a production deployment of AMPS.

Installing AMPS

To install AMPS, unpack the distribution for your platform where you want the binaries and libraries to be stored. For the remainder of this guide, the installation directory will be referred to as $AMPSDIR as if an environment variable with that name was set to the correct path.

Within $AMPSDIR are the following sub-directories:

Starting AMPS

The server sample configuration only provides configuration for subscribe/publish use of AMPS, and does not include any persistence for AMPS messages.

The file enables the instance administrative interface (the "Galvanometer"), including the ability to subscribe to topics using a websocket connection from the instance administrative interface.

On older processor architectures, ampServer will start the ampServer-compat binary. The ampServer-compat binary avoids using hardware instructions that are not available on these systems.

Once you have a configuration file saved to $AMPSDIR/amps_config.xml you can start AMPS with that file as follows:

If your first start-up is successful, you should see AMPS display a simple message similar to the following to let you know that your instance has started correctly.

The version numbers and dates will be appropriate for the version that you've started.

If you see this, congratulations! You have successfully cranked up the AMPS!

Command Line Options

The AMPS server binary supports the following command line options:

Production Configuration

To create a production configuration of AMPS, you configure the instance to meet the needs of the application (or applications) that will use the instance.

An overview of the most commonly used features is available in the Introduction to AMPS guide. This guide, the AMPS User Guide, describes the features in detail. The AMPS Configuration Guide lists the required and optional configuration for each feature.

Typically, all instances of AMPS will configure:

The instance Name (this is required).
The Admin interface for the instance, to make monitoring available. This typically includes setting a path to persist the instance statistics database.
Logging for the instance (at a minimum of info level for production instances, typically at trace level for development, testing, or UAT instances).
One or more Transports to allow incoming connections to the AMPS server.
Administrative actions to create a scheduled maintenance plan for the statistics database and the logs.

The ampServer binary will produce a minimal sample configuration to stdout if it is run with the --sample-config flag that shows a minimum configuration. Options that require site-specific information (for example, the path to the statistics database or log files) are commented out in the sample.

Instances of AMPS may then add configuration to take advantage of advanced messaging features (such as the State of the World, Aggregation and Analytics, the ability to Record and Replay Messages, and so on), to add resiliency by Replicating Messages Between Instances (typically required for Highly Available AMPS Installations), and so on.

Subscribe and Publish

AMPS is a rich message delivery system. At the core of the system, the AMPS engine is highly-optimized for publish and subscribe delivery. In this style of messaging, publishers send messages to a message broker (such as AMPS) which then routes and delivers messages to the subscribers. "Pub/Sub" systems, as they are often called, are a key part of most enterprise message buses, where publishers broadcast messages without necessarily knowing all of the subscribers that will receive them. This decoupling of the publishers from the subscribers allows maximum flexibility when adding new data sources or consumers.

AMPS can route messages from publishers to subscribers using a topic identifier and/or content within the message's payload. For example, in the figure above, there is a Publisher sending AMPS a message pertaining to the LN_ORDERS topic. The message being sent contains information on Ticker "IBM" with a Price of 125, both of these properties are contained within the message payload itself (i.e., the message content). AMPS routes the message to Subscriber 1 because it is subscribing to all messages on the LN_ORDERS topic. Similarly, AMPS routes the message to Subscriber 2 because it is subscribed to any messages having the Ticker equal to "IBM". Subscriber 3 is looking for a different Ticker value and is not sent the message.

Topics

A topic is a string that is used to declare a subject of interest for purposes of routing messages between publishers and subscribers. Topic-based Publish and Subscribe (e.g., Pub/Sub) is the simplest form of Pub/Sub filtering. All messages are published with a topic designation to the AMPS engine, and subscribers will receive messages for topics to which they have subscribed.

For example, in the diagram above there are two publishers: Publisher 1 and Publisher 2 which publish to the topics LN_ORDERS and NY_ORDERS, respectively. Messages published to AMPS are filtered and routed to the subscribers of a respective topic. For example, Subscriber 1, which is subscribed to all messages for the LN_ORDERS topic will receive everything published by Publisher 1. Subscriber 2, which is subscribed to the regular expression topic ".*_ORDERS" will receive all orders published by Publisher 1 and 2.

Regular expression matching makes it easy to create topic paths in AMPS. Some messaging systems require a specific delimiter for paths. AMPS allows you the flexibility to use any delimiter. However, 60East recommends using characters that do not have significance in regular expressions, such as forward slashes. For example, rather than using northamerica.orders as a path, use northamerica/orders.

AMPS does not restrict the characters that can be present in a topic name. However, notice that topic names that contain regular expression characters (such as . or *) will be interpreted as regular expressions by default, which may cause unexpected behavior.

Topics that begin with /AMPS are reserved. The AMPS server publishes messages to topics that begin with /AMPS as described in the Event Topics section. Some versions of the AMPS client libraries may internally publish to /AMPS/devnull. Your applications should not publish to topics that begin with /AMPS, as publishes to those topics may fail.

Each topic has an associated message type. Each client connection to AMPS also has an associated message type. A given client connection can only publish to topics with the same message type, and can only receive messages from topics with the same message type.

Ad Hoc Topics

AMPS does not require explicit configuration of a topic for publishers to send messages to the topic and subscribers to receive messages from the topic. However, if there is no configuration for the topic, AMPS does not persist messages to the topic, so no features that depend on having a persisted message state (for example; replay, aggregation, State of the World, and so on) are available for that topic. The message will be delivered to subscriptions that are active when the message is published, but the message will not be persisted or retained. These "ad hoc" topics are useful for low-latency delivery of messages that are only useful at the time that they are published.

Matching Multiple Topics With Regular Expressions

With AMPS, a subscriber can use a regular expression to simultaneously subscribe to multiple topics that match the given pattern. This feature can be used to effectively subscribe to topics without knowing the topic names in advance.

Notice that a message cannot be published to a topic pattern. The topic for a given message is unambiguously specified using a literal string. From the publisher’s point of view, it is publishing a message to a topic. A publisher does not publish to a topic pattern.

When a subscription is sent to AMPS, the topic for the subscription is interpreted as a regular expression if the topic includes special regular expression characters. Otherwise, the topic must be an exact match.

Some examples of regular expressions to match a set of topics are included in the table below:

Topic

Behavior

^trade$

Matches only "trade".

^client.*

Matches "client", "clients", "client001", etc.

.*trade.*

Matches "NYSEtrades", "ICEtrade", etc.

trade.info

Matches "trade/info", "trade-info", "every/trade/info", etc.

For more information regarding the regular expression syntax supported within AMPS, please see the Regular Expressions section for details.

AMPS can be configured to disallow regular expression topic matching for subscriptions. See the AMPS Configuration Guide for details.

Filtering Subscriptions by Content

One thing that differentiates AMPS from classic messaging systems is its ability to route messages based on message content. Instead of a publisher declaring metadata describing the message for downstream consumers, the publisher can simply publish the message content to AMPS and let AMPS examine the native message content to determine how best to deliver the message.

The ability to use content filters greatly reduces the problem of oversubscription that occurs when topics are the only facility for subscribing to message content. The topic space can be kept simple by using content filters to deliver only the desired messages. The topic space can reflect broad categories of messages and does not have to be polluted with metadata that is usually found in the content of the message. In addition, many of the advanced features of AMPS such as out-of-focus messaging, aggregation, views, and SOW topics rely on the ability to filter content.

Content-based messaging is somewhat analogous to database queries that include a WHERE clause. Topics can be considered tables into which rows are inserted (or updated). A subscription is similar to issuing a SELECT from the topic table with a WHERE clause to limit the rows which are returned. Topic-based messaging is analogous to a SELECT on a table with no limiting WHERE clause.

AMPS uses a combination of XPath-based identifiers and SQL-92 operators for content filtering. Some examples are shown below:

Example Filter for a JSON Message:

Example Filter for an XML Message:

Example Filter for a FIX Message:

Unlike some other messaging systems, AMPS lets you use a relatively small set of topics to categorize messages at a high level and use content filters to retrieve specific data published to those topics.

Examples of good, broad topic choices:

trades, positions, MarketData, Europe, alerts

This approach makes it easier to administer AMPS, easier for publishers to decide which topics to publish to and easier for subscribers to be sure that they've subscribed to all relevant topics.

Conflated Subscriptions

AMPS provides the ability for the server to conflate messages to a subscription. When a subscription requests conflation, the server will retain messages for that subscription for a certain period of time, the conflation interval, and provide the latest update to that message once a message has been retained for that interval. In effect, AMPS guarantees that a subscription will receive no more than one update for a given message per conflation interval.

Conflated subscriptions provide a way to reduce the bandwidth and processing for a subscriber in cases where a subscriber needs periodic updates with the current state of a message, rather than the complete message stream. AMPS provides per-subscription conflation for cases where only a small number of subscribers require conflation, or if conflation is required only in unusual cases. If multiple subscribers will have the same conflation needs, consider using Conflated Topics.

For example, imagine an application that monitors selected stocks and displays the current prices on a large screen, which refreshes every few seconds. This application may use the same topics as a trading desk, but has very different needs for data freshness and completeness. Since updates to each symbol will only be displayed every few seconds, the application only needs point-in-time updates of the prices, rather than the full stream of price changes. To meet this need, the application could specify that the subscription conflates price updates by tickerId with a conflation interval of two seconds. For each distinct value of the tickerId field, AMPS will retain messages for two seconds. If another message with the same tickerId is processed for the subscription during the conflation interval, that message completely replaces the previous message. At the end of the two second conflation interval, the message is delivered to the application. This lets the application receive an up-to-date price at most every two seconds, without having to process a large number of updates that will never be displayed. This approach also ensures that the price is never more than two seconds out of date, which means that each time the screen is refreshed, the price is current.

As mentioned in the example above, if the subscription uses tickerId for conflation and the following sequence of messages arrive during a conflation interval:

{ "tickerId" : "IBM", "price" : 150.34 }
{ "tickerId" : "IBM", "price" : 149.76 }
{ "tickerId" : "IBM", "price" : 149.32 }
{ "tickerId" : "IBM", "price" : 151.10 }

AMPS delivers only the last message for that tickerId:

{ "tickerId" : "IBM", "price" : 151.10 }

Notice that when a subscription is conflated, AMPS does not guarantee that messages are delivered precisely in the order in which they arrived in AMPS, since the latest update is delivered based on the conflation interval.

When the timestamp option is used with conflated subscriptions, AMPS provides the timestamp for the first message conflated.

When to Use Conflated Subscriptions

Conflated subscriptions reduce the bandwidth for a subscription, and may reduce the processing resources required for a subscription. However, rather than immediately delivering messages, AMPS retains messages in memory for the conflation interval. This can increase the memory required for the subscription.

AMPS contains other features for conflating messages and reducing bandwidth. Conflated subscriptions are most appropriate when:

Network bandwidth is at a premium, and you would like AMPS to spend slightly more processing time and potentially more memory to reduce the bandwidth needs of the application.
Each subscription has different conflation needs. For example, if each subscription has a dramatically different conflation interval, or needs to conflate by different fields. If most subscribers will use a similar conflation interval and use the same fields for conflation, using a Conflated Topic can provide equivalent results with lower overhead.
The conflation needs are relatively predictable and consistent for the subscription. If you need the application to conflate messages only when processing is slow or there are bursts of message traffic, client-side conflation provides that ability and may be a better choice than a conflated subscription. See the developer guide for your programming language of choice for details.

The considerations above are general guidance to help you consider options and choose a conflation strategy.

You can also combine approaches as necessary. For example, if most of your subscriptions require a 3 second conflation interval by tickerId, while a few subscriptions require a 15 second interval, you could create a Conflated Topic with a 3 second interval. Those subscriptions that require a 15 second interval could subscribe with that interval. This provides both sets of subscriptions with the intervals that they need.

Requesting Conflation on a Subscription

To request conflation on a subscription, set the following options on the subscription:

Option

Description

conflation=n

Specifies whether to conflate this subscription. The value provided can be a time interval, auto or none.

When present and set to a value other than none, enables conflation for the subscription.

Can also be set to auto, which requests that AMPS attempt to determine an appropriate conflation interval based on client consumption.

Recognizes the same time specifiers used in the AMPS configuration file (for example, 100ms or 1s or 1m).

Defaults to none.

conflation_key=[keys]

When conflation is enabled, specifies the fields to use to determine message uniqueness. The format of this option is a comma-delimited list of XPath identifiers within brackets.

For example, to conflate based on the value of the /tickerId and /customerId within a message. the value of this option would be:

[/tickerId,/customerId]

Defaults to the SOW key fields for SOW topics.

No default for non-SOW topics. This option is required for non-SOW topics.

This option is not valid with the oof option unless the keys provided are identical to the keys for the topic.

This option can only be used when conflation is also specified.

For example, to request a 10 second conflation interval with messages conflated on the [/orderId] field, you would use the following options string:

conflation=10s,conflation_key=[/orderId]

Replacing Subscriptions

AMPS provides the ability to perform atomic subscription replacement. This allows you to replace the filter, change the topic, or update the options for a subscription.

The most common use for this capability is for an application to change the filter for a subscription. For example, a GUI that is providing a view of a set of orders may need to add or remove an order from the set of orders being displayed. By replacing the content filter with a filter that tracks the updated set of orders, the application can do this without missing messages, getting duplicate messages, or having to manage more than one subscription.

Replacing a filter is an atomic operation. That is, the application is guaranteed not to miss messages that are in both the original and replacement subscription, and is guaranteed to receive all messages for the new subscription as of the point at which the replacement happens.

To replace a subscription, applications re-submit the subscription using the subscription ID of the previous subscription. See the Developer Guide of the client library you are using and the AMPS Command Reference for details.

When replacing a sow_and_subscribe command (described later in the guide), AMPS runs the SOW command again and provides any messages that were not previously in the result set to the application. See the section called Replacing Subscriptions with SOW and Subscribe for details.

Notice that some options on an initial subscription limit the support for replace on a subscription. In those cases, the limitation is described when the option is described.

Replacing the Content Filter on a Subscription

AMPS allows you to replace the content filter on an existing subscription. When this happens, AMPS begins sending messages on the subscription that match the new filter. When an application needs to bring more messages into scope, this can be more efficient than creating another subscription.

For example, an application might start off with a filter such as the following:

/region = 'WesternUS'

The application might then need to bring other regions into scope, for example:

/region IN ('WesternUS', 'Alaska', 'Hawaii')

Replacing the Topic on a Subscription

AMPS allows a subscription to replace the topic on a subscription. When the topic is replaced, AMPS re-evaluates the subscription as it does when a filter is replaced. If the subscription is updated to include a topic that the user does not have permission to subscribe to, the replace operation succeeds, but no messages will be delivered on that topic.

Replacing the Options on a Subscription

AMPS allows a subscription to replace some of the options on the subscription. In this case, the subscription is evaluated as though the topic or filter has been replaced. Any new messages generated after the subscription is replaced use the new options. However, AMPS does not replay or re-query previous messages to apply the options.

For example, if a sow_and_subscribe command did not previously specify Out-of-Focus tracking and adds this option, AMPS generates the appropriate Out-of-Focus messages from the replace point forward. AMPS does not recreate Out-of-Focus messages that would have previously been generated by the subscription.

If the subscription uses pagination (see Managing Result Sets), the replacement must contain the full set of pagination options provided on the original subscription. For a paginated subscription, the replacement may not change the topic of the subscription. Instead, close the existing subscription and create a new subscription with a different topic.

Messages in AMPS

Communication between applications and the AMPS server uses AMPS messages. AMPS messages are received or sent for every operation in AMPS. Each AMPS message has a specific type and consists of a set of headers and a payload. The headers are defined by AMPS and formatted according to the protocol specified for the connection. Typically, applications use the standard amps protocol which uses a JSON document for headers. The payload, if one is present, is the content of the message and is in the format specified by the message type.

Messages received from AMPS have the same format as messages to AMPS. These messages also have a specific type, with a header formatted according to the protocol and a payload of the specified message type. For example, AMPS uses ack messages, short for acknowledgment, to report the status of commands. AMPS uses publish messages to deliver messages on a subscription, and so on for other commands and other messages.

Let's consider a complete interaction between an application and the AMPS server as an example. When a client subscribes to a topic in AMPS, the client sends a subscribe message to AMPS that contains the information about the requested subscription and, by default, a request for an acknowledgment that the subscription has been processed. AMPS returns an ack message when the subscription is processed that indicates whether the subscription succeeded or failed, and then begins providing publish messages for new messages on the subscription. The publish messages continue as messages that match the subscription arrive at the AMPS server. If the application needs to stop the subscription, the application sends an unsubscribe message to the AMPS server, indicating the subscription to end. Once the AMPS server processes the unsubscribe message, the server will no longer send messages for that subscription to the application. Should the application disconnect, the AMPS server removes all subscriptions for that connection (whether or not the application sends an unsubscribe command first).

Messages to and from AMPS are described in more detail in the AMPS Command Reference, available on the 60East website and included in the AMPS client SDKs.

In this version of AMPS, the communication transports used by AMPS accept message sizes of up to 200MB in a single command to AMPS. Messages larger than 200MB may be rejected by the transport as invalid. Should your use of AMPS require larger message sizes, contact 60East support.

This version of AMPS limits messages to 200MB in total size

Introduction to AMPS Headers

The AMPS Command Reference contains a full list of headers for each command. The table below lists some commonly used headers.

Header

Description

Topic

The topic that the message applies to.

For commands to AMPS, this is the topic that AMPS will apply the command to. For messages from AMPS, this is the topic from which the message originated.

Command

The command type of the message. Each message has a specific command type.

For example, messages that contain data from a query over a SOW topic have a command of sow, while messages that contain data from a publish command have a command of publish and messages that acknowledge a command to AMPS have a command type of ack.

CommandId

An identifier used to correlate responses from AMPS with an initial command.

For example, ack messages returned by AMPS contain the CommandId provided with the command they acknowledge and subscriptions can be updated or removed using the CommandId provided with the subscribe command.

SowKey

This header is included on messages from a SOW topic by default. AMPS will omit this header when the subscription or SOW query includes the no_sowkey option.

CorrelationId

A user-specified identifier for the message.

Publishers can set this identifier on messages. AMPS does not parse, change, or interpret this identifier in any way.

This header is limited to characters used in Base64 encoding.

Status

Set on ack messages to indicate the results of the command, such as Success or Failure.

Reason

Set on ack messages to indicate the reason for the Status acknowledgment.

Timestamp

Optionally set on publish messages and sow messages to indicate the time at which the local AMPS instance processed the message.

To receive a timestamp, the SOW query or subscription must include the timestamp option on the command that creates the subscription or runs the query. The timestamp is returned in ISO-8601 format.

This section presents a few of the commonly used headers. See the AMPS Command Reference for a full description of AMPS messages.

AMPS does not provide the ability to add custom header fields. However, AMPS composite message types provide an easy way to add an additional section to a message type that contains metadata for the message. Since composite message type parts fully support AMPS content filtering, this approach provides more flexibility and allows for more sophisticated metadata than simply adding a header field. See the Composite Messages section for details.

Message Ordering

AMPS guarantees that, for each AMPS instance, each subscription to a topic receives messages in the order in which AMPS received the messages (with the exception of messages that have been returned to a message queue for redelivery or the results of a query). Before a given message is delivered to a subscriber, all previous messages for that topic are delivered to the subscriber. AMPS does this by enforcing a total order across the instance for all messages received from publishers, including messages received via replication. When AMPS is using a transaction log, that order is preserved in the transaction log for the instance, and persists across instance restarts. When replaying from the transaction log, a subscriber will always receive messages in the same order in which messages were originally delivered by that instance of AMPS.

This guarantee also applies across topics for subscriptions that involve multiple topics, for all topics except views, queues, and conflated topics. Views and queues guarantee that every message on the view or the queue appears in the order in which the message was published. However, the computation involved in producing messages for views and queues may introduce some amount of processing latency, and AMPS does not delay messages on other topics while performing these computations. For a queue that provides at-least-once delivery, if a processor fails and returns a message to the queue, that message will be redelivered (which means that the new processor may receive the message out of order). Likewise, when AMPS is providing conflation (either through a conflated topic or the conflation options on a subscription), AMPS does not provide ordering guarantees for conflated messages.

Applications often use this guarantee to publish checkpoint messages, indicating some external state of the system, to a checkpoint topic. For example, you might publish messages marking the beginning of a business day to a checkpoint topic, MARKERS, while the ORDERS topic records the orders during that day. Subscribers to the regular expression ^(ORDERS|MARKERS)$ are guaranteed to receive the message that marks the business day before any of the messages published to the ORDERS topic for that day, since AMPS preserves the original order of the messages.

For messages constructed by AMPS, such as the output of a view, AMPS processes messages for each topic in the order in which they arrive (unless conflation is requested) and delivers each calculated message to subscribers as soon as the calculation is finished and a message is produced. This keeps the latency low for each individual topic. However, this means that while AMPS guarantees the order in which messages are produced within each view, messages produced for views that do simple operations will generally take less time to be produced than messages for views that perform complex calculations or require more complicated serialization. This means that AMPS guarantees ordering within view topics, but does not guarantee that messages for separate view topics arrive in a particular order.

The figure below shows a possible ordering for messages received on an underlying topic and two views that use the topic:

Notice that within each topic, AMPS enforces an absolute order. However, the Simple View produces the results of Message 3 before the Complex View produces the results of Message 2. AMPS delivers the message for each topic as soon as possible.

Replicated Message Ordering

When providing messages received via replication (see Replicating Messages Between Instances), the principles on message ordering provided above still apply. AMPS records messages into the local transaction log in the order in which messages are received by the instance and provides messages to subscribers in that order. AMPS uses the sequence of publishes assigned by the original publisher and the order assigned by the upstream instance to ensure that all replicated messages are received and recorded in order with no gaps or duplicates.

Each instance of AMPS replicates messages to downstream destinations in the order in which messages are recorded in the transaction log.

AMPS does not enforce a global total ordering across a replication topology. This peer-to-peer approach means that an AMPS instance can continue accepting messages from publishers and providing messages to subscribers even when the remote side of a replication link is offline or if replication is delayed due to network congestion. However, if two messages are published to different instances at the same time by different publishers, the two instances may record a different overall message order for those messages, even though message order from each publisher is preserved.

Retrieving Part of a Message

AMPS has the ability to allow a subscriber to retrieve only the relevant parts of a message, in the same way that a SQL query can retrieve only specified fields from a table. For example, consider a topic that stores an event ID, a short description, and a detailed event record. A UI that presents an overview of the contents of the topic might only need the event ID and short description to present a high-level view of the topic contents, while retrieving the detailed event record when a user explicitly requests the details for a specific record.

With select lists, AMPS allows an individual subscription to control which fields are retrieved from a subscription or query. In the example above, the subscription would include a select list that requests that AMPS provide the event ID and description, while excluding any other field. To do this, the application would include the following option on the command used to retrieve data for the overview: select=[-/,+/event_id,+/description]

When provided by an application as a part of a command to AMPS, a select list is applied after any content filtering is applied. The select list specifies the contents of the subscription, but does not affect the underlying messages, and the contents of the subscription select list do not affect filter evaluation or query results.

To use a select list, the message type format must allow partial serialization of messages. Message formats that require a full message, or that interpret missing fields as having a specific value, cannot be used with select lists, since a partial message would either create an invalid message, or change the way the data in the message is interpreted.

Creating Select Lists

As mentioned above, to provide a select list on a command, add the keyword select and a comma-delimited list of field directives to the options for a subscription or query in AMPS.

Each field directive is a combination of an inclusion specifier and an AMPS identifier.

For example, the field directive +/event_id has an inclusion_specifier of + and the AMPS identifier of /event_id. This field directive specifies that the /event_id field is included in the message returned to the subscriber.

AMPS recognizes the following inclusion specifier values:

Specifier

Meaning

-

Explicitly exclude the field for the identifier immediately following.

+

Explicitly include the field for the identifier immediately following.

Identifiers for individual fields follow the syntax described in the Identifiers section.

For select lists, AMPS also recognizes the special field directive of -/ to specify that all fields should be excluded and the special field directive of +/ to specify that all fields should be included.

If no field directive in the select list applies to a given field in a message, that field is included in the message.

If a field is covered by multiple field directives, AMPS respects the most specific field directive. In other words, a select list that contains the field directives +/,-/details will include all fields except the details field. A select list that contains the field directives -/event,+/event/description will include the /event/description subfield, but no other contents of the /event field. (If an identifier is provided twice in the same select list, AMPS uses the first field specifier that contains the identifier.)

With select lists, AMPS does not create fields that are not in the original message. This means that if the select list requests a field that does not exist in the original message, the message delivered to the subscriber will not contain that field.

Notice that a select list only changes how a message is delivered to the subscriber that the select list applies to. The original message is unaffected, and the complete message is delivered to any subscriber that does not specify a select list.

AMPS contains related functionality that may be more appropriate for some applications:

To modify a message as it is published to AMPS, use Enrichment and Preprocessing. With those features, the original publish message is modified and the modified message is stored in AMPS and sent to all subscribers.
AMPS also offers the ability to create a view of a set of messages that aggregates data across a set of messages and produces a result (for example, the total value of all open orders for each customer). See the chapter on Aggregating and Analyzing Data in AMPS for more details.

Select List Examples

For example, consider an original message like the following JSON document:

{ "id": 42,
  "name":"Arthur",
  "day":"Thursday",
  "complaint":"Unannounced construction in neighborhood.",
  "pocket_contents":
        { "left":"twine",
          "right":"towel" }
}

An application might only need to see the id and complaint description. To retrieve just those fields of a message, the application could add the following option to the command that retrieves the message:

select=[-/,+/id,+/complaint]

This select list tells AMPS to remove all fields from the message except for the /id field and the /complaint field. With this select list, the message above will be delivered as:

{ "id": 42,
  "complaint":"Unannounced construction in neighborhood."
}

Likewise, an application could want to know the name of the person making the complaint and the contents of that person's left pocket:

select=[-/,+/name,+/pocket_contents/left]

From the original message, the result of providing this select list would be:

{ "name": "Arthur",
  "pocket_contents":
        { "left":"twine"}
}

Last, consider an application that wants to see everything in the message except the pocket_contents. That application could provide an option such as:

select=[-/pocket_contents]

With that specifier, AMPS provides any field in the message except the pocket_contents, producing the following result:

{ "id": 42,
  "name":"Arthur",
  "day":"Thursday",
  "complaint":"Unannounced construction in neighborhood."
}

Select lists are not available for struct message types, since these types represent a single, contiguous block of memory (and, therefore, cannot meaningfully have omitted fields).

AMPS Expressions

AMPS includes an expression language that combines elements of XPath and SQL-92's WHERE clause. This expression language is used whenever the AMPS server refers to the contents of a message, including:

Content filtering
Constructing fields for message enrichment
Creating projected fields for views

AMPS uses a common syntax for each of these purposes, and provides a common set of operators and functions. AMPS also provides special directives for message enrichment, and aggregation functions for projecting views.

For example, when an expression is used as a content filter, any message for which the expression returns true matches the content filter. When an expression is used to construct a field for message enrichment or view projection, the expression is evaluated and the result that the expression returns is used as the content of the field.

Expressions Overview

The quickest way to learn AMPS expressions is to think of each as a combination of identifiers that tell AMPS where to find data in a message, and operators that tell AMPS what to do with that data. Each AMPS expression produces a value. The way AMPS uses that value depends on where the expression is used. For example, in a content filter, AMPS uses the value of the expression to determine whether a message matches the filter. When constructing a field, AMPS uses the value of the expression as the contents of the field.

Consider a simple example of an expression used as a filter. Imagine AMPS receives the following JSON message:

{"name":"Gyro", "job":"kitten"}

Using an AMPS expression, you can easily construct a content filter that matches the message:

/name = 'Gyro'

There are three parts to this expression. The first part, /name, is an identifier that tells AMPS to look for the contents of the name field at the top level of the JSON document. The second part of the filter, =, is the equality operator, which tells AMPS to compare the values on either side of the operator and return true if the values match. The final part of the filter, 'Gyro', is a string literal for the equality operator to use in the comparison. When an expression is used in a content filter, a message matches the filter when the expression returns true. The expression returns true for the sample message, so the sample message matches the filter.

The identifier syntax is a subset of XPath, as described in the section on Identifiers. The comparison syntax is similar to SQL-92.

Notice that AMPS makes no rigid guarantees as to the number of times a given expression is evaluated or when that evaluation will take place. AMPS will evaluate the expression as needed.

Syntax

AMPS expressions are designed to work exactly as expected if you are familiar with XPath path specifiers and SQL-92 predicates. This section describes in detail how AMPS evaluates the syntax, operators, and functions available in the AMPS expression language.

AMPS expressions combine the following elements:

Identifiers specify a field in a message. When evaluating an expression, AMPS replaces identifiers with values from the message or set of messages being evaluated.
Literal values are explicit values in an AMPS expression, such as 'IBM' or 42.
Operators and functions such as =, <, >, *, and UNIX_TIMESTAMP().

Every AMPS expression produces a value. The way that AMPS uses the value depends on the context in which AMPS evaluates the expression. For example, if the expression is used for a filter, the message is considered to match the filter when the expression returns true. When an expression is used to project a field, the result of the expression is used as the value of the projected field.

Identifiers

AMPS identifiers use a subset of XPath to specify values in a message. AMPS identifiers specify the value of an attribute or element in an XML message, and the value of a field in a JSON, FIX or NVFIX message. Given that the identifier syntax is only used to specify values, the subset of XPath used by AMPS does not include wildcards, relative paths, array manipulation, predicates or functions.

For example, when messages are in this XML format:

<Order update="full">
    <ClientID>12345</ClientID>
    <Symbol>IBM</Symbol>
    <OrderQty>1000</OrderQty>
</Order>

The following identifier specifies the Symbol element of an Order message:

/Order/Symbol

The following identifier specifies the update attribute of an Order message:

/Order/@update

For FIX and NVFIX, you specify fields using / and the tag name. AMPS interprets FIX and NVFIX messages as though they were an XML fragment with no root element. For example, to specify the value of FIX tag 55 (symbol), use the following identifier:

/55

Likewise, for JSON or other types that represent an object, you navigate through the object structure using the / to indicate each level of nesting.

AMPS only guarantees support for field identifiers that are valid step names in XPath. For example, AMPS does not guarantee that it can process or filter on a field named Fits&Starts.

AMPS also supports an optional bracketed field identifier syntax that extends the characters available for field names. For example, the following step name:

[/Not Xpath Name]

refers to a field name of Not Xpath Name at the root level of the message. This syntax allows spaces to be used in field names in AMPS expressions, even though this is not a valid step name in XPath. Notice that not all message types support field names with embedded spaces or other special characters. For example, the Not Xpath Name identifier is not a valid element name in XML, nor would it be a valid field name in Google Protocol Buffers.

AMPS checks the syntax of identifiers when parsing an expression. AMPS does not try to predict whether an identifier will match messages within a particular topic. It is not an error to submit an identifier that can never match due to the limitations of the message type. For example, AMPS allows you to use an identifier like /OrderQty in a filter submitted for a FIX connection, even though FIX messages only use numeric tags, or an identifier like /DataPackage/RunDate in a filter submitted for a BFlat connection, even though BFlat does not support nested elements.

The message type is responsible for constructing a set of identifiers from a message. In most cases, the mapping is simple. However, see the documentation for the message type for details, or if the mapping is unclear. For example, a composite-local message type adds the number of the part to the beginning of each XPath within the part (so, a top-level field of /name in the first part of the message has an identifier of /0/name).

AMPS Data Types

Each value in AMPS is assigned a data type when the message type module parses the value. AMPS operators and functions attempt to convert values into compatible types, based on the type of operation. For example, the * operator (multiplication) will attempt to convert all values to numeric values, while the CONCAT function (string concatenation) will attempt to convert all values to strings. In effect, a value in AMPS can be transparently treated as any type to which it can be meaningfully converted.

Internally, AMPS uses the data types in the table below. As mentioned above, the message type module is responsible for assigning the type of a value from an incoming message as part of the parsing process. For some types, such as JSON, XML, FIX and NVFIX, the parser infers the type of the value from the field. For other types, such as MessagePack, BFLAT, Google Protocol Buffers or BSON, the message itself contains information about the type of the field.

As mentioned above, the AMPS expression language does not limit the value to the type assigned by the message type module. Instead, a value in AMPS can be used in any context.

For example, given the following JSON document:

The values of /a and /b can be used as either string values or numeric values. AMPS will automatically convert these values as necessary, and AMPS considers the string or numeric representation to be equally correct and valid.

The following table lists the data types in the AMPS expression language:

Numeric Types and Literals in AMPS Expressions

Numeric values in AMPS are always typed as either integers or floating point values. All numeric types that are less than or equal to the LONG_MAX limit in AMPS are signed, otherwise, the numeric type is unsigned. AMPS message types convert the original numeric types (or original representation for message types that do not have typed values) into the internal AMPS type system for the purposes of expression evaluation.

Within expressions, integer values are all numerals, with no decimal point, and can have a value in the same range as a 64-bit integer. For example:

Within expressions, all numerals with a decimal point are floating-point numbers. AMPS interprets these numerals as double-precision floating point values. For example:

or, in scientific notation:

AMPS automatically converts strings that contain numeric values to numbers when strings are used with an operator, function or comparison that expects a numeric value.

Type Promotion for Numeric Types

AMPS uses the following rules for type promotion when evaluating numeric expressions:

If any of the values in the expression is NaN, the result is NaN.
Otherwise, if any of the values in the expression is floating point, the result is floating point.
Otherwise, all of the values in the expression are integers, and the result is an integer.

Notice that, for division in particular, the results returned are affected by the type of the values. For example, the expression 1 / 5 evaluates to 0 since the result is interpreted as an integer. In comparison, the expression 1.0 / 5 evaluates to 0.2 since the result is interpreted as a floating point value.

String Literals in AMPS Expressions

When creating expressions for AMPS, string literals are indicated with single or double quotes. For example:

AMPS supports the following escape sequences within string literals:

Additionally, any character which follows a backslash will be treated as a literal character.

AMPS string operations have no restrictions on character set, and correctly handle embedded NULL characters (\x00) and characters outside of the 7-bit ASCII range. AMPS string operations are not unicode-aware.

NULL, NaN and IS NULL

XPath expressions are considered to be NULL when they evaluate to an empty or nonexistent field reference. NULL values follow SQL-92 semantics.

This means that comparisons with NULL are never true (in other words, even if /a is NULL, /a != NULL is false and /a == NULL is also false).

AMPS considers a zero-length string to be NULL.

In numeric expressions where the operands or results are not a valid number, the XPath expression evaluates to NaN (not a number). The rules for applying the AND and OR operators against NULL and NaN values are outlined in the tables below:

Likewise, direct comparisons with NULL are not ever true (so, if /b is NULL, /b == NULL does not produce a true value, and neither does /b != NULL). AMPS, like SQL-92, provides an IS NULL predicate for testing whether a value is NULL, and an IS NOT NULL predicate for testing whether a value is not NULL.

There also exists an IS NAN predicate for checking that a value is NaN (not a number.)

To reliably check for existence of a NULL value, you must use the IS NULL predicate such as the filter: /optionalField IS NULL

To reliably check that a value is not NULL, you must use the IS NOT NULL predicate or negate the value of an IS NULL test: /optionalField IS NOT NULL and NOT /optionalField IS NULL are equivalent.

AMPS also provides a COALESCE() function that accepts a set of values and returns the first value that is not NULL. For example, given the following filter expression:

AMPS will return the first value that is not NULL, and compare that value to the constant string 'restricted'. Notice that, to make the intent of the filter clear, this example provides a constant value for AMPS to return from the COALESCE if all of the field values are NULL.

Compound Types in AMPS

Many messaging applications are designed for high performance and use a simplified message structure. For applications that use compound types, AMPS includes the ability to parse and filter on the contents of nested data structures.

For performance, AMPS parses nested data structures into a set of values. As with single-valued (or scalar) values, the AMPS expression language refers to a parsed set of values that is common to all message types rather than the underlying data.

The AMPS message types treat compound data types as a set of paths with corresponding scalar values. A field that only contains other fields is represented as a step in the path to the primitive values that it contains.

AMPS parses compound types as follows:

Any field that contains a scalar value is represented as an identifier/value pair.
Any field that contains other fields is represented as a step in the path to that value.

The following JSON document is a simple example.

With this document, AMPS produces the following parsed value:

In the parsed representation, the outer and middle fields contain no data of their own. They serve only as containers for the inner field which contains data.

Notice that the intermediate paths do not have an explicit scalar value.

With a more complex document the parsed representation continues to follow the same principles, as shown in the following example.

The representation of the above message in the AMPS expression language would typically be as follows:

As with the first example, fields that do not directly contain a value do not have an explicit scalar value. Values with the same identifier are represented as an array of values with that identifier.

Grouping and Order of Evaluation

AMPS expressions allow you to group parts of the expression using parentheses. Parts of an expression inside parentheses are evaluated together. 60East recommends using parentheses to group independent parts of an expression to ensure that the expression is evaluated in the expected order. For example, in this expression:

The clause /counter % 3 is evaluated first, and the result of that evaluation is compared to 0.

Within a group, elements are evaluated left to right in precedence order. For example, given the filter below:

AMPS evaluates expression2, then expression3 (since AND has higher precedence than OR), and if they evaluate to false, then expression1 will be evaluated.

AMPS does not guarantee that all parts of an expression will be evaluated if the result of an expression can be determined after only evaluating part of the expression. For example, given the expression:

AMPS only guarantees that B_FUNCTION(/b) will be evaluated if A_FUNCTION(/a) returns false.

Logical Operators

The logical operators are NOT, AND, and OR, in order of precedence. These operators have the usual Boolean logic semantics.

/FIXML/Order/Instrmt/@Sym = 'IBM' OR /FIXML/Order/Instrmt/@Sym = 'MSFT'

As with other operators, you can use parentheses to group operators and affect the order of evaluation.

(/orderType = 'rush' AND /customerType IN ('silver', 'gold') )
OR  /customerType = 'platinum'

Comparison Operators

The comparison operators can be loosely grouped into equality comparisons and range comparisons. The basic equality comparison operators, in precedence order, are ==, =, >, >=, <, <=, !=, and <>. The == comparison and the = comparison are treated as the same operator and produce the same results.

If these binary operators are applied to two operands of different types, AMPS attempts to convert strings to numbers. If conversion succeeds, AMPS uses the numeric values. If conversion fails because the string cannot be meaningfully converted to a number, strings are always considered to be greater than numbers. The operators consider an empty string to be NULL.

The following table shows some examples of how AMPS compares different types.

There are also set and range comparison operators. The BETWEEN operator can be used to check the range values.

The range used in the BETWEEN operator is inclusive of both operands, meaning the expression /A BETWEEN 0 AND 100 is equivalent to /A >= 0 AND /A <= 100.

For example:

The IN operator can be used to perform membership operations on sets of values. The IN operator returns true when the value on the left of the IN appears in the set of values in the IN clause. For example:

The IN operator returns true for the set of records that would be returned by an equivalent set of = comparisons joined by OR. The following two statements return the same set of records:

This equivalence means that NULL values in either the field being evaluated, or the set of values provided to the IN clause, always return false.

This also means that, for string values, the IN operator performs exact, case-sensitive matching.

When evaluating against a set of values, the IN operator typically provides better performance than using a set of OR operators. That is, a filter written as /firstName IN ('Joe', 'Kathleen', 'Frank', 'Cindy', 'Mortimer') will typically perform better than an equivalent filter written as /firstName = 'Joe' OR /firstName = 'Kathleen' OR /firstName = 'Frank' OR /firstName = 'Cindy' OR /firstName = 'Mortimer'.

LIKE Operator

AMPS also provides a regular expression comparison operator, LIKE, to provide regular expression matching on string values. A pattern is used for the right side of the LIKE operator. A pattern must be provided as a literal, quoted value. For more on regular expressions and the LIKE comparison operator, please see the section on Regular Expressions.

The string comparison operators described in the section called String Comparison Functions are usually more efficient than equivalent LIKE expressions, particularly when used to compare multiple literal patterns, or when the only purpose of the regular expression is to perform case-insensitive matching. Use LIKE operations when it is not practical to represent the filter condition with the string comparison operators.

AMPS Server Documentation 5.3.4

Welcome to AMPS 5.3.4

Introduction to AMPS

Getting to Know AMPS

Overview of AMPS

AMPS Concepts

Feature Highlights

Getting Started With AMPS

Crank Up the AMPS

Installing AMPS

Starting AMPS

Command Line Options

JSON Messages - A Quick Primer

spark: the AMPS command-line client

Interacting with AMPS Using Spark

Evaluating AMPS on Windows or MacOS

Using Windows Subsystem for Linux 2

Creating a Virtual Machine Image

Virtual Box Settings

Choosing a Linux Distribution

Installing the Linux Distribution

Filesystem Considerations

Next Steps

Galvanometer and RESTful Statistics

AMPS Basics: Subscribe and Publish to Topics

Topics, Publish and Subscribe

Topic Configuration for Basic Pub/Sub

Regular Expression Subscriptions

Spark: Basic Publish and Subscribe Example

Content Filters

Spark: Subscription with Content Filter

Further Reading

State of the World (SOW): The Message Database

When Should I Store a Topic in the SOW?

How Does the SOW Work?

Configuration

Queries

Spark: Basic SOW Query Example

Atomic Query and Subscribe

Spark: Basic SOW Query and Subscribe Example

Advanced Messaging and the SOW

Record and Replay Messages with the AMPS Transaction Log

How Does the Transaction Log Work?

Configuration

Further Reading

Message Queues

How Do Queues Work?

When Should I Use Queues?

Configuration

Further Reading

Scenario and Feature Reference

Recovery Strategies

Getting Support

Technical Support and Assistance

Support Steps

Contacting 60East Technologies Support

Advanced Topics

Further Reading

Event Logging

Conflation for Topics and Subscriptions

View Topics and Aggregation

Paginated Subscriptions

Historical SOW Query

Utilities

Monitoring Interface

High Availability

Next Steps

Learning More

Operation and Deployment

Application Development

AMPS Evaluation Guide

Introduction

Welcome to 60East Technologies AMPS

Documentation Roadmap

AMPS Software Requirements

Obtaining Evaluation Licenses and the AMPS Server

Obtaining Evaluation Support

Support Steps

AMPS Versioning and Certification

Contacting 60East Technologies Support