Welcome to the AMPS Server Documentation! This set of documentation contains detailed information on the AMPS server itself.

If you are looking for developer docs for client libraries, or previous versions of the AMPS server documentation, see the documentation page on the 60East web site.

Here are some suggested starting points:

You can also visit the AMPS Server FAQ site for frequently asked questions, and the AMPS developer pages for resources on developing applications with AMPS.

60East strongly recommends setting up an AMPS environment for testing while you work with the documentation. Instructions for doing so are in the Introduction to AMPS.

Welcome to the Advanced Message Processing System (AMPS) from 60East Technologies.

AMPS is a feature-rich message processing system that delivers previously unattainable low-latency and high-throughput performance to users. AMPS provides both publish-and-subscribe messaging and high-performance message queuing. AMPS also provides current value caching / message database functionality, analysis and aggregation.

Welcome to the Advanced Message Processing System (AMPS) from 60East Technologies! AMPS is designed to help you quickly and easily develop and deploy data-intensive applications, with demanding requirements, for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems.

AMPS isn't a traditional database or messaging product. This guide presents a brief introduction to help you understand the capabilities of AMPS and how AMPS operates.

AMPS is widely used for applications such as:

• Tradeplant operations (including backtesting and historical analysis)

• Risk calculations

• Elastic worker farms

• View servers

• Message flow integration and "shock absorbers"

AMPS combines a set of capabilities that cut across traditional boundaries between applications that work with data.

AMPS is built around a fast messaging engine that supports both publish and subscribe (fan-out) and queued (competitive consumption) message delivery with full content filtering.

AMPS also provides an integrated database that applications can use as a current value cache, key/value document store, and fully queryable database -- or all of these at once. With this database, AMPS includes a built-in aggregation and analytics engine for near-real time analysis of streaming data, including aggregation across multiple topics or message formats.

Integrated message logging provides the ability to record and replay streams of messages with full fidelity.

AMPS is designed from the ground up for enterprise deployment at scale. AMPS provides an extensive set of high-availability features, including integrated replication and automatic failover and recovery for applications. Detailed monitoring and statistics are included from a RESTful interface for ease of data collection and integration with enterprise monitoring and management systems.

Authentication and entitlement capability applies to every operation in AMPS, for fine-grained control over permissions to meet enterprise policy and regulatory requirements. Access to data can be controlled at a topic level, at a message level (content-based security), or at the level of individual fields within a message (limiting the fields a given user has access to view).

60East developed AMPS to serve the needs of some of the most demanding data-intensive applications on the planet. The feature set and capabilities have been engineered for the highest levels of performance, designed for ease of use, and proven in production applications worldwide.

Getting to Know AMPS

AMPS is designed to be a developer friendly product. 60East recommends reading about AMPS with a running instance of AMPS and your development environment of choice available. Although 60East makes every effort to clearly describe how AMPS works, there is no substitute for seeing exactly how a running instance behaves (not to mention the advantages of being able to try out ideas or do quick prototyping while you read).

The table below lists the main parts of the AMPS documentation:

This guide uses the spark command line utility for basic examples for simplicity, although a production installation would use an application to perform these functions.

This section describes how to install and start AMPS. It describes the file structure of the AMPS distribution and how to configure a simple AMPS instance.

The section in the covers setting up a basic development environment. This section includes a reference to the AMPS server and provides information to help create a production deployment of AMPS.

This section describes the functions installed by default in the AMPS server.

Additional functions that ship with the AMPS server are provided in auxiliary modules, as described in the section on .

On the 60East website at http://www.crankuptheamps.com/evaluate the current release of AMPS is available for evaluation download.

To get started, download the Linux installation to a directory on your Linux system.

Installing AMPS is simply a matter of unpacking the distribution. The distribution contains the complete set of libraries and dependencies needed to run the AMPS server on a typical Linux server distribution. No additional software or packages are necessary for the server itself.

To install AMPS, unpack the distribution in the directory where you want the binaries and libraries to be stored. For the remainder of this guide, the installation directory will be referred to as $AMPSDIR as if an environment variable with that name was set to the correct path.

Within $AMPSDIR are the following sub-directories:

When the Admin interface is configured (as it is in the sample configuration), you can get information about the state of the AMPS instance using either the Galvanometer monitoring tool or the RESTful interface to the AMPS statistics.

The Galvanometer is a Javascript-based application that runs in your browser and provides a visualization of the data provided by the RESTful interface. The Galvanometer also includes a lightweight read-only AMPS client application, based on the Javascript client library.

The RESTFul admin interface is a lightweight view of the statistics database that AMPS maintains.

These two interfaces are available at the following URIs:

In the URIs above, <host> is the host the AMPS instance is running on and <port> is the administration port configured in the configuration file (this is 8085 in the sample configuration).

Monitoring applications typically collect information from the RESTful statistics interface. Interactive or ad hoc monitoring can use either the Galvanometer, or the interface offered by the monitoring application in use locally once the statistics are collected.

For more information on the monitoring capabilities available in AMPS, see the chapter on Monitoring AMPS in the AMPS User Guide. For detailed information on the statistics available, see the AMPS Monitoring Guide.

One of the core features of AMPS is the ability to persist the most recent update for each distinct message published to a topic. To enable this for a topic, you add the topic to the SOW.

You can think of the SOW as a database that maintains a specific set of topics, equivalent to tables. Each distinct message published to that topic is equivalent to updating a row in the table. AMPS allows applications to query the table for the current state of the topic.

SOW topics also provide full support for pub/sub messaging. Applications can use a combination of queries and subscriptions as necessary. AMPS also includes a set of commands that perform an atomic query and subscribe, allowing an application to query a SOW topic and register for updates to the topic in a single operation, without risk of missing messages or receiving duplicates.

The most common uses of SOW topics include:

• Quickly loading initial state for an application. For example, an application that tracks open orders can quickly retrieve a snapshot of all of the orders that are currently open, without having to wait for updates to the orders to be published.

• Queryable snapshots of data flows. For example, an application that monitors telemetry data may need to quickly determine if any telemetry source has not provided an update within a given period of time. With a SOW topic, the application can run a simple query over the current state of the topic.

• NoSQL document stores. SOW topics are frequently used as high-performance key/value stores: an application can choose to explicitly provide a key and store a document in the SOW. Documents can be efficiently retrieved by key, queried over the full content of the document, or any combination. As mentioned above, a consumer can retrieve the document and be automatically notified when the content of the document changes.

SOW topics are also the foundation of many of the more advanced capabilities of AMPS, including out-of-focus tracking, aggregation, and delta messaging. These are described later in this chapter.

For applications that are transitioning from topic-based routing and that, therefore, need to maintain the last value per topic for a large number of topics (hundreds, thousands, or more), AMPS provides the ability to reduce the overhead in creating a large number of identical topics that contain a single message. More details on the State of the World are available in the AMPS User Guide.

To create a SOW topic, you configure the topic in the SOW section of the AMPS configuration file.

At a minimum, SOW topics require a Name, and the MessageType of the messages to store in the SOW. If the SOW will be persistent, a FileName is required. Most often, SOW topics use AMPS to generate the SOW Key, and one or more Key definition elements are required to specify the fields that AMPS will use for the SOW Key.

For example, the following configuration file fragment specifies a SOW topic named test-sow. The topic stores JSON-format messages, and uses the /id field of incoming messages to that topic to uniquely identify messages. Records in this topic will be both maintained in memory and persisted to a file in the ./sow/ directory, so the contents of the topic will be retained across restarts of the AMPS instance. Notice that the file name specification uses the special format character %n as a placeholder for the topic name and message type.

<SOW>
   <Topic>
        <Name>test-sow</Name>
        <MessageType>json</MessageType>
        <FileName>./sow/%n.sow</FileName>
        <Key>/id</Key>
    </Topic>
</SOW>

The Configuring Topics in a SOW section of the AMPS User Guide contains full details on configuring a SOW topic.

The practical examples later in this section use the configuration above.

A SOW topic is the basis for many of the advanced messaging features in AMPS. While not all of these features are discussed in detail in this introduction, many features of AMPS are made possible because AMPS can retain the current state of each unique message.

The advanced messaging features that the SOW enables include:

• Views and aggregations over topics (including joins between topics)

• Publishing incremental updates to a message (called delta publishing in AMPS)

• Receiving incremental updates to a message (called delta subscription in AMPS)

• Determining when a message no longer matches a filter (called out-of-focus notification in AMPS)

• Providing a snapshot of an update to a rapidly changing record at regular intervals, rather than providing every update (called conflation in AMPS)

These features can greatly simplify the processing an application needs to perform, making it easier to develop applications and increasing application performance. However, for a messaging system to provide these features, whenever a message arrives, the messaging system must have access to both the current message and the previous, saved state of the message. SOW topics provide that access for AMPS, and enable the advanced messaging features.

Once you have done a basic evaluation of AMPS, there are two typical paths forward in usage of the product:

• On one path, you may want to learn how to configure, deploy, and administer an instance of AMPS. For this path, see the AMPS User Guide, which provides complete information for system administrators who are responsible for the deployment, availability and management of data to other users.

• Alternatively, you may need to develop an application to work with AMPS, using one of the Developer Guides for Java, Python, C++, or C#. For this path, download one of the client distributions from the AMPS developer page at https://www.crankuptheamps.com/develop/. The client distributions include a set of examples and an AMPS server configuration that works with the examples.

The following sections provide more information about each of these paths and also briefly describes some use cases for AMPS.

Operation and Deployment

In preparing to deploy your instance of AMPS, you must size your host environment according to multiple dimensions: memory, storage, CPU, and network. The Operation and Deployment chapter in the AMPS User Guide provides guidelines and best practices for configuring the host environment. The chapter also specifies recommended settings for running AMPS on a Linux operating system.

Application Development

Each language-specific Development Guide explains how to install, configure, and develop applications that use AMPS. In order to develop applications using an AMPS client, you must understand the basic concepts of AMPS, such as topics, subscriptions, messages and SOW.

You will also need an installed and running AMPS server to use the product. Typically, a team will use a server licensed for evaluation during the initial stages of development, then transition to a full license as the evaluation completes and the team prepares to deploy the application.

AMPS, the Advanced Message Processing System, is built around an incredibly fast messaging engine that supports both publish-subscribe messaging and queuing. AMPS combines the capabilities necessary for scalable high-throughput, low-latency messaging in realtime deployments such as in financial services. AMPS goes beyond basic messaging to include advanced features such as high availability, historical replay, aggregation and analytics, content filtering and continuous query, last value caching, focus tracking, and more.

Furthermore, AMPS is designed and engineered specifically for next generation computing environments. The architecture, design and implementation of AMPS allows the exploitation of parallelism inherent in emerging multi-socket, multi-core commodity systems and the low-latency, high-bandwidth of 10Gb Ethernet and faster networks. AMPS is designed to detect and take advantage of the capabilities of the hardware of the system on which it runs.

AMPS does more than just route and deliver messages. AMPS was designed to lower the latency in real-world messaging deployments by focusing on the entire lifetime of a message from the message's origin to the time at which a subscriber takes action on the message. AMPS considers the full message lifetime, rather than just the "in flight" time, and allows you to optimize your applications to conserve network bandwidth and subscriber CPU utilization -- typically the first elements of a system to reach the saturation point in real messaging systems.

AMPS offers both topic and content based subscription semantics, which makes it different than most other messaging platforms. Some of the highlights of AMPS include:

• Topic and content based publish and subscribe

• Message queuing, including content-based filtering and configurable strategies for delivery fairness

• Client development kits for popular programming languages such as Java, C#, C++, C, Python, and JavaScript

• Built-in support for FIX, NVFIX, JSON, BSON, MessagePack, BFlat, Google Protocol Buffer and XML messages. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing message types.

• State of the World queries

• Historical State of the World queries

• Easy to use command interface

• Full Perl compatible regular expression matching

• Content filters with SQL92 WHERE clause semantics

• Built-in latency statistics and client status monitoring

• Advanced subscription management, including delta publish and subscriptions and out-of-focus notifications

• Basic CEP capabilities for real-time computation and analysis

• Aggregation within topics and joins between topics, including joins between different message types

• Replication for high availability

• Fully queryable transaction log

• Message replay functionality

• Fully-integrated authentication and entitlement system, including content-based entitlement for fine-grained control

• Optional encryption (SSL) between client and server

• Extensibility API for adding message types, user-defined functions, user-specified actions, authentication, and entitlement functionality

When the AMPS server starts, it reads the configuration file, fully expands any environment variables, and fully processes any include directives in the file. The AMPS server stores this configuration file in memory, and the fully expanded version of the file is what is provided by the Administrative Console and Galvanometer.

Changes made to the configuration file on disk after the server has started will not take effect. To apply updates to the configuration, you must restart the AMPS server.

All AMPS configuration parameters are detailed in this guide and can be found in their appropriate sections.

This section provides a walkthrough of the minimal AMPS configuration file, explains the available unit abbreviations, and demonstrates how to use environment variables to set AMPS behavior at startup. It also covers command-line options for validating and expanding configuration files, or for generating a simple configuration outline, and options for composing an AMPS configuration from other files.

AMPS provides a command line option to help an administrator quickly set up an AMPS server. In addition to the quick setup discussed in Installing and Starting AMPS, AMPS also provides the following command line options to create a basic XML configuration file. Running the following command will create a configuration file named config.xml. The generated file is a bare-bones configuration that allows AMPS to start, process JSON messages, and provide monitoring through the admin interface.

ampServer --sample-config > config.xml

The AMPS server also provides the ability to perform basic validation of the config file, using the --verify-config flag.

ampServer --verify-config config.xml

The validation process checks for errors in the configuration that would prevent AMPS from starting, and reports warnings and informational messages about the configuration file. However, the validation process does not ensure that the configuration file provided is suitable for any particular purpose.

When a configuration file uses the Include directive or uses environment variable substitution, it can be useful to produce a fully expanded file. AMPS provides a --dump-config flag for this purpose. The command produces the fully expanded file to standard output.

ampServer --dump-config config.xml  > expanded.xml

The logical operators are NOT, AND, and OR, in order of precedence. These operators have the usual Boolean logic semantics.

/FIXML/Order/Instrmt/@Sym = 'IBM' OR /FIXML/Order/Instrmt/@Sym = 'MSFT'

As with other operators, you can use parentheses to group operators and affect the order of evaluation.

(/orderType = 'rush' AND /customerType IN ('silver', 'gold') ) OR /customerType = 'platinum'

AMPS supports the arithmetic operators +, -, *, /, %, and MOD in expressions. The result of arithmetic operators where one of the operands is NULL is undefined and evaluates to NULL.

AMPS distinguishes between floating point and integral types. When an arithmetic operator uses two different types, AMPS will convert the integral type to a floating point value as described in Numeric Types and Literals.

Examples of filter expressions using arithmetic operators:

/6 * /14 < 1000

/Order/@Qty * /Order/@Prc >= 1000000

AMPS numeric types are signed, and the AMPS arithmetic operators correctly handle negative numbers. The MOD and % operators preserve the sign of the first argument to the operator. That is, -5 % 3 produces a result of -2, while 5 % -3 produces a result of 2.

Software Requirements

The AMPS server is supported on the following platforms:

• Linux 64-bit (2.6 kernel or later) on x86 compatible processors

The AMPS distribution contains all of the supporting libraries and dependencies needed to run on a typical Linux server installation: no further software is required.

Some utilities provided with the AMPS server have additional dependencies. These utilities are not required to run the server, but can make it easier to troubleshoot and test on the system that hosts the AMPS instance:

• spark, a basic command line client that supports a subset of AMPS functionality, requires Java 1.7 or later.

• The utilities for inspecting AMPS files (amps_sow_dump, amps_clients_ack_dump, and so on) require a Python installation.

• amps-grep requires a Python installation.

AMPS expressions are designed to work exactly as expected if you are familiar with XPath path specifiers and SQL-92 predicates. This section describes in detail how AMPS evaluates the syntax, operators, and functions available in the AMPS expression language.

AMPS expressions combine the following elements:

• Identifiers specify a field in a message. When evaluating an expression, AMPS replaces identifiers with values from the message or set of messages being evaluated.

• Literal values are explicit values in an AMPS expression, such as 'IBM' or 42.

• Operators and functions such as =, <, >, *, and UNIX_TIMESTAMP().

Every AMPS expression produces a value. The way that AMPS uses the value depends on the context in which AMPS evaluates the expression. For example, if the expression is used for a filter, the message is considered to match the filter when the expression returns true. When an expression is used to project a field, the result of the expression is used as the value of the projected field.

One of the core features of AMPS is the ability to persist the most recent update for each distinct message published to a given topic. The State of the World (SOW) can be thought of as a database where messages published to AMPS are filtered into topics, and where the topics store the latest update to each distinct message. The SOW gives subscribers the ability to quickly resolve any differences between their data and updated data in the SOW by querying the current state of a topic or any set of messages inside a topic. Topics recorded in the SOW are also used for caching data, providing "point in time" snapshots of active data flows, providing key/value stores over data flows, and so on. Topics recorded in the SOW are the underlying sources for AMPS aggregation and analytics capabilities, and the ability to store the previous state of a message is the foundation of advanced messaging features such as delta messaging and out of focus notifications.

AMPS also provides the ability to keep historical snapshots of the contents of the SOW, which allows subscribers to query the contents of the SOW at a particular point in time and replay changes from that point in time.

AMPS can maintain the SOW for a topic in a persistent file, which will be available across restarts of the AMPS server. The SOW can also be transient, in which case the state of the SOW does not persist across server restarts.

Topics do not keep the current values in the SOW by default. To provide this capability for a topic, you must configure AMPS to maintain the topic in the SOW by adding a definition for the Topic to the SOW section of the AMPS configuration file.

Thank you for choosing the Advanced Message Processing System (AMPS) from 60East for evaluation.

This guide provides information to help you evaluate AMPS for your application.

This guide covers the following topics:

AMPS is a rich message delivery system. At the core of the system, the AMPS engine is highly-optimized for publish and subscribe delivery. In this style of messaging, publishers send messages to a message broker (such as AMPS) which then routes and delivers messages to the subscribers. "Pub/Sub" systems, as they are often called, are a key part of most enterprise message buses, where publishers broadcast messages without necessarily knowing all of the subscribers that will receive them. This decoupling of the publishers from the subscribers allows maximum flexibility when adding new data sources or consumers.

AMPS can route messages from publishers to subscribers using a topic identifier and/or content within the message's payload. For example, in the figure above, there is a Publisher sending AMPS a message pertaining to the LN_ORDERS topic. The message being sent contains information on Ticker "IBM" with a Price of 125, both of these properties are contained within the message payload itself (i.e., the message content). AMPS routes the message to Subscriber 1 because it is subscribing to all messages on the LN_ORDERS topic. Similarly, AMPS routes the message to Subscriber 2 because it is subscribed to any messages having the Ticker equal to "IBM". Subscriber 3 is looking for a different Ticker value and is not sent the message.

Crank Up the AMPS

This chapter is for users who are new to AMPS and want to quickly get a simple instance of AMPS running. This chapter will describe how to install AMPS on a Linux system, describe the layout of the AMPS distribution, and use the included spark command line AMPS client to send and receive a simple message. If you are on a Windows system without easy access to a Linux installation, a section at the end of the chapter includes information on configuring a Linux virtual machine.

This section covers the following topics:

Interacting with AMPS Using Spark

AMPS provides the utility as a command line interface to interacting with an AMPS server. spark provides many of the capabilities of the AMPS client libraries through this interface. The utility lets you execute AMPS commands from the command line. spark is a Java application, and requires Java runtime environment version 1.7 or later on the system.

spark is most commonly used for ad hoc testing or simple maintenance tasks. For more complicated tasks or more sophisticated maintenance, 60East recommends using one of the client libraries (such as the AMPS Python Client).

To test spark with the sample configuration, run the following command:

This command tests connectivity to the AMPS server running at port 9007 on the local system. It confirms that the server is listening on that port using the default protocol for AMPS and accepts JSON messages on that port. The command should produce output like the following:

You can read more about and other useful tools for troubleshooting AMPS in the chapter of the.

AMPS SOW topics persist the most recent update for each message, in the same way that a relational database stores the current state of each record. For performance, AMPS SOW topics store the full content of the message verbatim rather than storing a deserialized or "shredded" version of the message.

Each distinct record in a SOW topic is identified by a SOW key. AMPS treats the SOW key for a SOW topic the same way a relational database uses the primary key for a table: each distinct SOW key value is a unique message.

There are several ways to create a SOW key for a message. Each topic defines one of the following strategies:

• Most applications specify that AMPS will calculate a SOW key based on the content of the message. The configuration of the topic specifies the field, or fields, to be used for the key.

• A topic can also be configured to require that a publisher provide a SOW key for each message when publishing the message to AMPS. This is less commonly used than determining the key based on the message content, however, since this strategy does not require any explicit configuration, AMPS will default to this strategy for identifying messages if no other strategy is specified.

• AMPS also supports the ability for custom SOW key generation logic to be defined in an AMPS module, which will be invoked to generate the SOW key for each message.

Although the SOW key is derived from the content of the message in many cases, the SOW key itself is metadata, distinct from the content of the message. Each record in a SOW topic has a distinct SOW key, which is stored with the record.

For example, the diagram below shows how AMPS computes the SOW key for a topic named ORDERS with a key definition of /orderId. For each publish to the topic, AMPS uses the value of the key fields (in this case, simply /orderId) to compute a SowKey, then uses that SowKey to insert or update the appropriate record.

At any point in time, applications can issue SOW queries to retrieve all of the messages that match a given topic and content filter. When a query is executed, AMPS will test each message in the SOW against the content filter specified and all messages matching the filter will be returned to the client. The topic can be a literal topic name or a regular expression pattern. For more information on issuing queries, please see in the .

A SOW query is atomic. Updates that occur while the query is running, or while a client is receiving results, are not returned as part of the query.

Spark: Basic SOW Query Example

Here's how to use spark to query the current state of an AMPS SOW topic.

This example assumes that:

• You have configured a topic named test-sow in the AMPS server of message type JSON.

• The test-sow topic uses the /id field of the message to calculate the key for the topic.

To retrieve the current state of the topic, an application issues the sow command. Unlike a subscription, which stays active until it is explicitly stopped (or the application disconnects), the sow command provides results for a specific point in time. Once the results are returned, the command is over.

First, publish a message or two to the test-sow topic:

1. Open a new terminal in your Linux environment.

2. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to send a single message to AMPS:

3. spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the publish command, spark reads the message from the standard input and publishes the message to the JSON topic test-sow. The command produces output similar to the following line (the rate calculation will likely be different:

4. When the publisher sends the message, AMPS parses the message to determine the value of the Key fields in the message, and then either inserts the message for that key, or overwrites the existing message with that key.

5. You can publish any number of messages this way. Each distinct id value will create a distinct record in the topic.

Next, retrieve the current contents of the topic:

1. Open a new terminal in your Linux environment.

2. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to retrieve the contents of the topic:

3. spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). spark then sends the sow command to AMPS. This command requests the current contents of the test-sow topic. Since the command is finished once the query is complete, spark will exit when the query results are complete.

4. spark shows the current contents of the topic. Notice that the output is strictly the message data, separated by newline characters. spark does not show any of the metadata for a message.

Learning More

Now that you understand the basics of how AMPS works, you have two potential paths forward in your usage of the product:

• On one path, you may want to learn how to configure, deploy, and administer your own instance of AMPS. For this path, see the , which provides complete information for system administrators who are responsible for the deployment, availability and management of data to other users.

• Alternatively, you may need to develop an application to work with AMPS, using one of the Developer Guides for Java, Python, C++, or C#. For this path, visit the developer page at to download one of the evaluation kits.

The following sections provide more information about each of these paths and also briefly describe some use cases for AMPS.

Operation and Deployment

In preparing to deploy your instance of AMPS, you must size your host environment according to multiple dimensions: memory, storage, CPU, and network. The chapter in the provides guidelines and best practices for configuring the host environment. The chapter also specifies recommended settings for running AMPS on a Linux operating system.

When preparing to deploy AMPS, consult the Deployment Checklist whitepaper, available on the 60East website.

Advice on preparing to deploy AMPS in production is available under your support agreement from . 60East provides review of configuration and application architecture on demand, and new deployments are especially encouraged to take advantage of this review.

Application Development

Each language-specific Development Guide explains how to install, configure, and develop applications that use AMPS. In order to develop applications using an AMPS client, you must understand the basic concepts of AMPS, such as topics, subscriptions, messages and SOW.

You will also need an installed and running AMPS server to use the product. Although you can type and compile programs that use AMPS without a running server, you will get the most benefit by running the programs against a working server. Visit the 60East website at for an evaluation version of AMPS.

Thank you for choosing the Advanced Message Processing System (AMPS) from 60East Technologies. AMPS is a feature-rich message processing system that delivers previously unattainable low-latency and high-throughput performance to users. AMPS provides both publish-and-subscribe messaging and high-performance message queuing.

AMPS is designed to help you quickly and easily develop and deploy data-intensive applications with demanding requirements for low latency and high performance.

AMPS combines aspects of a traditional message bus, message queue, database, view server, analytics and event processing engine. The features that AMPS provides are designed to be easy to use, to work well together, and to provide high performance.

Documentation Resource Overview

The 60East documentation is intended to be used with a working (development) environment of AMPS available so that you can quickly explore the concepts discussed.

60East recommends starting with the guide to become familiar with AMPS, and then reading the sections of the AMPS User Guide for the features that your application will use.

• The provides an overall introduction to AMPS, including information on setting up a development environment, the basic concepts and features of AMPS, and general advice on which features . The 60East documentation is intended to be used with a working (development) environment of AMPS available so that you can quickly explore the concepts discussed.

• The provides advice on evaluating AMPS, included a suggested evaluation process, tips on monitoring and measuring performance in an evaluation environment, and information on how to effectively partner with 60East on an evaluation of AMPS.

• The AMPS User Guide -- this guide -- provides a complete overview of AMPS features, covering instance deployment, administration, and configuration. It also explains the AMPS configuration file and the options for defining instance behavior.

• The is a short document providing recommendations for deploying AMPS into a shared environment, whether that environment will be used for production, test, or development.

These guides cover the general features of AMPS. This site provides additional guides, such as guides for developing applications with AMPS, a guide to the statistics available for monitoring, and so on.

Resources for Developers

For developers, becoming familiar with the Developer Guide for the AMPS Client library that you will be using is also recommended. The contains reference material and links to download client libraries. Full source code (including example applications) is available for all client libraries. For many client libraries, 60East also includes pre-built binaries and makes binary distributions available through popular package management sites. Notice, however, that the pre-built distributions do not contain documentation, source code, or examples.

For developers, 60East also provides an that describes the commands to the AMPS server and responses from the AMPS server. Once you are familiar with the features you will use, as described in the user guide, the Developer Guide for your client library of choice and the AMPS Command Reference provide details on how an application communicates with the AMPS server.

To create a production configuration of AMPS, you configure the instance to meet the needs of the application (or applications) that will use the instance.

An overview of the most commonly used features is available in the guide. This guide, the AMPS User Guide provides detailed descriptions of those features, including the required and optional configurations for each.

Typically, all instances of AMPS will configure:

• The instance (this is required).

• The for the instance, to make monitoring available. This typically includes setting a path to persist the instance statistics database.

• for the instance (at a minimum of info level for production instances, typically at trace level for development, testing, or UAT instances).

• One or more to allow incoming connections to the AMPS server.

• to create a for the and the logs.

The ampServer binary will produce a minimal sample configuration to stdout if it is run with the --sample-config flag that shows a minimum configuration. Options that require site-specific information (for example, the path to the statistics database or log files) are commented out in the sample.

Instances of AMPS may then add configuration to take advantage of advanced messaging features (such as the , , the ability to , and so on), to add resiliency by (typically required for ), and so on.

One thing that differentiates AMPS from classic messaging systems is its ability to route messages based on message content. Instead of a publisher declaring metadata describing the message for downstream consumers, the publisher can simply publish the message content to AMPS and let AMPS examine the native message content to determine how best to deliver the message.

The ability to use content filters greatly reduces the problem of oversubscription that occurs when topics are the only facility for subscribing to message content. The topic space can be kept simple by using content filters to deliver only the desired messages. The topic space can reflect broad categories of messages and does not have to be polluted with metadata that is usually found in the content of the message. In addition, many of the advanced features of AMPS such as out-of-focus messaging, aggregation, views, and SOW topics rely on the ability to filter content.

Content-based messaging is somewhat analogous to database queries that include a WHERE clause. Topics can be considered tables into which rows are inserted (or updated). A subscription is similar to issuing a SELECT from the topic table with a WHERE clause to limit the rows which are returned. Topic-based messaging is analogous to a SELECT on a table with no limiting WHERE clause.

AMPS uses a combination of XPath-based identifiers and SQL-92 operators for content filtering. Some examples are shown below:

Example Filter for a JSON Message:

Example Filter for an XML Message:

Example Filter for a FIX Message:

For more information about how content is handled within AMPS and the syntax of AMPS filters, details are presented at and .

AMPS provides the ability to perform atomic subscription replacement. This allows you to replace the filter, change the topic, or update the options for a subscription.

The most common use for this capability is for an application to change the filter for a subscription. For example, a GUI that is providing a view of a set of orders may need to add or remove an order from the set of orders being displayed. By replacing the content filter with a filter that tracks the updated set of orders, the application can do this without missing messages, getting duplicate messages, or having to manage more than one subscription.

Replacing a filter is an atomic operation. That is, the application is guaranteed not to miss messages that are in both the original and replacement subscription, and is guaranteed to receive all messages for the new subscription as of the point at which the replacement happens.

To replace a subscription, applications re-submit the subscription using the subscription ID of the previous subscription. See the Developer Guide of the client library you are using and the for details.

When replacing a sow_and_subscribe command (described later in the guide), AMPS runs the SOW command again and provides any messages that were not previously in the result set to the application. See the section called for details.

Notice that some options on an initial subscription limit the support for replace on a subscription. In those cases, the limitation is described when the option is described.

Replacing the Content Filter on a Subscription

AMPS allows you to replace the content filter on an existing subscription. When this happens, AMPS begins sending messages on the subscription that match the new filter. When an application needs to bring more messages into scope, this can be more efficient than creating another subscription.

For example, an application might start off with a filter such as the following:

The application might then need to bring other regions into scope, for example:

Replacing the Topic on a Subscription

AMPS allows a subscription to replace the topic on a subscription. When the topic is replaced, AMPS re-evaluates the subscription as it does when a filter is replaced. If the subscription is updated to include a topic that the user does not have permission to subscribe to, the replace operation succeeds, but no messages will be delivered on that topic.

Replacing the Options on a Subscription

AMPS allows a subscription to replace some of the options on the subscription. In this case, the subscription is evaluated as though the topic or filter has been replaced. Any new messages generated after the subscription is replaced use the new options. However, AMPS does not replay or re-query previous messages to apply the options.

For example, if a sow_and_subscribe command did not previously specify Out-of-Focus tracking and adds this option, AMPS generates the appropriate Out-of-Focus messages from the replace point forward. AMPS does not recreate Out-of-Focus messages that would have previously been generated by the subscription.

If the subscription uses pagination (see ), the replacement must contain the full set of pagination options provided on the original subscription. For a paginated subscription, the replacement may not change the topic of the subscription. Instead, close the existing subscription and create a new subscription with a different topic.

AMPS includes an expression language that combines elements of XPath and SQL-92's WHERE clause. This expression language is used whenever the AMPS server refers to the contents of a message, including:

• Content filtering

• Constructing fields for message enrichment

• Creating projected fields for views

AMPS uses a common syntax for each of these purposes, and provides a common set of operators and functions. AMPS also provides special directives for message enrichment, and aggregation functions for projecting views.

For example, when an expression is used as a content filter, any message for which the expression returns true matches the content filter. When an expression is used to construct a field for message enrichment or view projection, the expression is evaluated and the result that the expression returns is used as the content of the field.

Expressions Overview

The quickest way to learn AMPS expressions is to think of each as a combination of identifiers that tell AMPS where to find data in a message, and operators that tell AMPS what to do with that data. Each AMPS expression produces a value. The way AMPS uses that value depends on where the expression is used. For example, in a content filter, AMPS uses the value of the expression to determine whether a message matches the filter. When constructing a field, AMPS uses the value of the expression as the contents of the field.

Consider a simple example of an expression used as a filter. Imagine AMPS receives the following JSON message:

Using an AMPS expression, you can easily construct a content filter that matches the message:

There are three parts to this expression. The first part, /name, is an identifier that tells AMPS to look for the contents of the name field at the top level of the JSON document. The second part of the filter, =, is the equality operator, which tells AMPS to compare the values on either side of the operator and return true if the values match. The final part of the filter, 'Gyro', is a string literal for the equality operator to use in the comparison. When an expression is used in a content filter, a message matches the filter when the expression returns true. The expression returns true for the sample message, so the sample message matches the filter.

The identifier syntax is a subset of XPath, as described in the section on . The comparison syntax is similar to SQL-92.

Notice that AMPS makes no rigid guarantees as to the number of times a given expression is evaluated or when that evaluation will take place. AMPS will evaluate the expression as needed.

AMPS expressions allow you to group parts of the expression using parentheses. Parts of an expression inside parentheses are evaluated together. 60East recommends using parentheses to group independent parts of an expression to ensure that the expression is evaluated in the expected order. For example, in this expression:

The clause /counter % 3 is evaluated first, and the result of that evaluation is compared to 0.

Within a group, elements are evaluated left to right in precedence order. For example, given the filter below:

AMPS evaluates expression2, then expression3 (since AND has higher precedence than OR), and if they evaluate to false, then expression1 will be evaluated.

AMPS does not guarantee that all parts of an expression will be evaluated if the result of an expression can be determined after only evaluating part of the expression. For example, given the expression:

AMPS only guarantees that B_FUNCTION(/b) will be evaluated ifA_FUNCTION(/a) returns false.

AMPS also provides a regular expression comparison operator, LIKE, to provide regular expression matching on string values. A pattern is used for the right side of the LIKE operator. A pattern must be provided as a literal, quoted value. For more on regular expressions and the LIKE comparison operator, please see the section on .

The string comparison operators described in the section called are usually more efficient than equivalent LIKE expressions, particularly when used to compare multiple literal patterns, or when the only purpose of the regular expression is to perform case-insensitive matching. Use LIKE operations when it is not practical to represent the filter condition with the string comparison operators.

AMPS contains support for a ternary conditional IF operator which allows for a Boolean condition to be evaluated to true or false, and will return one of the two parameters. The general format of the IF statement is:

In this example, the BOOLEAN_CONDITIONAL will be evaluated, and if the result is true, the VALUE_TRUE value will be returned otherwise the VALUE_FALSE will be returned.

For example:

The above example returns a count of the total number of orders that have been placed where the symbol is MSFT and the order contains a quantity more than 500.

The IF operator can also be used to evaluate results to determine if results are NULL or NaN. This is useful for calculating aggregates where some values may be NULL or NaN. The NULL and NaN values are discussed in more detail in the section.

For example:

AMPS allows applications to explicitly remove records from a SOW topic using the sow_delete command.

When removing records from a SOW, there are three different ways to indicate which message, or messages, will be deleted:

1. Using a content filter. AMPS will delete all messages in the SOW that match the content filter. To delete every message in the SOW, use the special filter 1=1 to indicate that the filter is true for every message, regardless of the contents of the message. (In essence, AMPS runs a query to locate the records to be deleted, and then deletes the matching records.)

2. Using the SOW key assigned to the message. AMPS accepts a list of SOW keys, and will remove the messages indicated by those SOW keys.

3. Using message data. The application provides message data with the sow_delete command. AMPS parses the message data to determine the SOW key for the record that would be updated if the command were a publish, and deletes that record (if one exists). Notice that if the topic is configured so that publishers must provide the SOW key, the key cannot be derived from the data, which means that using message data to delete messages may not produce the expected results.

When a record is removed from the SOW, AMPS sends an out-of-focus (OOF) message to any subscriptions that have requested OOF notifications. AMPS also updates any views that use the SOW topic, and the record will be removed from conflated topics at the next conflation interval.

When the SOW is configured with the History option to enable historical queries, the sow_delete command removes the message from the current set of messages in the SOW. The command does not remove previously saved versions of the message: the historical state of the SOW is unaffected by the sow_delete.

The most efficient way to delete a specific message or specific set of messages is to use the SOW key that AMPS assigns, when that key is available. You can provide these keys in the SowKeys header (a delete by keys), or by providing a filter expression that will be evaluated as a query on the primary key or a hash index. See the section for details on how AMPS determines if a hash index or primary key can be used for a filter.

When the SOW delete provides an example message to be deleted, AMPS parses that message to determine the SOW key and then uses that to key to delete the message, which is also relatively efficient.

Removing a message from a Topic in the State of the World removes the message from that Topic and notifies any View or ConflatedTopic that depends on this topic that the message has been removed (see for details on creating a View, see for details on creating a conflated topic). Removing a message from a Topic adds the delete command to the transaction log, but does not remove messages stored in the transaction log (see ).

If the Topic contains History, the sow_delete affects the current value of the Topic but does not remove previous state. AMPS will remove records that have not been current for longer than the retention Window, as described in the section.

AMPS includes support for a wide variety of message types, as well as the ability to develop custom message types and to send binary payloads. This section focuses on JSON as the main message type used for samples in this guide. We use JSON for the guide because the format is simple, easily readable, and already in use in many environments.

JSON format is a simple, standardized message format. JSON has two basic constructs:

• Objects that consist of key / value pairs

• Arrays of values

JSON supports hierarchical construction: the value for a key can be a single value, an array of values, or another set of key/value pairs. For example, the following JSON message includes two nested sets of key value pairs. Notice that a key only needs to be unique within each set of values -- the name value for the ship does not conflict with the name value for the character.

{
    "id" : 73,
    "character" : {
        "name" : "Han Solo",
        "occupation" : "smuggler",
        "ship" : {
            "name"  : "Millennium Falcon",
            "speed" : ".5 past light speed",
            "cargo" : [ "widgets", "baskets", "spice"]
        }
    }
}

Many AMPS applications use JSON as the payload. In addition, the amps protocol used to send commands to AMPS represents commands in a simplified subset of JSON. For example, a publish command might look like:

{"c":"publish","t":"test-topic"}{ "id" : 1, "message" : "Hello, World!" }

The command to AMPS, using the amps protocol, can be treated as a JSON document which contains the header information for AMPS -- in this case, a publish to the topic test-topic. The header is followed by the message body, the payload of the command.

While the amps protocol is implemented as a subset of JSON, you can use any message type with the amps protocol. The header for the command will still be JSON, while the body can be in the message type of your choice, as in the sample below, which publishes to an XML topic:

{ "c":"publish","t":"xml-topic"}<example><id>1</id><message>Hello, world!</message></example>

The AMPS client libraries create and parse AMPS headers. For example, the publish method in the AMPS client libraries creates the appropriate header for a publish command based on the provided parameters.

Your applications use the Message and Command interfaces of the AMPS client libraries to work with the AMPS headers. There is no need for your application to parse or serialize the AMPS headers directly.

The AMPS server runs on 64-bit Linux operating systems. If you do not have access to a Linux system or a recent version of Windows, 60East recommends creating a Linux virtual machine to host the instance of AMPS. This is a convenient option for development systems and allows you to easily experiment with different AMPS configurations on a dedicated system.

This section provides general information for creating a virtual machine image for use as a local development or evaluation environment.

This section assumes that you are familiar with Linux and the virtualization program you will be working with. It focuses on information specific to AMPS.

Using Windows Subsystem for Linux 2

If your development system is running a recent version of Windows, then Windows Subsystem for Linux 2 is a good option for developing with AMPS. Getting AMPS running is simply a matter of starting a Linux shell, downloading AMPS, and following the directions for Linux.

Notice that Windows Subsystem for Linux 2 does not provide access to some of the functionality that the AMPS server expects: in particular, the AMPS NUMA subsystem may not be able to determine the physical processor layout and may report warnings on startup. Nevertheless, this can be a very good option for doing AMPS evaluation and development on a Windows system.

Creating a Virtual Machine Image

When creating the virtual machine image, 60East recommends the following parameters:

• x64 processor

• At least 4GB of memory allocated to the virtual machine

• Minimum of 120GB drive space (most will be consumed by the operating system image)

• At least 2 virtual processors

AMPS itself can run with less memory, processor, and disk capacity than recommended here. However, these settings will typically provide reasonable performance and enough capacity to do basic development work.

Virtual Box Settings

When installing AMPS on Virtual Box, 60East strongly recommends setting the network hardware emulation to use the Paravirtualized network adapter (virtio-net). For recent versions of Linux, performance is dramatically improved (even over the loopback interface) when using this setting.

Choosing a Linux Distribution

AMPS runs well on any Linux distribution that meets the basic requirements. The Ubuntu Linux distribution is a good choice, and is frequently used by both customers and the 60East developers as a development workstation environment. Visit https://www.ubuntu.com/download to download the latest released version of Ubuntu.

Whichever distribution you choose, 60East recommends that you download the .iso file and use that file to install the operating system.

Installing the Linux Distribution

AMPS itself doesn't require anything beyond a basic operating system distribution. For the best experience while you are evaluating and getting to know AMPS, 60East recommends that you choose a profile optimized for software development or desktop use.

Select the following additional packages if your distribution does not already install them:

• Python 2.6/2.7 or 3. The utility scripts in the AMPS distribution require Python.

• Java runtime environment (1.7 or more recent). The spark command line AMPS client is written in Java, and requires a JRE. This guide assumes that you have a JRE available, and presents examples using spark.

• g++, gdb, and your IDE of choice if you will be developing C++ applications with AMPS.

• A web browser such as Firefox or Google Chrome

Filesystem Considerations

AMPS is designed and tested to use a Linux-based filesystem such as ext4 or filesystems that provide full native Linux filesystem semantics (for example, using nfs mounted filesystems for testing or archival purposes).

Mounting another type of filesystem (for example, an NTFS volume) in a VM, container, or WSL 2 may cause failures or unexpected results, since that approach may not provide all of the filesystem operations that AMPS uses.

When using a container, VM, or WSL2, make sure that AMPS and the files that AMPS creates are hosted on filesystems that support Linux file operations, in particular, that a process running under the Linux environment can memory map files hosted on that filesystem.

Next Steps

Once you have created the virtual machine image and installed your Linux distribution of choice, you can install and start AMPS as described in Installing AMPS.

Storing a topic in the State of the World is most useful when your application needs to use the current state of the data being tracked. Storing a topic in the State of the World can be especially useful if your application would benefit from automatically receiving updates as soon as they are made (described in more detail in the Atomic Query and Subscribe topic).

Below you will find common uses of a SOW topic, which include examples of practical use cases:

• An application needs the current state of a record, but does not need to recreate the message flow that created that record:

  An order fulfillment system presents a view of all currently pending orders when the application starts up.

• An application needs the current state of a record or set of records, even when the topic is high-volume or quickly changing:

  A warehouse management application locates the current inventory level for a product.

  A taxi dispatch company locates taxis currently within 10 blocks of an event.

• An application wants to be able to publish incremental updates to a record:

  A customer updates her shipping address. All pending orders for the customer are automatically updated without affecting any other information in the order, and processors working with the orders are notified of the change.

• An application wants to receive only the changed fields of a record:

  A mobile application displays the status of an order as the order progresses through the stages of validation: the application receives only the identifier for the record and the changed fields.

• An application needs the AMPS server to calculate values based on the current values of a record or set of records:

  A management console constantly calculates the real-time value of pending orders. The console uses a view, calculated based on data saved in a topic in the SOW.

• An application wants to store application state for quick retrieval:

  An order processing system publishes statistics on each step of the process: a separate process monitors and aggregates those statistics. The SOW also maintains historical state for the topic so the monitor can easily recreate a snapshot of the state at a point in time and compare day over day status.

Of course, the examples above are just a small sample of the ways the AMPS SOW can be used.

When a topic is recorded in the SOW, an application can request the current state of the topic and simultaneously subscribe to updates from the topic. In this case, AMPS first delivers all of the messages that match the query and then provides any update to a record that matches the query. AMPS guarantees that no updates are missed or duplicated between the query and the subscription. As with a simple query, AMPS will test each message currently in the SOW against the content filter specified and all messages matching the filter will be returned to the client. When the query begins, AMPS enters a subscription with the provided filter. After the query completes, AMPS delivers messages from the subscription. In the event that a record is updated while the query is running, AMPS saves the update and delivers it immediately after the query completes.

As with a simple SOW query, the topic can be a literal topic name or a regular expression pattern. For more information on issuing queries, please see Querying the State of the World in the AMPS User Guide.

Spark: Basic SOW Query and Subscribe Example

Here's how to use spark to query the current state of an AMPS SOW topic and subscribe to updates.

This example assumes that:

• You have configured a topic named test-sow in the AMPS server of message type JSON.

• The test-sow topic uses the /id field of the message to calculate the key for the topic.

To retrieve the current state of the topic and subscribe, an application issues the sow_and_subscribe command. Since the command includes a subscription, the command stays active until it is explicitly stopped (or the application disconnects).

First, publish a message or two to the test-sow topic:

1. Open a new terminal in your Linux environment.

2. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to send a single message to AMPS:

   Copy

   $ echo '{"id":1,"note":"Crank it up with a SOW!"}' | \
     $AMPS_DIR/bin/spark publish -server localhost:9007 \
     -type json -topic test-sow

3. spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). With the publish command, spark reads the message from the standard input and publishes the message to the JSON topic test-sow. The command produces output similar to the following line (the rate calculation will likely be different:

   Copy

   total messages published: 1 (333.33/s)

4. When the publisher sends the message, AMPS parses the message to determine the value of the Key fields in the message, and then either inserts the message for that key, or overwrites the existing message with that key.

5. You can publish any number of messages this way. Each distinct id value will create a distinct record in the topic.

Next, retrieve the current contents of the topic:

1. Open a new terminal in your Linux environment.

2. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to retrieve the contents of the topic:

   Copy

   $ $AMPS_DIR/bin/spark sow_and_subscribe -server localhost:9007 \
     -type json -topic test-sow

3. spark automatically connects to AMPS and sends a logon command with the default credentials (the current username and an empty password). spark then sends the sow_and_subscribe command to AMPS. This command requests the current contents of the test-sow topic and creates a subscription to the topic.

4. spark shows the current contents of the topic. Notice that the output is strictly the message data, separated by newline characters. spark does not show any of the metadata for a message.

5. spark remains running after the query completes, waiting for new publishes to arrive.

Publish more messages (or updates to the existing messages) to the topic. In the terminal you opened to publish the first messages:

1. Use the following command (with AMPS_DIR set to the directory where you installed AMPS) to send a message to AMPS:

   Copy

   $ echo '{"id":1,"note":"Crank it up with a SOW!"}' | \
     $AMPS_DIR/bin/spark publish -server localhost:9007 \
     -type json -topic test-sow

2. Notice that the subscription receives the message.

If you close the subscriber and re-run it, you will see that the second time the subscriber runs, it receives the updated messages in the query and, again, waits for changes to arrive.

AMPS runs on any 64-bit Linux system. For best performance in a development environment, 60East recommends that the system have a minimum of 4GB of memory available.

For basic functional evaluation and development, AMPS runs well in a virtual machine, in a container, or in a WSL2 shell on Windows, as well as on a Linux host.

The Introduction to AMPS includes information on how to set up a basic development environment for AMPS.

Product Overview

AMPS is designed to help you quickly and easily develop and deploy data-intensive applications with demanding requirements for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems.

AMPS is based on an incredibly fast messaging engine that supports multiple messaging paradigms, as well as providing persistent current value caching (effectively, an integrated database), content filtering and continuous query, historical replay, aggregation and analytics, message enrichment, focus tracking, partial updates and change tracking, and more.

Furthermore, AMPS is designed and engineered specifically for next generation computing environments. The architecture, design and implementation of AMPS allows the exploitation of parallelism inherent in emerging multi-socket, multi-core commodity systems and the low-latency, high-bandwidth of 10Gb Ethernet and faster networks. AMPS is designed to detect and take advantage of the capabilities of the hardware of the system on which it runs.

AMPS was designed to improve performance and reduce latency in real-world messaging deployments by focusing on the entire lifetime of a message from the message's origin to the time at which a subscriber takes action on the message. AMPS considers the full message lifetime, rather than just the "in flight" time, and allows you to optimize your applications to conserve network bandwidth and subscriber CPU utilization -- typically the first elements of a system to reach the saturation point in real messaging systems.

Understanding AMPS Features and Scenarios

For an overview of the features of AMPS, see the Overview of AMPS in the Introduction to AMPS.

To understand which features are most commonly used for a given scenario or application pattern, see the Scenario and Feature Reference. This provides a quick guide to help you focus on learning the features that are most relevant to the problem at hand.

For example, to distribute work across a set of independent processors, you would use AMPS message queues, whereas if your application requires a content-aware last value cache, you would use a Topic in the AMPS State of the World.

Evaluation Process Outline

60East provides access to technical support during the evaluation process.

To prepare to evaluate AMPS, 60East recommends the following process:

• Engage 60East support with a description of the evaluation goals, and to put in place any agreements necessary to make evaluation go more smoothly (such as mutual non-disclosure agreements).

• Define the detailed goals of the feasibility phase of the evaluation. Typically, these break down into:

  • Functional Capability - This represents what the evaluation project needs to be able to do. (For example: accurately receive NFVIX messages and deliver them to the appropriate subscriber or subscribers while maintaining the ability to replay 30 days of history at any point.)

  • Performance Goals - This represents the service level for the evaluation. (For example: Maximum latency to reach client processing of 250ms for current messages, no more than 1s to first message for beginning a replay at an arbitrary depth in history.)

  • Capacity Goals - This represents the total volume of work being evaluated. (For example: The system needs to reach capability and performance goals while processing 10,000 messages per second ingestion.)

• Develop initial design and testing plan. During this process, teams use the AMPS documentation to understand how to use AMPS to meet the goals of the evaluation. Teams engage 60East support as necessary to resolve any questions that emerge or get advice on tradeoffs and options to achieve the evaluation goals.

• Once a design and testing plan is complete, review the design and testing plan with the 60East engineering team, adjusting as necessary.

• Implement the design and tests. If questions or issues emerge, consult with 60East support to resolve the issues.

• Test and review the results with 60East.

• Evaluate the deployment and maintenance phase of the evaluation. Typically, this involves:

  • Operations and Performance at Scale - Evaluate the application in a production-like environment at scales at or near production volumes.

  • Maintenance Goals - Develop and test the maintenance, support, and upgrade plan as described in the 60East deployment checklist.

• Follow up on any open issues and complete the evaluation.

This manual is an introduction to the 60East Technologies AMPS product. It assumes that you have a working knowledge of Linux and uses the following conventions.

The AMPS documentation also includes the following types of notes:

Additionally, here are the constructs used for displaying content filters, XML, code, command line, and script fragments.

(expr1 = 1) OR (expr2 = 2) OR (expr3 = 3) OR (expr4 = 4) OR (expr5 = 5) OR (expr6 = 6) OR (expr7 = 7) OR (expr8 = 8)

Command lines will be formatted as in the following example:

$ find . -name *.java

This manual is divided into the following parts:

• Part One presents introductory material and a brief overview of AMPS

• Part Two explains the features of AMPS, including information on the following features:

  • Subscribe and Publish, the basic building blocks of AMPS applications

  • The expression language and functions used to take advantage of the content-aware features of AMPS are covered in AMPS Expressions and AMPS Functions

  • Record and Replay Messages using the AMPS transaction log

  • Competitive message consumption with Message Queues

  • The Message Types that AMPS supports for content-aware processing

  • Current value caching and database functions using State of the World (SOW) topics

  State of the World topics enable many of the other advanced features in AMPS, such as:

  • Aggregation and Analytics

  • Querying the State of the World

  • Out-of-Focus Messages

  • State of the World Message Enrichment

  • Incremental Message Updates

  • Receiving Only Updated Fields

  This section also contains detailed chapters on specific topics, such as the AMPS filter language. Both application developers and administrators should become familiar with this section.

• Part Three discusses AMPS deployment and operations, including:

  • Running AMPS as a Linux Service

  • Logging

  • Event Topics

  • Utilities

  • Monitoring AMPS

  • Configuring AMPS for Automation with Actions

  • Replicating Messages Between Instances

  • Highly Available AMPS Installations

  • Operation and Deployment

  • Securing AMPS

  • Troubleshooting AMPS

  This section is most useful for those with a focus on AMPS operations, although the information presented here is helpful for developers who want to design high-performance, high-availability applications that are easy to deploy and maintain.

• Additional chapters provide reference information:

  • Optionally-Loaded Modules describes special-purpose modules that are included in the AMPS distribution but are not loaded by default

  • File Format Versions lists the file formats used by each AMPS version

To install AMPS, unpack the distribution for your platform where you want the binaries and libraries to be stored. For the remainder of this guide, the installation directory will be referred to as $AMPSDIR as if an environment variable with that name was set to the correct path.

Within $AMPSDIR are the following sub-directories:

The AMPS engine binary is named ampServer and is found in $AMPSDIR/bin. Start the AMPS engine with a single command line argument that includes a valid path to an AMPS configuration file. You use the configuration file to enable and configure the AMPS features that your application will use. This guide discusses the full set of configuration options for each feature.

The AMPS server generates a minimal sample configuration file with the --sample-config option. You can save the sample configuration file to $AMPSDIR/amps_config.xml with the following command line:

$AMPSDIR/bin/ampServer --sample-config > $AMPSDIR/amps_config.xml

Once you have a configuration file saved to $AMPSDIR/amps_config.xml you can start AMPS with that file as follows:

$AMPSDIR/bin/ampServer $AMPSDIR/amps_config.xml

If your first start-up is successful, you should see AMPS display a simple message similar to the following to let you know that your instance has started correctly.

AMPS A.B.C.D.973814.e1a57f7 - Copyright (c) 2006-202X 60East Technologies Inc.
(Built: XXXX-YY-ZZT00:26:45Z)

The version numbers and dates will be appropriate for the version that you've started.

If you see this, congratulations! You have successfully cranked up the AMPS!

Command Line Options

The AMPS server binary supports the following command line options:

For production applications, AMPS configuration files can become large and complicated. In many cases, different instances of an AMPS server need to reuse the same definitions. For example, both servers in a High-Availability pair may need to use the same queue and SOW definitions.

To help you manage complicated configurations and more easily keep configuration consistent on different servers, AMPS allows you to include external files in the configuration file by using the Include directive.

For example, you could use this for a High-Availability pair to include a file that defines the queue, transaction log, and topic definitions. Both instances could include exactly the same file for those definitions, while having different instance names and port numbers.

When AMPS loads a configuration file that contains an Include directive, AMPS follows this process:

• Load and parse the configuration file

• If the file contains any Include directives, load and parse the files specified by those directives. If the included files contain Include directives, load and parse the files specified by those directives (and so forth until all Include directives have been processed).

• Once all files have been loaded and parsed, replace the Include directives in the original files with the parsed files.

AMPS does not process the configuration file until all of the Include directives have been resolved and the files have been parsed.

A file may not be included by any file that it includes, or it is impossible for AMPS to complete the parsing process.

Since each file is individually parsed, XML entities defined in a file are not defined for the files that are listed within the Include tag by that file.

To make it easier to identify which elements of the complete AMPS configuration file have been inserted through the Include mechanism, AMPS can include comments in the assembled file that indicate the source file for configuration elements. By default, this feature is off, and XML content is included verbatim. To change the default for the instance, use the ConfigIncludeCommentDefault configuration element to enable comments, by default, for every Include in the instance. To override commenting behavior for an individual Include, use the comment attribute.

Example

Consider a configuration file with the following Logging element defined:

<AMPSConfig>
    ...

    <Logging>
        <Include comment="true">filetarget.xml</Include>
    </Logging>

    ...
</AMPSConfig>

After parsing the configuration file, AMPS loads and parses the filetarget.xml file and replaces the Include element with the contents of that file.

Suppose filetarget.xml contains the following Target directive:

<Target>
    <Protocol>file</Protocol>
    <FileName>/var/log/amps-log-%n.log</FileName>
    <Level>info</Level>
</Target>

The configuration that AMPS uses will be effectively the same as if the configuration file contained the following XML:

<AMPSConfig>
    ...

    <Logging>
        <!-- Start <Include>filetarget.xml</Include> -->
        <Target>
            <Protocol>file</Protocol>
            <FileName>/var/log/amps-log-%n.log</FileName>
            <Level>info</Level>
        </Target>
        <!-- End <Include>filetarget.xml</Include> -->
    </Logging>

    ...
</AMPSConfig>

AMPS identifiers use a subset of XPath to specify values in a message. AMPS identifiers specify the value of an attribute or element in an XML message, and the value of a field in a JSON, FIX or NVFIX message. Given that the identifier syntax is only used to specify values, the subset of XPath used by AMPS does not include wildcards, relative paths, array manipulation, predicates or functions.

For example, when messages are in this XML format:

<Order update="full">
    <ClientID>12345</ClientID>
    <Symbol>IBM</Symbol>
    <OrderQty>1000</OrderQty>
</Order>

The following identifier specifies the Symbol element of an Order message:

/Order/Symbol

The following identifier specifies the update attribute of an Order message:

/Order/@update

For FIX and NVFIX, you specify fields using / and the tag name. AMPS interprets FIX and NVFIX messages as though they were an XML fragment with no root element. For example, to specify the value of FIX tag 55 (symbol), use the following identifier:

/55

Likewise, for JSON or other types that represent an object, you navigate through the object structure using the / to indicate each level of nesting.

AMPS only guarantees support for field identifiers that are valid step names in XPath. For example, AMPS does not guarantee that it can process or filter on a field named Fits&Starts.

AMPS also supports an optional bracketed field identifier syntax that extends the characters available for field names. For example, the following step name:

[/Not Xpath Name]

refers to a field name of Not Xpath Name at the root level of the message. This syntax allows spaces to be used in field names in AMPS expressions, even though this is not a valid step name in XPath. Notice that not all message types support field names with embedded spaces or other special characters. For example, the Not Xpath Name identifier is not a valid element name in XML, nor would it be a valid field name in Google Protocol Buffers.

AMPS checks the syntax of identifiers when parsing an expression. AMPS does not try to predict whether an identifier will match messages within a particular topic. It is not an error to submit an identifier that can never match due to the limitations of the message type. For example, AMPS allows you to use an identifier like /OrderQty in a filter submitted for a FIX connection, even though FIX messages only use numeric tags, or an identifier like /DataPackage/RunDate in a filter submitted for a BFlat connection, even though BFlat does not support nested elements.

The message type is responsible for constructing a set of identifiers from a message. In most cases, the mapping is simple. However, see the documentation for the message type for details, or if the mapping is unclear. For example, a composite-local message type adds the number of the part to the beginning of each XPath within the part (so, a top-level field of /name in the first part of the message has an identifier of /0/name).

The comparison operators can be loosely grouped into equality comparisons and range comparisons. The basic equality comparison operators, in precedence order, are ==, =, >, >=, <, <=, !=, and <>. The == comparison and the = comparison are treated as the same operator and produce the same results.

If these binary operators are applied to two operands of different types, AMPS attempts to convert strings to numbers. If conversion succeeds, AMPS uses the numeric values. If conversion fails because the string cannot be meaningfully converted to a number, strings are always considered to be greater than numbers. The operators consider an empty string to be NULL.

The following table shows some examples of how AMPS compares different types.

There are also set and range comparison operators. The BETWEEN operator can be used to check the range values.

For example:

/FIXML/Order/OrdQty/@Qty BETWEEN 0 AND 10000

/FIXML/Order/@Px NOT BETWEEN 90.0 AND 90.5

(/price * /qty) BETWEEN 0 AND 100000

The IN operator can be used to perform membership operations on sets of values. The IN operator returns true when the value on the left of the IN appears in the set of values in the IN clause. For example:

/Trade/OwnerID NOT IN ('JMB', 'BLH', 'CJB')

/21964 IN (/14*5, /6*/14, 1000, 2000)

/customer IN ('Bob', 'Phil', 'Brent')

The IN operator returns true for the set of records that would be returned by an equivalent set of = comparisons joined by OR. The following two statements return the same set of records:

/pet IN ('puppy', 'kitten', 'goldfish')

(/pet = 'puppy') OR (/pet = 'kitten') OR (/pet = 'goldfish)

This equivalence means that NULL values in either the field being evaluated, or the set of values provided to the IN clause, always return false.

This also means that, for string values, the IN operator performs exact, case-sensitive matching.

When using NOT IN, AMPS interprets this as a NOT unary operator applied to the IN operator. This means that the following expressions are equivalent:

/data NOT IN (1,2,3)

NOT /data IN (1,2,3)

NOT ((/data == 1) OR (/data == 2) OR (/data == 3))

AMPS maintains indices over SOW topics, views, and conflated topics to improve query efficiency.

There are two types of indices available:

1. Memo indices are created automatically when AMPS needs to use a particular field for a query. These indices maintain the value of a key, and can be used for any type of query, including regular expression queries, range queries, and comparisons such as less than or greater than. You can also request that AMPS pre-create an index of this type with the Index directive of the SOW topic configuration.

2. Hash indices are defined by the configuration for the topic, view or conflated topic. These indices maintain a hash derived from the values provided for the fields in the key. When the topic is configured so that AMPS generates the SOW key, AMPS automatically creates a hash index that contains all of the fields in the SOW Key. You can create any number of hash indexes for a SOW topic, with any combination of fields. Hash index queries are significantly faster than queries using memo indexes.

Both types of indices are maintained in memory. The section on Estimating AMPS Instance Memory Usage has more details.

A hash index can be created using any XPath Identifier in the message. For example, if you are using a composite-local message type, you can create a hash index using fields from any part of the message. If you are using an xml message, you can create a hash index that uses the XML attributes.

The values of hash indices are always evaluated as strings. Hash indices are only used for exact matches on the value of the fields or with the IN operator, and only for queries that use the exact set of fields in the hash index. Other operators or functions (for example, LIKE, !=, BETWEEN, IS NULL, IS NOT NULL, and so on) cannot use the hash index. To use a hash index, the comparison must use a literal string for comparison to specify that the comparison uses an exact string comparison and not a numeric comparison.

For example, if your configuration specifies a hash index that uses the fields /address/postalCode and /customerType, a filter such as /address/postalCode = '04109' AND /customerType = 'retail' will use the hash index. A filter such as /address/postalCode = '04109' AND /customerType LIKE 'retail|remainder' will not use the hash index, since this filter uses the LIKE operator rather than exact matching. Likewise, a comparison such as /address/postalCode = 04109 will not use the hash index, since the expression requests a numeric comparison rather than a string comparison.

Starting with AMPS 5.3.1.0, AMPS will also use a hash index for a compound filter if the first clause in the filter is an IN operator that can use a hash index and the other comparisons in the filter are evaluated using the AND operator. In this case, AMPS evaluates the IN clause first, and executes the rest of the expression against the results of the IN clause. For example, a filter like /id IN ('jon', 'jim', 'joy') AND /price > 50 will use a hash index to find matching records for /id and then compare the matching records to the rest of the filter (in this case, a numeric comparison on /price). (Notice that this optimization is not available if the other comparisons use the OR operator.)

AMPS uses a hash index for filters where possible. If the filter does not meet the requirements for using a hash index, AMPS uses memo indices for the fields in the filter if those are available. If one or more of the required memo indices is not available, AMPS creates the indexes during the query.

If your application frequently uses queries for an exact match on a specific set of fields (for example, retrieving a set of customers by the /address/postalCode field), creating a hash index can significantly improve the speed of those queries.

Applications that store topics in the SOW must consider the ongoing storage needs and file management for the SOW.

There are two aspects to SOW maintenance:

1. Ensuring that the host system has enough capacity to efficiently store and manage the topics in the SOW. Capacity planning guidelines are discussed in the Capacity Planning section in the operations section of this guide.

2. Setting and implementing a data retention policy for the contents of each topic in the SOW.

The data retention policy for a topic in the SOW is determined by the needs of your application.

Consider the following questions:

• Does the topic have a data set that tends to stay at a consistent size? If so, there may be no need to explicitly manage data retention. Many AMPS applications have topics that fall into this category.

  For example, an application that uses a SOW topic to track the current price of a specific set of ticker symbols has little need to set a data retention policy. The SOW will always contain the same number of records (one for each ticker symbol), and those records will always contain data of a consistent size. The application may choose to remove a record when a symbol is removed from the set, but otherwise rely on publishers to keep the data current.

• Is the data only valid for a fixed duration relative to when the data is published? If so, Setting Per-Message Lifetime using message expiration may be a good way to manage the SOW.

  For example, an application that needs to ensure that quotes are removed from the system after 10 minutes from the time the quote is published could use SOW expiration to remove records after 10 minutes. Managing this expiration using SOW expiration may be more efficient than using an action, since messages may expire at any point in time.

• Is the data valid until a certain condition becomes true? If so, having the application remove records from the SOW that are no longer needed or configuring a Scheduled Maintenance action may be a good way to manage the SOW.

  For example, an application that needs to clear the state of the SOW every 24 hours during a maintenance window could use an action to remove those records. An application that can determine when a record is no longer needed can remove the record immediately, which means that the topic only contains data that the application needs at any given time.

Regardless of the approach an application takes, 60East recommends that every application that uses a SOW consider capacity and explicitly consider the data retention needs of each topic and the application.

The AMPS server and the AMPS client libraries provide various options for recovering and resuming subscriptions. Use this cross-reference to choose the recovery strategy that best matches the needs of your application.

The scenarios above describe just a few of the most common recovery scenarios for a subscription. For recovery scenarios that aren't described above, contact 60East at http://support.crankuptheamps.com/ for advice and guidance.

Technical Support and Assistance

At 60East, the most important part of what we do is helping people deploy systems that utilize AMPS and supporting them in maintaining the ongoing and intended operation of those systems. Considering that AMPS is often used in essential systems that push the limits of hardware, network and storage capacity, we know that support is essential to help you build, deploy and maintain the kinds of cutting-edge applications that we built AMPS to handle.

During the evaluation and development stages, we encourage you to share details with us about what you are building. This way, we can provide assistance with the design and architecture process. Once your application goes into production, use 60East support to help diagnose and correct issues that fall outside of the normal operation of the application.

The level of support you have available is dependent on your support agreement. For an outline of your specific support policies, please see your 60East Technologies License Agreement. Support contracts can be purchased through your 60East Technologies account representative.

Support Steps

You can save time if you complete the following steps before you contact 60East Technologies Support:

1. Check the documentation

   The problem may already be solved and documented in the AMPS User Guide or Configuration Guide for the product. Check the support site at http://support.crankuptheamps.com where 60East Technologies also provides answers to frequently asked support questions.

2. Isolate the problem

   If you require Support Services, please isolate the problem to the smallest test case possible. Capture erroneous output into a text file along with the commands used to generate the errors.

3. Collect your information

   • Your product version number.

   • Your operating system and its kernel version number.

   • The expected behavior, observed behavior and all input used to reproduce the problem.

   • Submit your request.

   • In your email to [email protected] include the minidump file if you have one.

The AMPS version number used when reporting your product version number follows a format listed below. The version number is composed of the following:

MAJOR.MINOR.FEATURE.HOTFIX.TIMESTAMP.TAG

Contacting 60East Technologies Support

Please contact 60East Technologies Support Services according to the terms of your 60East Technologies License Agreement. Visit the support site at http://support.crankuptheamps.com for evaluations.

Support is offered through the United States:

Other support options (such as support via phone, dedicated engineers, and so on) may be available, depending on the terms of your support agreement.

A topic is a string that is used to declare a subject of interest for purposes of routing messages between publishers and subscribers. Topic-based Publish and Subscribe (e.g., Pub/Sub) is the simplest form of Pub/Sub filtering. All messages are published with a topic designation to the AMPS engine, and subscribers will receive messages for topics to which they have subscribed.

For example, in the diagram above there are two publishers: Publisher 1 and Publisher 2 which publish to the topics LN_ORDERS and NY_ORDERS, respectively. Messages published to AMPS are filtered and routed to the subscribers of a respective topic. For example, Subscriber 1, which is subscribed to all messages for the LN_ORDERS topic will receive everything published by Publisher 1. Subscriber 2, which is subscribed to the regular expression topic ".*_ORDERS" will receive all orders published by Publisher 1 and 2. Subscriber 3, which is subscribed to all messages for the NY_ORDERS topic will receive everything published by Publisher 2.

Regular expression matching makes it easy to create topic paths in AMPS. Some messaging systems require a specific delimiter for paths. AMPS allows you the flexibility to use any delimiter. However, 60East recommends using characters that do not have significance in regular expressions, such as forward slashes. For example, rather than using northamerica.orders as a path, use northamerica/orders.

AMPS does not restrict the characters that can be present in a topic name. However, notice that topic names that contain regular expression characters (such as . or *) will be interpreted as regular expressions by default, which may cause unexpected behavior.

Topics that begin with /AMPS are reserved. The AMPS server publishes messages to topics that begin with /AMPS as described in the Event Topics section. Some versions of the AMPS client libraries may internally publish to /AMPS/devnull. Your applications should not publish to topics that begin with /AMPS, as publishes to those topics may fail.

Each topic has an associated message type. Each client connection to AMPS also has an associated message type. A given client connection can only publish to topics with the same message type, and can only receive messages from topics with the same message type.

Ad Hoc Topics

AMPS does not require explicit configuration of a topic for publishers to send messages to the topic and subscribers to receive messages from the topic. However, if there is no configuration for the topic, AMPS does not persist messages to the topic, so no features that depend on having a persisted message state (for example; replay, aggregation, State of the World, and so on) are available for that topic. The message will be delivered to subscriptions that are active when the message is published, but the message will not be persisted or retained. These "ad hoc" topics are useful for low-latency delivery of messages that are only useful at the time that they are published.

Matching Multiple Topics With Regular Expressions

With AMPS, a subscriber can use a regular expression to simultaneously subscribe to multiple topics that match the given pattern. This feature can be used to effectively subscribe to topics without knowing the topic names in advance.

Notice that a message cannot be published to a topic pattern. The topic for a given message is unambiguously specified using a literal string. From the publisher’s point of view, it is publishing a message to a topic. A publisher does not publish to a topic pattern.

When a subscription is sent to AMPS, the topic for the subscription is interpreted as a regular expression if the topic includes special regular expression characters. Otherwise, the topic must be an exact match.

Some examples of regular expressions to match a set of topics are included in the table below:

For more information regarding the regular expression syntax supported within AMPS, please see the Regular Expressions section for details.

AMPS can be configured to disallow regular expression topic matching for subscriptions. See Instance-Level Configuration for details.

AMPS guarantees that, for each AMPS instance, each subscription to a topic receives messages in the order in which AMPS received the messages (with the exception of messages that have been returned to a message queue for redelivery or the results of a query). Before a given message is delivered to a subscriber, all previous messages for that topic are delivered to the subscriber. AMPS does this by enforcing a total order across the instance for all messages received from publishers, including messages received via replication. When AMPS is using a transaction log, that order is preserved in the transaction log for the instance, and persists across instance restarts. When replaying from the transaction log, a subscriber will always receive messages in the same order in which messages were originally delivered by that instance of AMPS.

This guarantee also applies across topics for subscriptions that involve multiple topics, for all topics except views, queues, and conflated topics. Views and queues guarantee that every message on the view or the queue appears in the order in which the message was published. However, the computation involved in producing messages for views and queues may introduce some amount of processing latency, and AMPS does not delay messages on other topics while performing these computations. For a queue that provides at-least-once delivery, if a processor fails and returns a message to the queue, that message will be redelivered (which means that the new processor may receive the message out of order). Likewise, when AMPS is providing conflation (either through a conflated topic or the conflation options on a subscription), AMPS does not provide ordering guarantees for conflated messages.

Applications often use this guarantee to publish checkpoint messages, indicating some external state of the system, to a checkpoint topic. For example, you might publish messages marking the beginning of a business day to a checkpoint topic, MARKERS, while the ORDERS topic records the orders during that day. Subscribers to the regular expression ^(ORDERS|MARKERS)$ are guaranteed to receive the message that marks the business day before any of the messages published to the ORDERS topic for that day, since AMPS preserves the original order of the messages.

For messages constructed by AMPS, such as the output of a view, AMPS processes messages for each topic in the order in which they arrive (unless conflation is requested) and delivers each calculated message to subscribers as soon as the calculation is finished and a message is produced. This keeps the latency low for each individual topic. However, this means that while AMPS guarantees the order in which messages are produced within each view, messages produced for views that do simple operations will generally take less time to be produced than messages for views that perform complex calculations or require more complicated serialization. This means that AMPS guarantees ordering within view topics, but does not guarantee that messages for separate view topics arrive in a particular order.

The figure below shows a possible ordering for messages received on an underlying topic and two views that use the topic:

Notice that within each topic, AMPS enforces an absolute order. However, the Simple View produces the results of Message 3 before the Complex View produces the results of Message 2. AMPS delivers the message for each topic as soon as possible.

Replicated Message Ordering

When providing messages received via replication (see Replicating Messages Between Instances), the principles on message ordering provided above still apply. AMPS records messages into the local transaction log in the order in which messages are received by the instance and provides messages to subscribers in that order. AMPS uses the sequence of publishes assigned by the original publisher and the order assigned by the upstream instance to ensure that all replicated messages are received and recorded in order with no gaps or duplicates.

Each instance of AMPS replicates messages to downstream destinations in the order in which messages are recorded in the transaction log.

AMPS does not enforce a global total ordering across a replication topology. This peer-to-peer approach means that an AMPS instance can continue accepting messages from publishers and providing messages to subscribers even when the remote side of a replication link is offline or if replication is delayed due to network congestion. However, if two messages are published to different instances at the same time by different publishers, the two instances may record a different overall message order for those messages, even though message order from each publisher is preserved.

AMPS offers a wide array of messaging features to solve a variety of messaging scenarios. This section presents some basic mappings between common messaging scenarios and the AMPS features that support those scenarios. Of course, this list is just a sampling of the types of applications that use AMPS.

The scenarios above describe just a few of the more common scenarios in which AMPS is used. For messaging scenarios that aren't described above, contact 60East at for advice and guidance.

One of the most common questions during evaluation of AMPS is how best to measure and quantify the overall performance of the application that uses AMPS.

There are several factors that are included in any meaningful discussion of performance:

Performance of the Underlying Hardware

AMPS is designed to use the underlying hardware as efficiently as possible, and does not suffer from artificial bottlenecks that limit performance.

The implications of this, though, are that the performance available from an installation of AMPS depends on the capacity of the underlying hardware.

In particular, pay attention to:

• Storage device speed and bandwidth (for applications that persist data)

• Memory speed

• Network speed and capacity

Often, you can come up with the theoretical maximum performance of a system based on the underlying hardware. For example, storage that can only write 80MB/s would be unsuitable for a system that needs to retain messages that arrive at a sustained rate of 100MB/s.

Likewise, a system with 64GB of memory would see reduced performance for lookups on a 128GB data set, so benchmarking an application that retains 128GB of active data on a system with 64GB of memory will produce very different results than the same benchmark run on a system with 256GB of memory.

Operating System Performance

Most Linux distributions and installations are, by default, tuned for interactive desktop usage. This is convenient when developing applications, but can produce reduced performance as compared with a well-tuned server.

in the discusses the Linux settings that are most often configured in a way that limits the performance of AMPS on a host. Before taking final benchmarks, tune the Linux host according to those guidelines.

Realistic Data Complexity and Volumes

AMPS is designed for high-throughput, low latency messaging. This means that AMPS typically performs better with a realistic workload than with a very small number of messages. It is typically not useful to run a performance test with a small number of messages and then attempt to extrapolate the performance at scale.

As an example, imagine a test that deploys a Docker container from scratch, starts AMPS, sends and receives a single message, and then shuts down the container and uses the elapsed time from the start of the test to the time that the container shuts down as the "single message throughput time". That number will be orders of magnitude slower than the actual time that it takes for AMPS to deliver the message: most of the time in the test is consumed by overhead unrelated to delivering an individual message.

Although it would be unlikely that anyone would create a test with as much overhead as the scenario above, it is not uncommon to have hidden overhead in a test. Likewise, there are often "economies of scale" that the system (including AMPS) can take advantage of production-level usage that is not available at unrealistically low messaging rates.

A realistic test should avoid measuring overhead that would not be present in a production environment. If the requirement of the application is to have latency within a certain threshold when AMPS is processing messages at a sustained rate from a dozen publishers, the results of a test will be more accurate the more closely the test approximates that scenario.

In particular, as much as possible, build your tests to:

• Have similar use of connections as the production application. If a given application will have multiple subscribers in production, do not use a single subscriber in performance testing and assume that parallel processing offers no benefit.

• Have similar message volumes as the production application. Do not assume that you can use a rate of 100 messages per second to predict latency or processing time of an application that will need to process 1000 or 10000 messages per second.

• Have similar message sizes as the production application. Do not assume that a 1MB message size in test will have the same performance characteristics as a 250KB message (or a 5MB message) in production.

Compare Equivalent Work

When benchmarking different implementation ideas, compare equivalent work. In some cases, having the AMPS server do additional work does not add noticeable latency due to the efficiencies (and parallel processing) in AMPS. In other cases, having the server do additional work may add more latency. In either case, accurately measuring throughput and latency must measure the cost of doing equivalent work in the application.

For example, if your application will use AMPS delta subscriptions (that is, have AMPS automatically calculate the differences between an update to a message and the current state of the message), rather than comparing throughput for a subscription that uses that option to a subscription that does not use that option based solely on when messages arrive at the client, compare the differences between having AMPS calculate the difference versus having the application calculate the difference, and evaluate this difference based on the total throughput numbers for a realistic number of subscribers.

Use AMPS Capabilities

AMPS is carefully designed to include functionality that reduces end-to-end throughput in the system, and to provide server-side capability where performing those functions on the server improves overall performance.

When evaluating performance, take advantage of those capabilities to get an accurate measure of how an application would perform in a production environment.

For example, if your application needs to append a calculated field to every published message, use message enrichment (or the AMPS delta publish functionality) rather than a process that extracts, rewrites, and updates the full message. Likewise, if your application will only process a subset of messages to the topic, use AMPS content filtering to ensure that AMPS only provides actionable messages rather than oversubscribing and discarding messages in your application.

If you have questions on whether your application is using the built-in capabilities of AMPS in the most effective way possible, contact 60East support for an engineer to review your design.

Below you'll find links to detailed pages for each configuration element. Click on any of the cards to quickly navigate to the specific configuration options you need for your AMPS instance.

Much like tables in a relational database, topics in the AMPS SOW persist the most recent update for each message. AMPS identifies a message by using a unique key for the message. The SOW key for a given message is similar to the primary key in a relational database: each value of the key is a unique message. The first time a message is received with a particular SOW key, AMPS adds the message to the SOW. Subsequent messages with the same SOW key value update the message.

There are several ways to create a SOW key for a message:

• Most applications specify that AMPS assigns a SOW key based on the content of the message. The fields to use for the key are specified in the SOW topic definition, and consist of one or more XPath expressions. AMPS finds the specified fields in the message and computes a SOW key based on the name of the topic and the values in these fields. 60East recommends this approach unless an application has a specific need for a different approach.

• A topic can also be configured to require that a publisher provide a SOW key for each message when publishing the message to AMPS.

• AMPS also supports the ability for custom SOW key generation logic to be defined in an AMPS module, which will be invoked to generate the SOW key for each message. While these SOW keys are generated automatically by AMPS, rather than being provided by the publisher, the logic to generate these keys is provided by the module, and the configuration required (if any) is determined by the module.

The following diagrams demonstrate how the SOW works, using a SOW topic that is configured to have AMPS determine the SOW key based on the /orderId field within the message. As each message comes in, AMPS uses the contents of the /orderId field to generate a SOW key for the message. The SOW key is used to identify unique records in the SOW, so AMPS will store a distinct record for each distinct /orderId value published to this topic. The calculated SOW key will be returned in the SowKey header of messages received from the topic in the SOW.

In the previous diagram, two messages are published where neither of the messages have matching keys existing in the ORDERS topic. The messages are both inserted as new messages.

Some time after these messages are processed, an update comes in for the order with an orderId of 2. This message changes the price from 120 to 95. Since the incoming message has an orderId of 2, this matches an existing record and overwrites the existing message for the same SOW key, as seen in the diagram below. AMPS replaces the entire record with the contents of the update.

Although the SOW key is derived from the content of the message in many cases, the SOW key is distinct from the content of the message. Each record in a SOW topic has a distinct SOW key, which is stored with the record. The SOW stores the full message in the message type format for performance. There is no re-serialization required to send a message to subscribers.

By default, a topic recorded in the SOW is persistent. For these topics, AMPS stores the contents of the SOW for that topic in a dedicated, memory-mapped file. This means that the total SOW does not need to fit into memory, and that the contents of the SOW database are maintained across server restarts. You can also define a transient SOW topic, which does not store the contents of the SOW to a persisted file.

The SOW file is separate from the transaction log, and you do not need to configure a transaction log to use a SOW. When a transaction log is present that covers the SOW topic, on restart AMPS uses the transaction log to keep the SOW up to date. When the latest transaction in the SOW is more recent than the last transaction in the transaction log (for example, if the transaction log has been deleted), AMPS takes no action. If the transaction log has newer transactions than the SOW, AMPS replays those transactions into the SOW to bring the SOW file up to date. If the SOW file is missing or damaged, AMPS rebuilds the SOW by replaying the transaction log from the beginning of the log.

When the SOW for a Topic is transient, AMPS does not store the SOW for this topic across restarts. In this case, AMPS will synchronize the SOW with the transaction log when the server starts to restore the state of the topic. By default, this recovery processes the entire transaction log. You can use the RecoveryPoint configuration option to specify that the topic should have only new publishes or should recover from a specific point in time (for example, you could use an environment variable to provide a timestamp to the RecoveryPoint so that AMPS recovers only the last 24 hours of messages.)

State of the World topics are used for several different purposes:

Queries / Point in Time Database

At any point in time, applications can issue SOW queries to retrieve all of the messages that match a given topic and content filter. When a query is executed, AMPS will test each message in the SOW against the content filter specified and all messages matching the filter will be returned to the client. The topic can be a literal topic name or a regular expression pattern. For more information on issuing queries, please see the section on .

Atomic Query and Subscribe

If the application needs to receive updates or needs the ongoing state of the topic rather than running a one-time query, the application can query the State of the World and simultaneously subscribe to updates to the topic. This is typically much more efficient than running repeated queries of the topic.

This command, sow_and_subscribe, is described in the topic in the section.

Enable Advanced Messaging Features

Because the State of the World maintains a record of the current state of messages, it enables several of the advanced messaging features provided by AMPS.

Out-of-Focus Messages

A subscription to a topic can optionally request to be notified when a message is removed or no longer matches the subscription when the topic is recorded in the State of the World. See the section for details.

Message Enrichment

AMPS can optionally enrich messages when they are published to a topic that is recorded in the State of the World. The enrichment can include logic based on the previous state of the message. See the section for details.

Publishing Incremental Updates

Because a topic stored in the State of the World maintains the current value of a message, applications do not need to republish the full message when making updates to a message. See the section for details.

Aggregation and Analysis

Since a State of the World topic maintains a complete set of current values for a topic, a State of the World topic is the foundation of analysis and aggregation of the messages published to a topic. See the section for more details.

Receiving Updated Fields Only

When a message in the State of the World is replaced or updated, AMPS can determine which fields (if any) have changed from the previous values. A subscriber can optionally request to be delivered only fields that have changed from the previous values. See for details.

Application Scenarios

The topics titled and in the provide an introduction to some of the application scenarios that can benefit from using a State of the World topic.

AMPS Concepts

This section describes the overall AMPS approach and the features AMPS provides.

The AMPS messaging system is designed around a few simple principles:

• Parallelize work and minimize waits and blocking to take full advantage of modern multisocket, multicore systems.

• Eliminate redundant or unused work by only performing tasks that are necessary to provide the functionality requested by a given operation.

• Reduce or eliminate cross-system coordination by solving the full range of data delivery and storage problems commonly faced by data-intensive applications.

• Provide a small, flexible set of commands for ease of use.

• Provide multiple delivery paradigms supporting both publish-subscribe delivery (many to many) and message queues (single consumption of a message) as well as the ability to query the state of a topic at a point in time.

• Stay application-focused to provide exactly the capabilities that are heavily used in demanding high-performance applications.

• Stay hardware aware and build for the future by engineering for next-generation commodity hardware and designing AMPS to fully exploit non-uniform memory access (NUMA), flash-based storage, and high-bandwidth networking.

These concepts are the foundation of how AMPS works and are helpful for understanding how to best use AMPS.

To best take advantage of AMPS, applications typically use the built-in features of AMPS rather than their traditional equivalents.

For example, rather than keeping a separate, independent record of each message published to AMPS for audit purposes, applications most often use one of the data persistence features in AMPS. This speeds development and simplifies deployment by eliminating integration effort, and also solves potential correctness issues which could be caused if messages in persistent storage become inconsistent with the messages provided through the messaging system. With AMPS, the messaging system itself can contain a fully-queryable and replayable record of the system.

As another example, AMPS provides integrated replication rather than relying on an external process. AMPS replication is aware of the format and semantics of the transaction log, the configuration of the instance, and the commands sent by publishers. This integration allows AMPS to very efficiently provide a full-fidelity message stream and to provide "self-healing" for an instance to catch up when it has been offline. Further, the message store used for replication (the AMPS transaction log) is also used for durable subscriptions and message replay. Designing and implementing these features together reduces complexity, storage requirements, and overhead to enable both capabilities. Within the AMPS server, the implementation uses a sophisticated parallelized algorithm for storage and replay that reduces overall latency and prevents slow consumers or replication destinations from affecting faster consumers. The overall result is to simplify configuration and application development, provide strong consistency and reliability guarantees, and provide the highest possible level of performance.

As a final example, rather than requiring a complex topic structure, requiring applications to oversubscribe and discard messages, AMPS provides both topic filtering and content filtering. AMPS includes an expressive filter grammar to provide precise selection of messages of interest to an individual subscriber. AMPS provides this capability to fully decouple publishers and subscribers. With AMPS, there's no need to maintain and administer a granular topic structure. Precise filtering and routing improves both network and processor utilization by providing only actionable messages to a subscriber. Likewise, for many applications, there is no need for a publisher to be aware of the processing performed by the subscriber or by AMPS itself.

The examples above highlight just a few of the capabilities AMPS provides and how the AMPS approach simplifies development, administration, and operations while providing reliability and performance benefits over conventional systems.

Feature Highlights

Some of the highlights of AMPS features include:

• Topic based publish and subscribe, including full support for regular expressions to specify topic names.

• Content filtering based on XPath identifiers (to specify the fields of a message) and SQL-92 (to form a predicate), with added support for Perl-Compatible Regular Expressions (PCRE2).

• Message queues including content filtering for both publishers and subscribers, configurable strategies for delivery fairness, and truly distributed queues that can efficiently enforce queue semantics and delivery guarantees across a replicated network of AMPS instances.

• Content-aware messaging support for a wide range of message types, including standard formats such as JSON, FIX, MessagePack, XML, Google Protocol Buffers, and BSON. AMPS also supports simple key/value pairs in FIX format (called NVFIX to emphasize that the format uses name/value pairs rather than FIX tags), and a high-performance binary protocol called BFlat. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing types to easily combine messages of different types in a single payload.

• An integrated database and record-aware current value storage (called State of the World, or SOW), with optional historical query capability.

• Historical replay of message streams, including the ability to preserve the total message ordering across independent topics.

• Integrated replication and high availability, including automatic resynchronization for instances that fail over.

• Aggregation and Complex Event Processing (CEP), including the ability to aggregate information across different message streams and message streams of different formats.

• Advanced messaging capabilities such as atomic query-and-subscribe, incremental (delta) updates, and out-of-focus notifications that tell a subscription when a record no longer matches.

• Built in statistics and monitoring, with data provided via a standard RESTful interface.

• Integrated authentication and entitlement across all AMPS features.

• Actions for automating AMPS functionality, including both routine maintenance tasks and dataflow-aware processing (such as alerting in response to slowdowns or invalid data).

• Client development kits for popular programming languages such as Java, C#/.NET, C++, Python, JavaScript, and Go.

• Extensibility API in the AMPS server for adding message types, extending the functions available to the AMPS query language, adding new actions, integrating with enterprise authentication and entitlement systems, and more.

This guide provides an overview of the most commonly used functionality of AMPS, but it is not intended to cover all of the features of AMPS or provide an exhaustive discussion of any individual feature. As mentioned earlier, the AMPS User Guide provides full details about AMPS features.

The AMPS engine binary is named ampServer and is found in $AMPSDIR/bin. Start the AMPS engine with a single command line argument that includes a valid path to an AMPS configuration file. You use the configuration file to enable and configure the AMPS features that your application will use. This guide discusses the most commonly used configuration options for each feature. The full set of options is described in the AMPS User Guide.

The AMPS server generates a minimal sample configuration file with the --sample-config option. You can save the sample configuration file to $AMPSDIR/amps_config.xml with the following command line:

$AMPSDIR/bin/ampServer --sample-config > $AMPSDIR/amps_config.xml

The server sample configuration only provides configuration for using AMPS to subscribe to and publish to ad hoc topics. The sample configuration file does not include any persistence for AMPS messages.

The file enables the instance monitoring interface (the "Galvanometer"), including the ability to query and subscribe to topics using a websocket connection.

A production configuration would likely provide persistent event and error logging to a file to allow an operations team to troubleshoot the instance and would typically persist monitoring statistics to a file. Such a configuration would likely enable additional message delivery features for certain topics and would also include configuration for high-availability and disaster recovery. The configuration would typically configure AMPS actions to perform routine maintenance.

On older processor architectures (and in some emulated environments) ampServer will start the ampServer-compat binary. The ampServer-compat binary avoids using hardware instructions that are not available on these systems.

You can also set the AMPS_PLATFORM_COMPAT environment variable to force ampServer to start the ampServer-compat binary. 60East recommends using this option only on systems that do not support the hardware instructions used in the standard binary. The ampServer-compat binary will not perform as well as ampServer, since it uses fewer hardware optimizations.

Once you have a configuration file saved to $AMPSDIR/amps_config.xml you can start AMPS with that file as follows:

$AMPSDIR/bin/ampServer $AMPSDIR/amps_config.xml

If your first start-up is successful, you should see AMPS display a simple message similar to the following to let you know that your instance has started correctly.

AMPS A.B.C.D.973814.e1a57f7 - Copyright (c) 2006-202X 60East Technologies Inc.
(Built: XXXX-YY-ZZT00:26:45Z)

The version numbers and dates will be appropriate for the version that you've started.

If you see this, congratulations! You have successfully cranked up the AMPS!

Command Line Options

The AMPS server binary supports the following command line options:

Further Reading

While there is much more content beyond the scope of this document, here are some of the topics to learn about after reading this guide.

Event Logging

AMPS provides a rich logging framework that supports logging to many different targets including the console, syslog, and files. Every error and event message within AMPS is uniquely identified and can be filtered out or explicitly included in the logger output. The Logging section in the AMPS User Guide describes the AMPS logger configuration and the unique settings for each logging target.

Conflation for Topics and Subscriptions

Another challenge that faces developers working with high-volume data flows is the fact that not every consumer can keep up with the rate at which data arrives.

For example, an application may display a view of data that is updated hundreds or thousands of times a second. The update rate for some data can be faster than the UI framework can redraw the grid that holds the data. Without a strategy for managing these updates, the application can be unresponsive, show outdated data, consume a large amount of memory -- or have all of those problems at the same time.

To help in this situation, AMPS provides built-in support for limiting the volume of updates. This feature is called conflation.

With AMPS conflation, an application receives updates for a particular message at most once within a specified interval. When an update for a record is sent to the application, the update contains the most current state of the record at that time. The application always receives the most current data. However, no matter how many times the record is updated during the interval, the application only receives the most current update at the end of the interval.

AMPS provides two forms of conflation:

• Conflated topics are declared in the server configuration. The AMPS server keeps only a single copy of the current message state for all subscribers.

• Conflated subscriptions are requested by an individual subscriber. AMPS keeps a copy of the current message state for each subscription, and that copy of the message state is not shared between subscriptions.

As an example of the value of conflation, imagine a SOW topic called PRICING that contains the current price for a set of instruments, and imagine that updates to the pricing are being published to the topic in real time. Several applications subscribe to this topic to display the latest prices for a subset of the instruments in a GUI front-end.

If this GUI front-end only needs updates in two second intervals from the PRICING topic, then more frequent updates would be wasteful of network and client-side processing resources. Likewise, if the GUI front end attempted to process and display every update to the prices, the incoming volume of updates might well outpace the ability of the grid to update. Using conflation in this case can both reduce network traffic and ease the load on the application.

In this case, every instance of the application is likely to have the same performance characteristics and benefit from the same interval for conflation. Therefore, configuring a conflated topic for the server would be a good approach. If there were only a single instance of this application or the application ran intermittently (for example, a monitoring or diagnostic tool), using a conflated subscription might be more appropriate.

The User Guide provides more info on conflation, conflated topics, and conflated subscriptions.

View Topics and Aggregation

AMPS contains a high-performance aggregation engine, which can be used to project one topic onto another, similar to the CREATE VIEW functionality found in most RDBMS software. Views can JOIN multiple topics together, including topics with different message types.

In addition to views configured by an administrator, individual subscriptions can create ad hoc aggregates and views on demand.

Paginated Subscriptions

For some use cases, in particular interactive applications that display large sets of records, it's useful to be able to display a subset of all of the records of interest. This saves network bandwidth by only delivering records that the application intends to display to a user, and saves CPU time in the application by removing the requirement for the application to process and discard records that aren't in the current result set.

For example, a web application may potentially show thousands of orders, but may only need to render a page of 20 records at any given time. With a paginated subscription, the application can request exactly the records it needs to render, and can be notified when those records change, are deleted, or if another record is inserted within the page.

Historical SOW Query

AMPS allows you to configure a SOW topic to retain the historical state of the SOW, on a configurable granularity.You can then query for the state of the SOW at a point in time, and retrieve results from the saved state.

Utilities

AMPS provides several utilities that are not essential to message processing, but can be helpful in troubleshooting or tuning an AMPS instance. The User Guide and Utility Reference describe these utilities in detail. The utilities include:

• spark - a command-line client, which is a useful tool for diagnostics, such as checking the contents of a SOW topic. The spark client can also be used for simple scripting to run queries, place subscriptions and publish data.

• ampserr - used to expand and examine error messages that may be observed in the logs. This utility allows a user to input a specific error code, or a class of error codes, examine the error message in more detail, and where applicable, view known solutions to similar issues.

• amps-grep - used to search the AMPS errors and events log or AMPS journal files to quickly locate items of interest. The AMPS User Guide includes information on the utility, including command-line templates for common searches in the Find Information in Error Log or Transaction Log topic.

• amps_sow_dump - used to inspect the contents of a SOW topic store.

• amps_journal_dump - used to examine the contents of an AMPS journal file during debugging and program tuning.

More information about each of these utilities, including usage and examples, can be found in the Utilities chapter of the AMPS User Guide.

Monitoring Interface

AMPS provides a monitoring interface which contains information about the state of the host system (CPU, memory, disk and network) as well as statistics about the state of the AMPS instance it is monitoring (clients, SOW state, Journal state and more). AMPS provides this information through a RESTful interface for ease of integration into existing enterprise monitoring systems.

AMPS can also record statistics in a persistent SQLite database, which can be queried using the standard SQLite toolset.

More information about the monitoring system provided in AMPS can be found in the Monitoring AMPS chapter of the AMPS User Guide. The AMPS Monitoring Guide contains information about the statistics available and how the monitoring statistics are recorded in the statistics database.

High Availability

The Replicating Messages Between Instances chapter and the Highly Available AMPS Installations chapter in the AMPS User Guide explains the powerful high availability features that AMPS provides. This chapter describes how to use the AMPS transaction log and AMPS replication to provide failover strategies and high availability guarantees.

To provide high availability and failover, AMPS provides replication of topics between instances. A set of features in the AMPS clients work with the AMPS server to provide reliable publishing and resumable subscriptions.

The transaction log, described earlier, is the foundation of AMPS replication. AMPS replication is designed to ensure that the messages in the transaction log of one AMPS instance are also in the transaction log of another AMPS instance.

The AMPS client libraries provide optional reliable publication functionality, using a local store to retain messages, until the AMPS server notifies the publisher that the message has met the persistence guarantees that the server is configured for. Typically, the persistence guarantee is configured to be the point at which the message has been confirmed to have been written to both instances in a high availability pair, but stronger guarantees (such as also having been written to an offsite disaster recovery instance, or having been written to an instance in another region) can also be configured.

The AMPS approach to high availability is based around the principle that each topic is a stream of messages. The basic concepts behind AMPS replication include:

• AMPS replication is always treated as a message stream from a source instance, which pushes messages, to a destination instance, which receives messages. In many cases, an application will use two-way replication so that instances contain the same messages. For two-way replication, replication is configured in both directions.

• The intent of AMPS replication is to ensure that every replicated message that the source instance is responsible for replicating has been recorded in the transaction log of the destination instance.

• AMPS replicates sequences of commands (that is, each individual publish or delete) rather than the cumulative state of a set of publishes.

• For a given data source (that is, an individual publisher), AMPS guarantees that it will preserve the order in which that data source provided messages. The order must be consistent both within an instance and between instances.

For full details on AMPS replication, including recommendations and best practice advice, see the Replicating Messages Between Instances chapter and the Highly Available AMPS Installations chapter in the AMPS User Guide.

This section describes a typical process for developing an AMPS configuration file.

1. Add mandatory information

   An AMPS configuration starts with an AMPSConfig element at the outermost level of the configuration. Every AMPS instance must have a Name. This name identifies the instance, and must be unique within a set of instances.

   In this case, we will define the name test-AMPS-1 for the instance, as shown below:

   Copy

     <AMPSConfig>
         <Name>test-AMPS-1</Name>
     </AMPSConfig>

   With this configuration, AMPS will start but will not allow any incoming connections to process messages. This instance also does not provide any monitoring or logging.

1. Describe how AMPS will receive connections from applications

   AMPS has no default settings for receiving commands from applications or for delivering messages to applications.

   To enable applications to connect to AMPS, specify a Transports element that defines one or more incoming Transport definitions. These are described in detail in the Configuring Transports section.

   For example, it's common to define one transport for application connections, and one transport for the use of web applications (JavaScript / TypeScript) and data queries or subscriptions from the Galvanometer admin tool.

   In this case we'll allow browser-based applications (such as Galvanometer) to connect to port 9008, and applications that use the other client libraries to connect to port 9007.

   After this step, the configuration might look something like this:

   Copy

     <AMPSConfig>
       <Name>test-AMPS-1</Name>
   
       <Transports>
         <Transport>
             <Name>any-tcp</Name>
             <Type>tcp</Type>
             <InetAddr>9007</InetAddr>
             <MessageType>json</MessageType>
             <Protocol>amps</Protocol>
         </Transport>
   
         <Transport>
             <Name>any-websocket</Name>
             <Type>tcp</Type>
             <InetAddr>9008</InetAddr>
             <MessageType>json</MessageType>
             <Protocol>websocket</Protocol>
         </Transport>
       </Transports>
   
     </AMPSConfig>

1. Enable administrative monitoring

   AMPS does not enable administrative monitoring by default. However, all instances of AMPS should enable this, as described in the Admin Server and Statistics section.

   For sample purposes, we will use port 8085 to host the administrative interface and Galvanometer. This configuration also tells Galvanometer to use the any-ws transport for the embedded JavaScript client.

   Copy

     <AMPSConfig>
       <Name>test-AMPS-1</Name>
   
       <Transports>
         <Transport>
             <Name>any-tcp</Name>
             <Type>tcp</Type>
             <InetAddr>9007</InetAddr>
             <MessageType>json</MessageType>
             <Protocol>amps</Protocol>
         </Transport>
   
         <Transport>
             <Name>any-websocket</Name>
             <Type>tcp</Type>
             <InetAddr>9008</InetAddr>
             <MessageType>json</MessageType>
             <Protocol>websocket</Protocol>
         </Transport>
       </Transports>
   
       <Admin>
           <InetAddr>8085</InetAddr>
           <SQLTransport>any-ws</SQLTransport>
       </Admin> 
     </AMPSConfig>
   

1. Enable error and event logging

   Last, but not least, it's important to be notified of any errors or important events in AMPS. AMPS does not enforce any default for this logging, but instead requires this logging to be explicitly configured.

   For a production instance, AMPS would typically log messages to a file. For a quick sample, though, we can just log anything at warning level or above to stdout.

   Copy

     <AMPSConfig>
       <Name>test-AMPS-1</Name>
   
       <Transports>
         <Transport>
             <Name>any-tcp</Name>
             <Type>tcp</Type>
             <InetAddr>9007</InetAddr>
             <MessageType>json</MessageType>
             <Protocol>amps</Protocol>
         </Transport>
   
         <Transport>
             <Name>any-websocket</Name>
             <Type>tcp</Type>
             <InetAddr>9008</InetAddr>
             <MessageType>json</MessageType>
             <Protocol>websocket</Protocol>
         </Transport>
       </Transports>
   
       <Admin>
           <InetAddr>8085</InetAddr>
           <SQLTransport>any-ws</SQLTransport>
       </Admin> 
   
       <Logging>
         <Target>
             <Protocol>stdout</Protocol>
             <Level>warning</Level>
         </Target>
       </Logging>
   
     </AMPSConfig>

1. Define advanced messaging behavior

   The configuration in the four sections above is all you need for basic subscribe and publish functionality.

   For more advanced functionality, AMPS can be configured to provide the necessary capabilities.

   For example, to configure a last-value-cache for a topic, you would define a Topic in the State of the World (SOW). To enable full, message-by-message replay for a topic or set of topics, you would define a Transaction Log.

All instances of AMPS, regardless of the application they support, should follow this minimum outline: define the instance Name, the instance Transports, define monitoring through the Admin console, and specify the Logging for the instance.

Most applications also take advantage of the advanced messaging capabilities of AMPS, as described in detail in this guide. The Scenario and Feature Reference section in the Introduction to AMPS guide provides helpful starting points for different scenarios.

AMPS has the ability to allow a subscriber to retrieve only the relevant parts of a message, in the same way that a SQL query can retrieve only specified fields from a table. For example, consider a topic that stores an event ID, a short description, and a detailed event record. A UI that presents an overview of the contents of the topic might only need the event ID and short description to present a high-level view of the topic contents, while retrieving the detailed event record when a user explicitly requests the details for a specific record.

With select lists, AMPS allows an individual subscription to control which fields are retrieved from a subscription or query. In the example above, the subscription would include a select list that requests that AMPS provide the event ID and description, while excluding any other field. To do this, the application would include the following option on the command used to retrieve data for the overview: select=[-/,+/event_id,+/description]

When provided by an application as a part of a command to AMPS, a select list is applied after any content filtering is applied. The select list specifies the contents of the subscription, but does not affect the underlying messages, and the contents of the subscription select list do not affect filter evaluation or query results.

Creating Select Lists

As mentioned above, to provide a select list on a command, add the keyword select and a comma-delimited list of field directives to the options for a subscription or query in AMPS.

Each field directive is a combination of an inclusion specifier and an AMPS identifier.

For example, the field directive +/event_id has an inclusion_specifier of + and the AMPS identifier of /event_id. This field directive specifies that the /event_id field is included in the message returned to the subscriber.

AMPS recognizes the following inclusion specifier values:

Identifiers for individual fields follow the syntax described in the Identifiers section.

For select lists, AMPS also recognizes the special field directive of -/ to specify that all fields should be excluded and the special field directive of +/ to specify that all fields should be included.

If no field directive in the select list applies to a given field in a message, that field is included in the message.

If a field is covered by multiple field directives, AMPS respects the most specific field directive. In other words, a select list that contains the field directives +/,-/details will include all fields except the details field. A select list that contains the field directives -/event,+/event/description will include the /event/description subfield, but no other contents of the /event field. (If an identifier is provided twice in the same select list, AMPS uses the first field specifier that contains the identifier.)

With select lists, AMPS does not create fields that are not in the original message. This means that if the select list requests a field that does not exist in the original message, the message delivered to the subscriber will not contain that field.

Notice that a select list only changes how a message is delivered to the subscriber that the select list applies to. The original message is unaffected, and the complete message is delivered to any subscriber that does not specify a select list.

AMPS contains related functionality that may be more appropriate for some applications:

• To modify a message as it is published to AMPS, use Enrichment and Preprocessing. With those features, the original publish message is modified and the modified message is stored in AMPS and sent to all subscribers.

• AMPS also offers the ability to create a view of a set of messages that aggregates data across a set of messages and produces a result (for example, the total value of all open orders for each customer). See the chapter on Aggregating and Analyzing Data in AMPS for more details.

Select List Examples

For example, consider an original message like the following JSON document:

{ "id": 42,
  "name":"Arthur",
  "day":"Thursday",
  "complaint":"Unannounced construction in neighborhood.",
  "pocket_contents":
        { "left":"twine",
          "right":"towel" }
}

An application might only need to see the id and complaint description. To retrieve just those fields of a message, the application could add the following option to the command that retrieves the message:

select=[-/,+/id,+/complaint]

This select list tells AMPS to remove all fields from the message except for the /id field and the /complaint field. With this select list, the message above will be delivered as:

{ "id": 42,
  "complaint":"Unannounced construction in neighborhood."
}

Likewise, an application could want to know the name of the person making the complaint and the contents of that person's left pocket:

select=[-/,+/name,+/pocket_contents/left]

From the original message, the result of providing this select list would be:

{ "name": "Arthur",
  "pocket_contents":
        { "left":"twine"}
}

Last, consider an application that wants to see everything in the message except the pocket_contents. That application could provide an option such as:

select=[-/pocket_contents]

With that specifier, AMPS provides any field in the message except the pocket_contents, producing the following result:

{ "id": 42,
  "name":"Arthur",
  "day":"Thursday",
  "complaint":"Unannounced construction in neighborhood."
}

This section describes general performance considerations for the AMPS expression language and content filters. The considerations here are aspects of AMPS performance to be aware of in the general case. However, since the AMPS expression language operates on specific data, the structure and size of the messages that your application uses may have more effect on overall performance than the specific expressions used. For example, parsing and filtering a 20MB XML document is inherently more expensive than parsing and filtering a 400 byte BFlat document.

Use Short-Circuiting

When clauses in an expression are joined by OR, AMPS will only evaluate the right side of an OR expression if the left side of the expression is false.

When constructing an expression, this means that there can be a performance advantage to having relatively less expensive clauses on the left hand sides of the OR. For example, in the following clause:

/code = 'restricted' OR /notes LIKE 'restricted|limited'

The regular expression comparison is only evaluated if the comparison /code = 'restricted' is false. If the comparison is true, then the overall clause is true and there is no need to evaluate the regular expression.

Avoid Redundant Expressions

AMPS does not reorder or recombine complex expressions. Where feasible, your application can save work at the server by combining expressions. In particular, if an application is constructing a filter by reading options from various sources, performance can be improved by combining the queries.

For example, in a filter like the following:

/id = '12345' OR /id IN ('12345','23456','34567','45678')
              OR /id IN ('12345','45678','90909')

The comparison against '12345' will be evaluated three times in cases where the value of /id does not match any of the values in the filter.

This filter is equivalent to:

/id IN ('12345','23456','34567','45678','90909')

The same results are produced, but only evaluates the /id field against a given value one time.

Use Specialized Operators for Simple Comparisons, Use LIKE when Necessary

The LIKE operator offers access to full Perl-Compatible Regular Expressions within the AMPS expression language. This flexibility allows for very precise filtering, and the PCRE engine performs well.

However, for comparisons for which AMPS provides a named function, the named function is highly-optimized and will perform somewhat better than the general-purpose regular expression engine.

For example, given a choice between two equivalent expressions:

/state BEGINS WITH('North')

and

/state LIKE '^North'

The version that uses BEGINS WITH will typically perform slightly better than the version that uses the regular expression.

This doesn't mean that regular expressions or the LIKE operator perform poorly. The LIKE operator can efficiently match patterns that would be difficult or impossible to match using the other operators. However, for very simple comparisons where AMPS provides a dedicated operator, that operator typically performs slightly better than a regular expression.

The following table shows some examples of regular expressions and the AMPS operator equivalent.

Optimize for Partial Parsing

Most AMPS message types have the ability to partially parse messages. That is, rather than parsing the entire message, the message type can simply find the identifiers that will be used, and stop the parsing process as soon as those identifiers are found.

This optimization is most useful for larger messages. For example, if the SOW key for a topic is based on the /id field of a message and there are active content filters that use both the /id field and the /code field, while no other field is being indexed, then, considering the message below:

{"id":24,"code":"A12347","notes":"entered on behalf of a sloth",
     // ... 100K of other data ...
}

The AMPS parser can stop parsing after processing only the /id and the /code fields. In this case, halting the parsing after processing these two fields avoids the expense of parsing the remaining parts of the message.

Notice that this optimization will only improve performance in cases where AMPS doesn't need to parse the entire message. For example, if there is a delta_subscribe active for the topic, or if the command being processed is a delta_publish, AMPS will parse the message completely to be able to calculate the deltas. Likewise, if any filter refers to a field that doesn't appear in the message, AMPS will parse the message completely to be able to determine that the field does not appear in the message.

SOW Queries and Indexing

Queries over topics in the State of the World (SOW) have additional performance considerations. AMPS maintains indexes over SOW topics to help locate messages in response to a query.

• Queries over a topic in the SOW can use SOW topic indexes. Where possible, use an exact string match and create a hash index to take advantage of hash indexes.

• When a query is submitted with an XPath identifier for which no index exists, AMPS will create and populate a memo index for that XPath identifier. This can add to the amount of time a query takes the first time a given XPath identifier is queried. You can specify that AMPS creates a memo index for a given identifier by using the Index configuration item in the Topic definition. Once an index is created, AMPS will continue to search for that XPath identifier in incoming messages for that topic to keep the index up to date.

Notice that SOW topic indexes are only used for sow commands and during the sow portion of a sow_and_subscribe (or sow_and_delta_subscribe) command. Once the subscription to current updates begins, the subscription does not use a SOW topic index because there is no need to locate a message. During a subscription, filters are run against the current message.

See the section on Indexing for State of the World topics for details.

AMPS includes functions that can be used to refer to the current message being processed.

When used in view construction or aggregate definition, these functions refer to the incoming message that is prompting the update to the view or aggregate, not to the constructed message that is the result of the update. For example, a Field like this in a view projection:

<Projection>
  ...
  <Field>TOPIC_NAME() AS /theTopic</Field>
  ...
</Projection>

will return the topic name of the topic that prompted the update to the view, not the name of the view itself.

The SOW section of the configuration file specifies the configuration for the AMPS State of the World (SOW). AMPS supports several different types of SOW topics that can be configured as part of the SOW.

A Topic acts as a last value cache to store data. For more information on SOW Topic last value caching, see the sections on State of the World (SOW) Topics, Querying the State of the World (SOW), Out-of-Focus Messages, State of the World Message Enrichment, Incremental Message Updates, and Receiving Only Updated Fields.

A Queue, LocalQueue, or GroupLocalQueue provides a mechanism for ensuring that messages are processed by an application once, as described in the Message Queues section of this guide. The differences between the queue types specify how the queue will behave in a replicated set of instances, as described in Queue Replication Types. The current state of messages that have not been delivered to applications can be queried as described in Querying the State of the World (SOW).

View topics are configured using one or more of Topic, Queue, View, or ConflatedTopic as the underlying source of information. The Aggregation and Analytics section of this guide describes how to configure a View. A View also supports queries and subscriptions as described in Querying the State of the World (SOW), Out-of-Focus Messages, and Receiving Only Updated Fields.

A ConflatedTopic is a way to mitigate message velocities that are too high for subscribers to efficiently process. It provides a way for those subscribers to consume data from a Topic, View, or another ConflatedTopic, while also supporting queries and subscriptions as described in Querying the State of the World (SOW), Out-of-Focus Messages, and Receiving Only Updated Fields.

Described below are the configuration items available for SOW. Expand each item for more details.

AMPS provides a pair of functions, REPLACE and REGEXP_REPLACE, that replace text within strings. The REPLACE function does a literal match of the string to be replaced, while REGEXP_REPLACE uses a PCRE pattern to find the string to be replaced.

The following expressions all evaluate as true:

REPLACE('fandango', 'dan', 'din') == 'fandingo'

REGEXP_REPLACE('fandango','n.*n', 'r') == 'fargo'

Communication between applications and the AMPS server uses AMPS messages. AMPS messages are received or sent for every operation in AMPS. Each AMPS message has a specific type and consists of a set of headers and a payload. The headers are defined by AMPS and formatted according to the protocol specified for the connection. Typically, applications use the standard amps protocol which uses a JSON document for headers. The payload, if one is present, is the content of the message and is in the format specified by the message type.

Messages received from AMPS have the same format as messages to AMPS. These messages also have a specific type, with a header formatted according to the protocol and a payload of the specified message type. For example, AMPS uses ack messages, short for acknowledgment, to report the status of commands. AMPS uses publish messages to deliver messages on a subscription, and so on for other commands and other messages.

Let's consider a complete interaction between an application and the AMPS server as an example. When a client subscribes to a topic in AMPS, the client sends a subscribe message to AMPS that contains the information about the requested subscription and, by default, a request for an acknowledgment that the subscription has been processed. AMPS returns an ack message when the subscription is processed that indicates whether the subscription succeeded or failed, and then begins providing publish messages for new messages on the subscription. The publish messages continue as messages that match the subscription arrive at the AMPS server. If the application needs to stop the subscription, the application sends an unsubscribe message to the AMPS server, indicating the subscription to end. Once the AMPS server processes the unsubscribe message, the server will no longer send messages for that subscription to the application. Should the application disconnect, the AMPS server removes all subscriptions for that connection (whether or not the application sends an unsubscribe command first).

Messages to and from AMPS are described in more detail in the AMPS Command Reference, available on the 60East website and included in the AMPS client SDKs.

In this version of AMPS, the communication transports used by AMPS accept message sizes of up to 200MB in a single command to AMPS. Messages larger than 200MB may be rejected by the transport as invalid. Should your use of AMPS require larger message sizes, contact 60East support.

Introduction to AMPS Headers

The AMPS Command Reference contains a full list of headers for each command. The table below lists some commonly used headers.

This section presents a few of the commonly used headers. See the AMPS Command Reference for a full description of AMPS messages.

AMPS does not provide the ability to add custom header fields. However, AMPS composite message types provide an easy way to add an additional section to a message type that contains metadata for the message. Since composite message type parts fully support AMPS content filtering, this approach provides more flexibility and allows for more sophisticated metadata than simply adding a header field. See the Composite Messages section for details.

For an outline of your specific support policies, please see your 60East Technologies License Agreement. Support contracts can be purchased through your 60East Technologies account representative.

Support Steps

You can save time if you complete the following steps before you contact 60East Technologies Support:

1. Check the documentation

   The problem may already be solved and documented in the User Guide for the product. 60East Technologies also provides answers to frequently asked support questions on the support website at: .

2. Isolate the problem

   If you require Support Services, please isolate the problem to the smallest test case possible. Capture erroneous output into a text file along with the commands used to generate the errors.

3. Collect your information

   • Your product version number.

   • Your operating system and its kernel version number.

   • The expected behavior, observed behavior and all input used to reproduce the problem.

   • Submit your request.

   • If you have a minidump file, be sure to include that in your email to .

The AMPS version number used when reporting your product version number follows a format listed below. The version number is composed of the following:

AMPS Versioning and Certification

Each AMPS version number component has the following breakdown:

The certification levels are defined in the following table. Notice that, in all cases, 60East will certify at a higher level if time permits or if a change involves a critical part of AMPS (such as replication or internal utility classes that are widely used).

Contacting 60East Technologies Support

Please contact 60East Technologies Support Services according to the terms of your 60East Technologies License Agreement.

Support is offered through the United States:

Other support options (such as support via phone), may be available depending on the terms of your support agreement.

AMPS supports filters that operate on arrays in messages. There are two simple principles behind how AMPS treats arrays:

1. Binary operators that yield true or false (for example, =, <, LIKE) are array aware, as is the IN operator. These operators work on arrays as a whole, and evaluate every element in the array.

2. Arithmetic operators, functions, user-defined functions and other scalar operators, are not array aware, and use the first element in the array.

With these simple principles, you can predict how AMPS will evaluate an expression that uses an array. For any operator, an empty array evaluates to NULL.

Let's look at some examples. For the purposes of this section, we will consider the following JSON document:

While these arrays are presented using JSON format for simplicity, the same principles apply to arrays in other message formats.

Here are some examples of ways to use an array in an AMPS filter:

Determining if any element in an array meets a criteria

To determine this, you provide the identifier for the array, and use a comparison operator.

Determine whether a specific value is at a specific position

To determine this, use the subscript operator [] on the XPath identifier to specify the position, and use the equality operator to check the value at that position.

Determine whether any value in one array is present in another array

Determine whether an array contains one of a set of values

These patterns and principles hold regardless of the original representation of the array in a document.

When creating an expression that uses a field in a compound value, keep in mind that AMPS represents compound values as described in the section on .

For views, aggregated subscriptions, and SOW topic enrichment, AMPS allows you to construct new fields based on existing data.

When you construct a field, there are two components required:

1. A source expression that produces a value. This expression can include XPath identifiers that extract values from a message, literal values, operators, and functions.

2. A destination identifier that specifies the identifier where the message type will serialize the value produced by the source expression.

The source expression and the destination identifier are separated by the AS keyword. The format for a field construction expression is as follows:

For example, to create a field in a view that calculates the total value of an order by multiplying the /price field times the /qty field, construct the field as shown below:

This constructs a field using /price * /qty as the source expression. Both /price and /qty are taken from the incoming message. When the result of this expression is computed, the value will be produced with the XPath identifier /total as the destination. That value will then be serialized to a message (with the exact format and syntax determined by the message type).

Notice that the grammar for constructing fields does not specify precisely how the field is represented in the message. AMPS constructs the value and provides the XPath identifier to the message type. The message type itself is responsible for serializing the value into the correct representation and structure for that message type.

All of the AMPS operators and functions that are available for filters are available to use in source expressions, including any user-defined functions loaded into the instance.

Depending on the context for field construction, there are additional capabilities available when constructing fields, as described in the following sections.

Constructing Preprocessing Fields

Preprocessing field constructors operate on a single message and construct fields based on that message. The results of the preprocessing field constructor are merged into the incoming message. Any field in the source message that is not changed or removed during preprocessing is left unchanged, so it is not necessary to include all fields in the message in the Preprocessing block.

Since preprocessing fields apply to a specific message, preprocessing fields cannot specify the topic or message type in an XPath identifier. All identifiers in the source expression are evaluated as identifiers in the message being preprocessed. Preprocessing fields are evaluated during the preprocessing phase, so they cannot refer to the previous state of a message.

Using HINT to Control Field Construction

Preprocessing can be used to remove fields from a message. By default, AMPS serializes any field that has an empty string or NULL value after preprocessing. Preprocessing fields can include a directive that specifies that a field that contains a NULL value should be removed from the set of fields rather than serialized with a NULL value. The directive HINT OPTIONAL applied to the XPath identifier specifies that if the result of the source expression is NULL, AMPS does not provide the value for the message type to serialize. For example, the following field constructor removes the /source field from the message if the value provided is not in a specific list of values:

By default, AMPS considers the results of field construction (the processed message) to be distinct from the current message. AMPS rewrites the current message after preprocessing is completed. This means that, by default, the results of fields constructed during preprocessing are not available to other fields within preprocessing. The HINT SET_CURRENT option immediately inserts or updates values in the current message, which makes the new value available to all subsequent Field declarations.

In the sample below, AMPS enriches the message by performing an expensive operation (implemented as a user-defined function) on two input fields, and immediately updates the current message with the output of that operation. AMPS then sets other fields in the processed message using the updated value in the current message.

Notice that using HINT SET_CURRENT requires AMPS to process Field declarations in order, which may prevent future optimizations.

Hints can be combined as follows:

In this case, if the projected field would be NULL, the field is removed from the current message.

Constructing Enrichment Fields

Enrichment field constructors operate on a single message and construct fields based on that message. Enrichment expressions operate on the current message and change the current message. The results of the enrichment directives are merged into the incoming message. Any field in the source message that is not changed or removed during preprocessing is left unchanged, so it is not necessary to include all fields in the message in the Enrichment directive.

Since enrichment fields apply to a specific message, enrichment fields cannot specify the topic or message type in an XPath identifier. All identifiers in the source expression are evaluated as identifiers in the message being enriched.

Enrichment fields are constructed during the enrichment phase, so enrichment fields can refer to the previous state of a message. Within an enrichment expression, AMPS provides two special modifiers for XPath identifiers that specify whether an XPath identifier refers to the current incoming message or the previous state of the message. These modifiers apply only to the source expression, and cannot be used in the destination identifier. The modifiers are as follows:

Using HINT to Control Field Construction

Enrichment can be used to remove fields from a message. By default, AMPS serializes any field that has an empty string or NULL value after enrichment. Enrichment Field elements can include a directive that specifies that a field that contains a NULL value should be removed from the message rather than serialized with a NULL value. The directive HINT OPTIONAL applied to the XPath identifier specifies that if the result of the source expression is NULL, AMPS does not provide the value for the message type to serialize. For example, the following field constructor removes the /source field from the message if the value provided is not in a specific list of values:

By default, AMPS considers the results of field construction (the enriched message) to be distinct from the current message. AMPS rewrites the current message after enrichment is completed. This means that, by default, the results of fields constructed during enrichment are not available to other fields within enrichment. The HINT SET_CURRENT option immediately inserts or updates values in the current message, which makes the new value available to all subsequent Field declarations.

In the sample below, AMPS enriches the message by performing an expensive operation (implemented as a user-defined function) on two input fields, and immediately updates the current message with the output of that operation. AMPS then sets other fields in the processed message using the updated value in the current message.

Notice that using HINT SET_CURRENT requires AMPS to process Field declarations in order, which may prevent future optimizations.

Hints can be combined as follows:

In this case, if the projected field would be NULL, the field is removed from the current message.

Constructing View Fields

View field constructors operate over groups of messages, and construct a single output message for each distinct group, as specified by the Grouping element in the View configuration.

When constructing a field in a view, all identifiers used in the source expression must be in one of the underlying topics for the view. When the view uses a Join, the identifiers must include the topic identifier. If the topics in the Join are of different message types, the identifiers must include both the message type and the topic identifier.

For example, the following Field definition multiplies the /quantity from the NVFIX topic orders by the /price from the JSON topic items, and projects the result into the /total field of the view.

AMPS provides a set of aggregation functions that can be used in a Field constructor for a view and in the projection option of an aggregated subscription. These functions return a single value for each distinct group of messages, as identified by distinct combinations of values in the Grouping clause.

These functions produce an aggregation over a literal value, an identifier directing AMPS to extract the value from the message, or the result of a function.

For example, given a set of messages like the following:

With a view definition that has a Projection clause and Grouping clause like the following:

AMPS will produce the following record:

Notice that the first SUM() function simply extracts the value of the /qty from each message, while the second SUM() function uses the output of the IF statement for each message.

Since aggregate functions operate over groups of messages, these functions are only available when constructing fields for aggregate purposes, either in a view or an aggregated subscription. The functions described in this section are not available to filters, and are not available for constructing fields during SOW topic enrichment.

The set of functions provided in AMPS have been chosen to be efficient to compute over high volumes of rapidly changing data.

Null values are not included in aggregate expressions with AMPS, nor in ANSI SQL. COUNT will count only non-null values, SUM will add only non-null values, AVG will average only non-null values, and MIN and MAX ignore NULL values, and so on.

MIN and MAX can operate on either numbers or strings, or a combination of the two. AMPS compares values using the principles described for comparison operators. For MIN and MAX, AMPS determines order based on these rules:

• Numbers sort in numeric order.

• String values sort in ASCII order.

• When comparing a number to a string, convert the string to a number, and use a numeric comparison. If that is not successful, the value of the string is higher than the value of the number.

For example, given a field that has the following values across a set of messages:

MIN will return 1.3, MAX will return 'cat'. Notice that different message types may have different support for converting strings to numeric values: AMPS relies on the parsing done by the message type to determine the numeric value of a string.

AMPS includes a pair of functions that provide the instance name and group name of the current instance.

AMPS provides the CONCAT function, that can be used for constructing strings. The CONCAT function takes any number of parameters and returns a string constructed from those parameters. The function can accept both XPath identifiers and literal values.

The CONCAT function can be used in any AMPS expression that uses a string. For example, you could CONCAT in a filter as follows:

CONCAT can be combined with other expressions, including conditional expressions. A mailingAddressName field in a view could be constructed as follows:

AMPS includes a function for calculating the distance from a signed latitude and longitude.

AMPS provides the ability to record topics and replay those topics at a later time. This capability is called the transaction log.

The AMPS transaction log fully supports topic and content filtering. You configure the transaction log to keep a journal of incoming messages for one or more topics, and then you can replay those messages, in order, from any point in time. With the (optional) high-availability features in the AMPS client libraries, this also provides a way to ensure that in case of failure, an application can resume the subscription without missing messages or receiving duplicate messages.

The AMPS transaction log is most often used for:

• Fully Resumable Subscriptions - With the transaction log, you can ensure that an application receives all messages of interest, even in the event of a failure.

• Backtesting and Audit - The transaction log allows you to replay the precise messages published, in order across all topics in the instance, at a configurable maximum rate. You can use this feature to easily audit the flow of messages, perform backtesting, or replay a sequence of events.

• Capacity Planning and Stress Testing - Since the transaction log allows you to set the maximum replay rate to be a multiple of the original publish rate, you can use the transaction log to measure the load on a system at various rates, and measure the capacity of the system and the ability of your application to correctly handle increased volumes.

The transaction log is also the source of messages for:

• Message Queues

• Replicating Messages Between Instances

The AMPS transaction log can typically record messages at the maximum throughput of the underlying storage device.

60East recommends storing the transaction log on a device that supports fast sequential writes, and ensuring that the device has the speed and capacity necessary to support the expected throughput. (The Operation and Deployment chapter of the AMPS User Guide includes guidance on capacity planning.)

How Does the Transaction Log Work?

The AMPS transaction log records messages that are published to the topics specified in the configuration file. Every publish is stored, in the order in which the AMPS instance processed the message.

For ease of maintenance, the AMPS server writes multiple sequential files, called journal files, for the transaction log rather than writing a single large file. The journals contain the full content of each message, as well as information on the topic, the publisher, the time at which the message was processed, and so on. The AMPS configuration sets the maximum size of a journal file. When a file reaches that size, AMPS begins writing to the next file.

When AMPS records a message into the transaction log, it assigns each message a bookmark. The bookmark identifies a single message, that is, a specific point in the transaction log of the local instance.

The AMPS server does not modify the contents of journal files. Once a message is written to a journal file, it is part of the transaction log and is considered to be immutable.

Since journal files form part of the persistent state of the server, those files should not be modified or removed while the AMPS process is running except by the AMPS process itself. The AMPS server provides a set of maintenance actions for managing journal files (see the AMPS User Guide for details).

Configuration

To create a Transaction Log, you add the TransactionLog configuration element to your AMPS configuration file. You then specify a location for AMPS to create journal files, and specify the topics that you want recorded in the file.

The following configuration is the minimum configuration to create a transaction log and record a single topic:

<TransactionLog>
    <JournalDirectory>./journals</JournalDirectory>
    <Topic>
       <Name>some-topic</Name>
       <MessageType>json</MessageType>
    </Topic>
 </TransactionLog>

The configuration above writes journal files to the journals directory underneath the AMPS server's current working directory. The configuration records a single topic, some-topic, of message type json to the transaction log. The Name option of the Topic configuration element can be either a literal topic name, or a regular expression that matches the names of a set of topics to be recorded.

Although this configuration works perfectly well, AMPS provides a number of additional options that are useful for managing transaction logs in production. AMPS also provides a set of administrative actions for setting the archival and retention policy for journal files.

A more complete configuration might include options along the following lines:

<TransactionLog>
    <JournalDirectory>./journals</JournalDirectory>
    <JournalArchiveDirectory>/mnt/high-capacity/journals</JournalArchiveDirectory>
    <JournalSize>100MB</JournalSize>
    <Topic>
       <Name>^/orders</Name>
       <MessageType>json</MessageType>
    </Topic>
    <Topic>
       <Name>^/status/customer</Name>
       <MessageType>fix</MessageType>
    </Topic>
    <Topic>
       <Name>/audit/events</Name>
       <MessageType>binary</MessageType>
    </Topic>
</TransactionLog>

<Actions>
   <Action>
     <On>
         <Module>amps-action-on-schedule</Module>
         <Options>
           <Every>21:30</Every>
           <Name>Daily Journal Maintenance Plan</Name>
         </Options>
     </On>
     <Do>
         <Module>amps-action-do-archive-journals</Module>
         <Options>
           <Age>3d</Age>
         </Options>
     </Do>
     <Do>
         <Module>amps-action-do-remove-journals</Module>
         <Options>
           <Age>7d</Age>
         </Options>
     </Do>
   </Action>
</Actions>

In this configuration, journals are created in the journals directory underneath the AMPS server's current working directory, as before. This configuration records two sets of topics and one individual topic. Taking these in the order in which they appear in the configuration file, this instance of AMPS will record:

• Any topic that begins with /orders and is of message type JSON.

• Any topic that begins with /status/customer and is of message type FIX.

• The topic /audit/events of message type binary.

The sample above also includes a basic journal maintenance configuration. Configuring journal maintenance is strongly recommended for any instance of AMPS that will be running on a regular basis.

For this AMPS installation, the size of the journal files has been reduced from the default 1GB size to a 100MB size. This typically indicates that the instance stores less than 1GB of messages during a day, so the default journal size would include multiple days worth of messages.

This configuration specifies a two-step maintenance process:

• After 3 days, journal files will be archived to the /mnt/high-capacity/journals directory -- the directory specified in the JournalArchiveDirectory parameter of the transaction log. These journal files remain a part of the transaction log but are moved to a different location (typically on a different device with higher capacity).

• After 7 days, journal files will be deleted.

AMPS will run this maintenance plan every day at 21:30 (9:30 PM) local time.

When journal files are moved to the archive directory, they continue to be part of the transaction log, but they do not have to be on the same device as the JournalDirectory. Most often, a production installation of AMPS will keep journal files that are very active on fast storage and keep a longer period of history on storage that is higher capacity and lower cost. Since these devices typically also have lower throughput, these devices are best for files that must still be retained but are infrequently used.

Full details on these options are available in the Configuring a Transaction Log section of the AMPS User Guide.

Further Reading

See chapter on Record and Replay Messages in the AMPS User Guide for a more complete discussion of the transaction log and message replay.

The AMPS client libraries include samples for publishing messages and replaying messages from the transaction log. See the client library distribution for those samples.

AMPS includes functions that compute a CRC checksum over a string. This function is useful for creating a numeric identifier from a string representation. This is commonly-used to create a shortened representation of the string, or to provide input for a MOD calculation.

AMPS includes functions that return information about the currently connected client. As with the message functions, these functions return information about the client that prompted the operation, if one is present.

AMPS includes a function, REGEXP_MATCH, that returns the first match of text within a string. REGEXP_MATCH can be either a literal match or PCRE pattern to match the text to. If there is no match, then a NULL value is returned.

The following expressions all evaluate as true:

REGEXP_MATCH('sheepdog', 'dog') > 0

REGEXP_MATCH("30dogs", "^[0-9]*") == "30"

The following snippet shows REGEXP_MATCH constructed as a field:

<Field>COALESCE(REGEXP_MATCH(/petDescription, "cat|dog|bear"), "back away slowly, I don't know what that is") as /animal_type</Field>

With the following data:

{"petDescription": "tabby cat"}
{"petDescription": "havanese dog"}
{"petDescription": "grizzly bear"}
{"petDescription": "blue gerbil"}

Would result in:

{"petDescription":"tabby cat","animal_type":"cat"}
{"petDescription":"havanese dog","animal_type":"dog"}
{"petDescription":"grizzly bear","animal_type":"bear"}
{"petDescription":"blue gerbil","animal_type":"back away slowly, I don't know what that is"}

AMPS provides the UPPER and LOWER functions to produce a string in a specific case. This can be useful when constructing fields, or when an expression needs case-insensitive comparisons against a group of values using the IN clause.

As described above in String Comparison Functions, AMPS provides INSTR_I and STREQUAL_I functions for performing case-insensitive comparisons. In some cases, particularly when using strings with the IN clause, it is more efficient to simply convert the string to a known case.

The UPPER and LOWER functions are not unicode-aware; these functions will not produce the correct data when used with multibyte characters. For example, you might compare an incoming field of unknown case to a set of known values as follows:

UPPER(/ticker) IN ('MSFT', 'IBM', 'RHAT', 'DIS')

AMPS provides the ability for the server to conflate messages to a subscription. When a subscription requests conflation, the server will retain messages for that subscription for a certain period of time, the conflation interval, and provide the latest update to that message once a message has been retained for that interval. In effect, AMPS guarantees that a subscription will receive no more than one update for a given message per conflation interval.

Conflated subscriptions provide a way to reduce the bandwidth and processing for a subscriber in cases where a subscriber needs periodic updates with the current state of a message, rather than the complete message stream. AMPS provides per-subscription conflation for cases where only a small number of subscribers require conflation, or if conflation is required only in unusual cases. If multiple subscribers will have the same conflation needs, consider using .

For example, imagine an application that monitors selected stocks and displays the current prices on a large screen, which refreshes every few seconds. This application may use the same topics as a trading desk, but has very different needs for data freshness and completeness. Since updates to each symbol will only be displayed every few seconds, the application only needs point-in-time updates of the prices, rather than the full stream of price changes. To meet this need, the application could specify that the subscription conflates price updates by tickerId with a conflation interval of two seconds. For each distinct value of the tickerId field, AMPS will retain messages for two seconds. If another message with the same tickerId is processed for the subscription during the conflation interval, that message completely replaces the previous message. At the end of the two second conflation interval, the message is delivered to the application. This lets the application receive an up-to-date price at most every two seconds, without having to process a large number of updates that will never be displayed. This approach also ensures that the price is never more than two seconds out of date, which means that each time the screen is refreshed, the price is current.

As mentioned in the example above, if the subscription uses tickerId for conflation and the following sequence of messages arrive during a conflation interval:

AMPS delivers only the last message for that tickerId:

Notice that when a subscription is conflated, AMPS does not guarantee that messages are delivered precisely in the order in which they arrived in AMPS, since the latest update is delivered based on the conflation interval.

When the timestamp option is used with conflated subscriptions, AMPS provides the timestamp for the first message conflated.

When to Use Conflated Subscriptions

Conflated subscriptions reduce the bandwidth for a subscription, and may reduce the processing resources required for a subscription. However, rather than immediately delivering messages, AMPS retains messages in memory for the conflation interval. This can increase the memory required for the subscription.

AMPS contains other features for conflating messages and reducing bandwidth. Conflated subscriptions are most appropriate when:

• Network bandwidth is at a premium, and you would like AMPS to spend slightly more processing time and potentially more memory to reduce the bandwidth needs of the application.

• Each subscription has different conflation needs. For example, if each subscription has a dramatically different conflation interval, or needs to conflate by different fields. If most subscribers will use a similar conflation interval and use the same fields for conflation, using a Conflated Topic can provide equivalent results with lower overhead.

• The conflation needs are relatively predictable and consistent for the subscription. If you need the application to conflate messages only when processing is slow or there are bursts of message traffic, client-side conflation provides that ability and may be a better choice than a conflated subscription. See the developer guide for your programming language of choice for details.

The considerations above are general guidance to help you consider options and choose a conflation strategy.

You can also combine approaches as necessary. For example, if most of your subscriptions require a 3 second conflation interval by tickerId, while a few subscriptions require a 15 second interval, you could create a with a 3 second interval. Those subscriptions that require a 15 second interval could subscribe with that interval. This provides both sets of subscriptions with the intervals that they need.

Requesting Conflation on a Subscription

To request conflation on a subscription, set the following options on the subscription:

For example, to request a 10 second conflation interval with messages conflated on the [/orderId] field, you would use the following options string:

AMPS includes a set of functions designed to operate over an array element in a message and produce a value. These functions take an array within a single message as input, and reduce that array to a single value as output.

For many applications, messages need to be removed at specific times (for example, at the end of a trading day, or when the message reaches a certain state) rather than having a message-specific time to live.

These applications typically configure a scheduled maintenance plan using AMPS actions to manage the SOW and remove unneeded information.

For full details on AMPS actions, see the section in the User Guide.

Sample Maintenance Plan

Below is an example of a configuration section for a SOW topic definition, where records will need to be removed when they have reached a state of completed and been inactive for more than 24 hours. Since this is intended to manage the size of the saved state of the topic, it isn't necessary for messages to be removed precisely when they reach that state. Removing messages once a day, before activity begins for that day, is enough.

To create this maintenance plan, we configure an AMPS action that runs at 02:00 local time and removes the messages that the topic no longer needs.

Notice that there are two parts to this action. The On element specifies when the action should run -- in this case, every day at 02:00 local time. The Do element directs AMPS to delete messages from the ORDERS topic (of message type nvfix) where the /status is closed, and where the last update for the message is earlier than 24 hours (86400 seconds) from the time the action was started. At the scheduled time, AMPS internally runs a sow_delete command that removes the specified messages. This command is also written to the transaction log and replicated to other instances.

With this configuration, AMPS can efficiently maintain the SOW topic based on the needs of the application.

AMPS includes functions for explicitly constructing constant values of various types.