Overview of AMPS

AMPS Concepts

This section describes the overall AMPS approach and the features AMPS provides.

The AMPS messaging system is designed around a few simple principles:

• Parallelize work and minimize waits and blocking to take full advantage of modern multisocket, multicore systems.

• Eliminate redundant or unused work by only performing tasks that are necessary to provide the functionality requested by a given operation.

• Reduce or eliminate cross-system coordination by solving the full range of data delivery and storage problems commonly faced by data-intensive applications.

• Provide a small, flexible set of commands for ease of use.

• Provide multiple delivery paradigms supporting both publish-subscribe delivery (many to many) and message queues (single consumption of a message) as well as the ability to query the state of a topic at a point in time.

• Stay application-focused to provide exactly the capabilities that are heavily used in demanding high-performance applications.

• Stay hardware aware and build for the future by engineering for next-generation commodity hardware and designing AMPS to fully exploit non-uniform memory access (NUMA), flash-based storage, and high-bandwidth networking.

These concepts are the foundation of how AMPS works and are helpful for understanding how to best use AMPS.

To best take advantage of AMPS, applications typically use the built-in features of AMPS rather than their traditional equivalents.

For example, rather than keeping a separate, independent record of each message published to AMPS for audit purposes, applications most often use one of the data persistence features in AMPS. This speeds development and simplifies deployment by eliminating integration effort, and also solves potential correctness issues which could be caused if messages in persistent storage become inconsistent with the messages provided through the messaging system. With AMPS, the messaging system itself can contain a fully-queryable and replayable record of the system.

As another example, AMPS provides integrated replication rather than relying on an external process. AMPS replication is aware of the format and semantics of the transaction log, the configuration of the instance, and the commands sent by publishers. This integration allows AMPS to very efficiently provide a full-fidelity message stream and to provide "self-healing" for an instance to catch up when it has been offline. Further, the message store used for replication (the AMPS transaction log) is also used for durable subscriptions and message replay. Designing and implementing these features together reduces complexity, storage requirements, and overhead to enable both capabilities. Within the AMPS server, the implementation uses a sophisticated parallelized algorithm for storage and replay that reduces overall latency and prevents slow consumers or replication destinations from affecting faster consumers. The overall result is to simplify configuration and application development, provide strong consistency and reliability guarantees, and provide the highest possible level of performance.

As a final example, rather than requiring a complex topic structure, requiring applications to oversubscribe and discard messages, AMPS provides both topic filtering and content filtering. AMPS includes an expressive filter grammar to provide precise selection of messages of interest to an individual subscriber. AMPS provides this capability to fully decouple publishers and subscribers. With AMPS, there's no need to maintain and administer a granular topic structure. Precise filtering and routing improves both network and processor utilization by providing only actionable messages to a subscriber. Likewise, for many applications, there is no need for a publisher to be aware of the processing performed by the subscriber or by AMPS itself.

The examples above highlight just a few of the capabilities AMPS provides and how the AMPS approach simplifies development, administration, and operations while providing reliability and performance benefits over conventional systems.

Feature Highlights

Some of the highlights of AMPS features include:

• Topic based publish and subscribe, including full support for regular expressions to specify topic names.

• Content filtering based on XPath identifiers (to specify the fields of a message) and SQL-92 (to form a predicate), with added support for Perl-Compatible Regular Expressions (PCRE2).

• Message queues including content filtering for both publishers and subscribers, configurable strategies for delivery fairness, and truly distributed queues that can efficiently enforce queue semantics and delivery guarantees across a replicated network of AMPS instances.

• Content-aware messaging support for a wide range of message types, including standard formats such as JSON, FIX, MessagePack, XML, Google Protocol Buffers, and BSON. AMPS also supports simple key/value pairs in FIX format (called NVFIX to emphasize that the format uses name/value pairs rather than FIX tags), and a high-performance binary protocol called BFlat. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing types to easily combine messages of different types in a single payload.

• An integrated database and record-aware current value storage (called State of the World, or SOW), with optional historical query capability.

• Historical replay of message streams, including the ability to preserve the total message ordering across independent topics.

• Integrated replication and high availability, including automatic resynchronization for instances that fail over.

• Aggregation and Complex Event Processing (CEP), including the ability to aggregate information across different message streams and message streams of different formats.

• Advanced messaging capabilities such as atomic query-and-subscribe, incremental (delta) updates, and out-of-focus notifications that tell a subscription when a record no longer matches.

• Built in statistics and monitoring, with data provided via a standard RESTful interface.

• Integrated authentication and entitlement across all AMPS features.

• Actions for automating AMPS functionality, including both routine maintenance tasks and dataflow-aware processing (such as alerting in response to slowdowns or invalid data).

• Client development kits for popular programming languages such as Java, C#/.NET, C++, Python, JavaScript, and Go.

• Extensibility API in the AMPS server for adding message types, extending the functions available to the AMPS query language, adding new actions, integrating with enterprise authentication and entitlement systems, and more.

This guide provides an overview of the most commonly used functionality of AMPS, but it is not intended to cover all of the features of AMPS or provide an exhaustive discussion of any individual feature. As mentioned earlier, the AMPS User Guide provides full details about AMPS features.