LogoLogo
AMPS Server Documentation 5.3.4
AMPS Server Documentation 5.3.4
  • Welcome to AMPS 5.3.4
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
      • Production Configuration
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Configuring a Transaction Log
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Advanced Queue Configuration
      • Queue Subscriptions Compared to Bookmark Replays
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Configuring Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to a File
      • Logging to a Compressed File
      • Logging to Syslog
      • Logging to the Console
      • Looking up Errors with ampserr
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Automating AMPS with Actions
    • Replicating Messages Between Instances
      • Replication Basics
      • Configuring Replication
      • Replication Configuration Validation
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
      • Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Optional Functions
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Optional SOW Key Generator
        • Chaining Key Generator
      • Optional Authentication/Entitlements Modules
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • Simple Access Entitlements Module
      • Optional Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • AMPS Statistics
    • File Format Versions
  • AMPS Configuration Guide
    • AMPS Configuration Basics
      • Getting Started With AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working With Configuration Files
      • Including External Files
    • Instance Level Configuration
    • Admin Server and Statistics
    • Modules
    • Message Types
    • Transports
    • Logging
    • State of the World (SOW)
      • SOW/Topic
      • SOW/*Queue
      • SOW/ConflatedTopic
      • SOW/View
    • Replication
      • Replication Validation
    • Transaction Log
    • Authentication
    • Entitlement
    • Actions
      • Configuration for Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Protocols
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu
      • disks
      • memory
      • name
      • network
    • AMPS Instance Statistics
      • api
      • clients
      • config.xml
      • config_path
      • conflated_topics
      • cpu
      • cwd
      • description
      • environment
      • lifetimes
      • logging
      • memory
      • message_types
      • name
      • name_hash
      • pid
      • processors
      • queues
      • queries
      • replication
      • sow
      • statistics
      • subscriptions
      • timestamp
      • transaction_log
      • transports
      • tuning
      • uptime
      • user_id
      • version
      • views
  • AMPS Command Reference
    • Commands to AMPS
      • logon
      • Publishing
        • publish
        • delta_publish
      • Subscribing to and Querying Topics
        • subscribe
        • sow
        • sow_and_subscribe
        • unsubscribe
        • delta_subscribe
        • sow_and_delta_subscribe
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat
      • flush
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end : Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
  • AMPS Concepts
  • Feature Highlights
Export as PDF
  1. Introduction to AMPS

Overview of AMPS

AMPS Concepts

This section describes the overall AMPS approach and the features AMPS provides.

The AMPS messaging system is designed around a few simple principles:

  • Parallelize work and minimize waits and blocking to take full advantage of modern multisocket, multicore systems.

  • Eliminate redundant or unused work by only performing tasks that are necessary to provide the functionality requested by a given operation.

  • Reduce or eliminate cross-system coordination by solving the full range of data delivery and storage problems commonly faced by data-intensive applications.

  • Provide a small, flexible set of commands for ease of use.

  • Provide multiple delivery paradigms supporting both publish-subscribe delivery (many to many) and message queues (single consumption of a message) as well as the ability to query the state of a topic at a point in time.

  • Stay application-focused to provide exactly the capabilities that are heavily used in demanding high-performance applications.

  • Stay hardware aware and build for the future by engineering for next-generation commodity hardware and designing AMPS to fully exploit non-uniform memory access (NUMA), flash-based storage, and high-bandwidth networking.

These concepts are the foundation of how AMPS works and are helpful for understanding how to best use AMPS.

To best take advantage of AMPS, applications typically use the built-in features of AMPS rather than their traditional equivalents.

For example, rather than keeping a separate, independent record of each message published to AMPS for audit purposes, applications most often use one of the data persistence features in AMPS. This speeds development and simplifies deployment by eliminating integration effort, and also solves potential correctness issues which could be caused if messages in persistent storage become inconsistent with the messages provided through the messaging system. With AMPS, the messaging system itself can contain a fully-queryable and replayable record of the system.

As another example, AMPS provides integrated replication rather than relying on an external process. AMPS replication is aware of the format and semantics of the transaction log, the configuration of the instance, and the commands sent by publishers. This integration allows AMPS to very efficiently provide a full-fidelity message stream and to provide "self-healing" for an instance to catch up when it has been offline. Further, the message store used for replication (the AMPS transaction log) is also used for durable subscriptions and message replay. Designing and implementing these features together reduces complexity, storage requirements, and overhead to enable both capabilities. Within the AMPS server, the implementation uses a sophisticated parallelized algorithm for storage and replay that reduces overall latency and prevents slow consumers or replication destinations from affecting faster consumers. The overall result is to simplify configuration and application development, provide strong consistency and reliability guarantees, and provide the highest possible level of performance.

As a final example, rather than requiring a complex topic structure, requiring applications to oversubscribe and discard messages, AMPS provides both topic filtering and content filtering. AMPS includes an expressive filter grammar to provide precise selection of messages of interest to an individual subscriber. AMPS provides this capability to fully decouple publishers and subscribers. With AMPS, there's no need to maintain and administer a granular topic structure. Precise filtering and routing improves both network and processor utilization by providing only actionable messages to a subscriber. Likewise, for many applications, there is no need for a publisher to be aware of the processing performed by the subscriber or by AMPS itself.

The examples above highlight just a few of the capabilities AMPS provides and how the AMPS approach simplifies development, administration, and operations while providing reliability and performance benefits over conventional systems.

Feature Highlights

Some of the highlights of AMPS features include:

  • Topic based publish and subscribe, including full support for regular expressions to specify topic names.

  • Content filtering based on XPath identifiers (to specify the fields of a message) and SQL-92 (to form a predicate), with added support for Perl-Compatible Regular Expressions (PCRE2).

  • Message queues including content filtering for both publishers and subscribers, configurable strategies for delivery fairness, and truly distributed queues that can efficiently enforce queue semantics and delivery guarantees across a replicated network of AMPS instances.

  • Content-aware messaging support for a wide range of message types, including standard formats such as JSON, FIX, MessagePack, XML, Google Protocol Buffers, and BSON. AMPS also supports simple key/value pairs in FIX format (called NVFIX to emphasize that the format uses name/value pairs rather than FIX tags), and a high-performance binary protocol called BFlat. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing types to easily combine messages of different types in a single payload.

  • An integrated database and record-aware current value storage (called State of the World, or SOW), with optional historical query capability.

  • Historical replay of message streams, including the ability to preserve the total message ordering across independent topics.

  • Integrated replication and high availability, including automatic resynchronization for instances that fail over.

  • Aggregation and Complex Event Processing (CEP), including the ability to aggregate information across different message streams and message streams of different formats.

  • Advanced messaging capabilities such as atomic query-and-subscribe, incremental (delta) updates, and out-of-focus notifications that tell a subscription when a record no longer matches.

  • Built in statistics and monitoring, with data provided via a standard RESTful interface.

  • Integrated authentication and entitlement across all AMPS features.

  • Actions for automating AMPS functionality, including both routine maintenance tasks and dataflow-aware processing (such as alerting in response to slowdowns or invalid data).

  • Client development kits for popular programming languages such as Java, C#/.NET, C++, Python, JavaScript, and Go.

  • Extensibility API in the AMPS server for adding message types, extending the functions available to the AMPS query language, adding new actions, integrating with enterprise authentication and entitlement systems, and more.

PreviousIntroduction to AMPSNextGetting Started With AMPS

Last updated 1 year ago

This guide provides an overview of the most commonly used functionality of AMPS, but it is not intended to cover all of the features of AMPS or provide an exhaustive discussion of any individual feature. As mentioned earlier, the provides full details about AMPS features.

AMPS User Guide