LogoLogo
AMPS Server Documentation 5.3.4
AMPS Server Documentation 5.3.4
  • Welcome to AMPS 5.3.4
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
      • Production Configuration
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Configuring a Transaction Log
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Advanced Queue Configuration
      • Queue Subscriptions Compared to Bookmark Replays
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Configuring Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to a File
      • Logging to a Compressed File
      • Logging to Syslog
      • Logging to the Console
      • Looking up Errors with ampserr
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Automating AMPS with Actions
    • Replicating Messages Between Instances
      • Replication Basics
      • Configuring Replication
      • Replication Configuration Validation
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
      • Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Optional Functions
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Optional SOW Key Generator
        • Chaining Key Generator
      • Optional Authentication/Entitlements Modules
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • Simple Access Entitlements Module
      • Optional Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • AMPS Statistics
    • File Format Versions
  • AMPS Configuration Guide
    • AMPS Configuration Basics
      • Getting Started With AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working With Configuration Files
      • Including External Files
    • Instance Level Configuration
    • Admin Server and Statistics
    • Modules
    • Message Types
    • Transports
    • Logging
    • State of the World (SOW)
      • SOW/Topic
      • SOW/*Queue
      • SOW/ConflatedTopic
      • SOW/View
    • Replication
      • Replication Validation
    • Transaction Log
    • Authentication
    • Entitlement
    • Actions
      • Configuration for Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Protocols
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu
      • disks
      • memory
      • name
      • network
    • AMPS Instance Statistics
      • api
      • clients
      • config.xml
      • config_path
      • conflated_topics
      • cpu
      • cwd
      • description
      • environment
      • lifetimes
      • logging
      • memory
      • message_types
      • name
      • name_hash
      • pid
      • processors
      • queues
      • queries
      • replication
      • sow
      • statistics
      • subscriptions
      • timestamp
      • transaction_log
      • transports
      • tuning
      • uptime
      • user_id
      • version
      • views
  • AMPS Command Reference
    • Commands to AMPS
      • logon
      • Publishing
        • publish
        • delta_publish
      • Subscribing to and Querying Topics
        • subscribe
        • sow
        • sow_and_subscribe
        • unsubscribe
        • delta_subscribe
        • sow_and_delta_subscribe
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat
      • flush
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end : Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
  • Guaranteed Publishing
  • Durable Publication and Subscriptions
  • Durable Subscriptions
  • Durable Publishing
  • Client Support
  • Heartbeat in High Availability
Export as PDF
  1. AMPS User Guide
  2. Highly Available AMPS Installations

Details of High Availability

PreviousExample: Hub and Spoke / Expandable MeshNextSlow Client Management and Capacity Limits

Last updated 5 months ago

AMPS High Availability, which includes multi-site replication and the transaction log, is designed to provide long uptimes and speedy recovery from disasters. Replication allows deployments to improve upon the already rock-solid stability of AMPS. Additionally, AMPS journaling provides the persisted state necessary to make sure that client recovery is fast, painless, and error free.

Guaranteed Publishing

An interruption in service while publishing messages could be disastrous if the publisher doesn't know which message was last persisted to AMPS. To prevent this from happening, AMPS has support for guaranteed publishing.

With guaranteed publishing, the AMPS client library is responsible for retaining and retransmitting the message until the server acknowledges that the message has been successfully persisted to the server and has been acknowledged as persisted by any replication destinations that are configured for synchronous replication. This means that each message always has at least one part of the system (either the client library or the AMPS server) responsible for persisting the message, and if failover occurs, that part of the system can retain and recover the message as necessary.

An important part of guaranteed publishing is to be able to uniquely identify messages. In AMPS, the unique identifier for a message is a bookmark, which is formed from a combination of a number derived from the client name and a sequence number managed by the client. A sequence number is simply an ever-increasing number assigned by a publisher to any operation that changes the state of persistent storage in AMPS (that is, publish or sow_delete commands).

The AMPS clients automatically manage sequence numbers when applications use the named methods or the Command interface. The libraries set the sequence number on each published message, ensure that the sequence number increases as appropriate, and initialize the sequence number at logon using information retrieved from the server acknowledgment of the logon command. The sequence number is also used for acknowledgments. The persisted acknowledgment returned in response to a publish command contains the sequence number of the last message persisted rather than the CommandId of the publish command message (for more details see ).

The logon command supports a processed acknowledgment message, which will return the Sequence of the last record that AMPS has persisted. When the processed acknowledgment message is returned to the publisher, the Sequence corresponds to the last message persisted by AMPS. The publisher can then use that sequence to determine if it needs to 1) re-publish messages that were not persisted by AMPS, or 2) continue publishing messages from where it left off. Acknowledging persisted messages across logon sessions allows AMPS to guarantee publishing. The HAClient classes in the AMPS clients manage sequence numbers, including setting a meaningful initial sequence number based on the response from the logon command, automatically.

Connections should request a processed acknowledgment message with every logon command. This ensures that the Sequence returned in the acknowledgment message matches the publisher's last published message. The 60East AMPS clients do this automatically when using the named logon methods. If you are building the command yourself or using a custom client, you may need to add this request to the command yourself.

In addition to the acknowledgment messages, AMPS also keeps track of the published messages from a client based on the client's name. The client name is set during the logon command, so to set a consistent client name, it is necessary for an application to log on to AMPS. A logon is required by default in AMPS versions 5.0 and later, and optional by default in AMPS versions previous to 5.0.

All publishers must set a unique client name field when logging on to AMPS. This allows AMPS to correlate the sequence numbers of incoming publish messages to a specific client, which is required for reliable publishing, replication, and duplicate detection in the server. In the event that multiple publishers have the same client name, AMPS can no longer reliably correlate messages using the publish sequence number and client name.

When a transaction log is enabled for AMPS, it is an error for two clients to connect to an instance with the same name.

Durable Publication and Subscriptions

The AMPS client libraries include features to enable durable subscription and durable publication. In this chapter we've covered how publishing messages to a transaction log persists them. We've also covered how the transaction log can be queried (subscribed) with a bookmark for replay. Now, putting these two features together yields durable subscriptions.

Durable Subscriptions

A durable subscriber is one that receives all messages published to a topic (including a regular expression topic), even when the subscriber is offline. In AMPS this is accomplished through the use of the bookmark subscription on a client. Implementation of a durable subscription in AMPS is accomplished on the client by persisting the last observed bookmark field received from a subscription. This enables a client to recover and resubscribe from the exact point in the transaction log where it left off.

Durable Publishing

A durable publisher maintains a persistent record of messages published until AMPS acknowledges that the message has been persisted. In the AMPS system, a durable publisher stores outgoing messages until AMPS sends a persisted acknowledgement that indicates the message has been persisted or cannot be persisted due to an error. Once the message is acknowledged, the publisher can remove the message. Should the publisher fail over, it can resend any messages in the store that have not been acknowledged by AMPS.

The AMPS server uses the sequence number in the message to discard any duplicates. This helps ensure that no messages are lost, and provides fault-tolerance for publishers. The sequence number is also used in persisted acknowledgement messages to indicate which message, or messages, the acknowledgement applies to.

The the AMPS client libraries provide publish stores that manage durable publishing and maintaining and assigning sequence numbers to messages.

Client Support

The AMPS client libraries each provide different implementations of persistent subscriptions and persistent publication. Please refer to the High Availability chapter of the Developer Guide for the language of your choice to see how this feature is implemented.

Heartbeat in High Availability

Use of the heartbeat feature allows your application to quickly recover from detected connection failures. By default, connection failure detection occurs when AMPS receives an operating system error on the connection. This default method may result in unpredictable delays in detecting a connection failure on the client, particularly when failures in network routing hardware occur, and the client primarily acts as a subscriber.

The heartbeat feature of the AMPS server and the AMPS clients allows connection failure to be detected quickly. Heartbeats ensure that regular messages are sent between the AMPS client and server on a predictable schedule. The AMPS server assumes disconnection has occurred if these regular heartbeats cease, ensuring disconnection is detected in a timely manner.

Heartbeats are initialized by the AMPS client by sending a heartbeat message to the AMPS server. To enable heartbeats in your application, refer to the High Availability chapter in the Developer Guide for your specific client language.

To reduce network bandwidth, the AMPS server will conflate successful persisted acknowledgements when a transaction log is configured. That is, the server will acknowledge messages at a regular interval, as described in the section on .

Acknowledgement Conflation and Publish Acknowledgements
Acknowledgement Conflation and Publish Acknowledgements