LogoLogo
AMPS Server Documentation 5.3.4
AMPS Server Documentation 5.3.4
  • Welcome to AMPS 5.3.4
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
      • Production Configuration
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Configuring a Transaction Log
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Advanced Queue Configuration
      • Queue Subscriptions Compared to Bookmark Replays
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Configuring Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to a File
      • Logging to a Compressed File
      • Logging to Syslog
      • Logging to the Console
      • Looking up Errors with ampserr
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Automating AMPS with Actions
    • Replicating Messages Between Instances
      • Replication Basics
      • Configuring Replication
      • Replication Configuration Validation
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
      • Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Optional Functions
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Optional SOW Key Generator
        • Chaining Key Generator
      • Optional Authentication/Entitlements Modules
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • Simple Access Entitlements Module
      • Optional Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • AMPS Statistics
    • File Format Versions
  • AMPS Configuration Guide
    • AMPS Configuration Basics
      • Getting Started With AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working With Configuration Files
      • Including External Files
    • Instance Level Configuration
    • Admin Server and Statistics
    • Modules
    • Message Types
    • Transports
    • Logging
    • State of the World (SOW)
      • SOW/Topic
      • SOW/*Queue
      • SOW/ConflatedTopic
      • SOW/View
    • Replication
      • Replication Validation
    • Transaction Log
    • Authentication
    • Entitlement
    • Actions
      • Configuration for Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Protocols
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu
      • disks
      • memory
      • name
      • network
    • AMPS Instance Statistics
      • api
      • clients
      • config.xml
      • config_path
      • conflated_topics
      • cpu
      • cwd
      • description
      • environment
      • lifetimes
      • logging
      • memory
      • message_types
      • name
      • name_hash
      • pid
      • processors
      • queues
      • queries
      • replication
      • sow
      • statistics
      • subscriptions
      • timestamp
      • transaction_log
      • transports
      • tuning
      • uptime
      • user_id
      • version
      • views
  • AMPS Command Reference
    • Commands to AMPS
      • logon
      • Publishing
        • publish
        • delta_publish
      • Subscribing to and Querying Topics
        • subscribe
        • sow
        • sow_and_subscribe
        • unsubscribe
        • delta_subscribe
        • sow_and_delta_subscribe
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat
      • flush
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end : Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
Export as PDF
  1. AMPS Configuration Guide

Transaction Log

PreviousReplication ValidationNextAuthentication

Last updated 3 months ago

AMPS includes the ability to record and replay messages. This capability can be used by applications for durable subscriptions, reliable publish, and historical replay. The AMPS transaction log is also the foundation of the high availability features in AMPS. To enable message recording and replay, configure a TransactionLog to keep a journal of messages published to an AMPS instance.

The section of the covers how to use the transaction log for historical replay, durable publish, and durable subscriptions.

The AMPS User Guide sections on and cover how AMPS uses the TransactionLog (and other features) in high availability.

Element

Description

JournalDirectory (required)

Filesystem location where journal files will be stored.

This is the directory where AMPS will create new journal files as messages are recorded to the transaction log.

A journal directory should be dedicated to a single instance of AMPS.

JournalArchiveDirectory

Filesystem location where journal files are archived.

The archive directory is intended to allow older files to be stored on a higher-capacity (but potentially slower) device.

Journal files in this directory are part of the transaction log. AMPS can replicate from these files, provide bookmark replay from these files, distribute queue messages from these files, and so on. Specifying that the JournalArchiveDirectory and the JournalDirectory are on the same storage device is not recommended.

A journal archive directory should be dedicated to a single instance of AMPS.

Use AMPS actions to move files from the JournalDirectory to the JournalArchiveDirectory.

PreallocatedJournalFiles

The number of journal files AMPS will create as part of the server startup.

Default: 2

Minimum: 1

JournalSize

Sets the target size for AMPS to use when calculating the size of journal files.

AMPS allocates journal files based on the size of an internal buffer. This option sets the target size for the journal file: AMPS will use the smallest file size that is an even multiple of the internal buffer without going under the specified JournalSize. Notice that AMPS does not grow journal files once they are allocated. When a journal file is full, AMPS uses the next journal file.

AMPS accepts MinJournalSize as a synonym for JournalSize.

Default: 1GB

Minimum: 10M

Topic

The topic to include in the transaction log.

When no Topic is specified, AMPS initializes transaction log management for the instance, but does not persist messages. If a Topic is specified, then all messages which match exactly the specified topic or regular expression will be included in the transaction log. If you want all topics of a specific message type to be persisted, use the regular expression .* for the name of the topic.

Multiple Topic elements can be included in a TransactionLog element. See the following table for the elements included within a Topic element.

There is no default for this element. If no Topic elements are configured, the transaction log will record no messages.

FlushInterval

AMPS batches writes to the transaction log to optimize for maximum sustained throughput. If a batch is not full within a certain period of time, AMPS will write the partially-filled batch to the transaction log so that the messages can be replicated, delivered to subscribers, and so on. The interval at which messages will be flushed to the journal file during periods of slow activity is the FlushInterval.

Setting this explicitly has the potential to reduce latency during periods of low traffic, at the risk of somewhat lower performance during periods of higher traffic. Reducing this interval below the default may produce larger journal files during periods of low traffic, since AMPS may write more partial batches to the transaction log.

60East recommends leaving this option at the default unless the application is intended to optimize for low data rates and testing at production volumes on the storage that will be used for production demonstrates a performance advantage from reducing the interval.

Default: 100ms

Maximum: 1000ms

Minimum: 1us

O_DIRECT

Where supported, O_DIRECT will perform DMA directly from/to physical memory to a user space buffer. Having this enabled can improve AMPS performance, however not all devices support O_DIRECT. When O_DIRECT is disabled, data loss can occur because the operating system can acknowledge that data has been written to the device before the actual write has happened.

60East does not recommend disabling O_DIRECT unless the device that holds the transaction log does not support O_DIRECT.

Default: enabled

InactiveClientAckExpiration

Sets the amount of time to retain records for an inactive publisher. On recovery, AMPS will remove the clients.ack entry for any publisher that has not published a message for longer than the interval set in this option.

This option can help reduce clients.ack growth for installations that have large numbers of short-lived publishers. If this option is specified, the interval should be longer than the expected time that a given publisher would be inactive and longer than the time that messages from that publisher would be retained. For example, if an application has publishers that are active at the end of every week and the installation retains journals for 14 days, the interval should be longer than 14 days.

Default: Retain records indefinitely.

CompressedJournalCacheMemoryLimit

Sets the maximum amount of server memory to use for caching compressed journal files when one or more bookmark replays is using a compressed journal file.

Increasing this setting can improve performance when replaying bookmark subscriptions from compressed journals, in exchange for temporarily consuming more memory to service those replays.

Default: 10% of server memory or 10GB, whichever is lower.

The Topic element in a TransactionLog can contain the following elements:

Element

Description

Name (required)

The name of the topic to record.

This element can be a literal name or a regular expression.

To capture logical topics stored in a physical SOW topic (that is, to capture a State of the World Topic that uses a Pattern element), the Topic directive should match the Name of the physical topic. (The names of the logical topics within the physical topic do not matter in this case).

This element is required. There is no default for this element.

MessageType (required)

The message type of the topic.

This must be one of the message types loaded by default or a message type declared in the configuration file.

This element is required. There is no default for this element.

The following example demonstrates a transaction log where the journal file will be written to ./amps/journal. When AMPS starts, a single journal file will be pre-allocated as noted by the PreallocatedJournalFiles setting; and when the first journal file is completely full, a new journal file will be created. This journal will contain those messages which match the topic orders and also have a message type of fix and any messages published to a topic that contains the string LOGGED_ of a message type of json.

<AMPSConfig>
    ...

    <TransactionLog>
        <JournalDirectory>./amps/journal/</JournalDirectory>
        <PreallocatedJournalFiles>1</PreallocatedJournalFiles>
        <JournalSize>10MB</JournalSize>
        <Topic>
            <Name>orders</Name>
            <MessageType>nvfix</MessageType>
        </Topic>
        <Topic>
            <Name>LOGGED_.*</Name>
            <MessageType>json</MessageType>
        </Topic>
    </TransactionLog>

    ...
</AMPSConfig>
Record and Replay Messages
AMPS User Guide
Replicating Messages Between Instances
Highly Available AMPS Installations