The AMPS transaction log supports durable subscriptions, reliable publish, and historical replay, while also serving as the foundation for high availability features in AMPS. To enable message recording and replay, configure a TransactionLog to keep a journal of messages published to an AMPS instance. For further details on using the transaction log, refer to other parts of this section - .
The sections on and cover how AMPS uses the TransactionLog (and other features) in high availability.
Described below are the configuration items for defining a TransactionLog. Expand each item for more details.
JournalDirectory (required)
Filesystem location where journal files will be stored.
This is the directory where AMPS will create new journal files as messages are recorded to the transaction log.
A journal directory should be dedicated to a single instance of AMPS.
JournalArchiveDirectory
Filesystem location where journal files are archived.
The archive directory is intended to allow older files to be stored on a higher-capacity (but potentially slower) device.
Journal files in this directory are part of the transaction log. AMPS can replicate from these files, provide bookmark replay from these files, distribute queue messages from these files, and so on.
Specifying that the JournalArchiveDirectory and the JournalDirectory are on the same storage device is not recommended.
A journal archive directory should be dedicated to a single instance of AMPS.
Use AMPS actions to move files from the JournalDirectory to the JournalArchiveDirectory.
PreallocatedJournalFiles
The number of journal files AMPS will create as part of the server startup.
Default: 2
Minimum: 1
JournalSize
Sets the target size for AMPS to use when calculating the size of journal files.
AMPS allocates journal files based on the size of an internal buffer. This option sets the target size for the journal file: AMPS will use the smallest file size that is an even multiple of the internal buffer without going under the specified JournalSize.
Notice that AMPS does not grow journal files once they are allocated. When a journal file is full, AMPS uses the next journal file.
AMPS accepts MinJournalSize as a synonym for JournalSize.
Default: 1GB
Minimum: 10M
Topic
The topic to include in the transaction log.
When no Topic is specified, AMPS initializes transaction log management for the instance, but does not persist messages. If a Topic is specified, all messages that exactly match the specified topic or regular expression will be included in the transaction log. If you want all topics of a specific message type to be persisted, use the regular expression .* for the name of the topic.
Multiple Topic elements can be included in a TransactionLog element. See the following section for the elements included within a Topic element.
To capture logical topics stored in a physical SOW topic (that is, to capture all of the topics within a State of the World Topic that uses a Pattern element), the Topic directive should match the Name of the physical topic. (The names of the logical topics within the physical topic do not matter in this case.)
There is no default for this element. If no Topic elements are configured, the transaction log will not record any messages.
FlushInterval
AMPS batches writes to the transaction log to optimize for maximum sustained throughput. If a batch is not full within a certain period of time, AMPS will write the partially-filled batch to the transaction log so that the messages can be replicated, delivered to subscribers, and so on. The interval at which messages will be flushed to the journal file during periods of slow activity is the FlushInterval.
Setting this explicitly has the potential to reduce latency during periods of low traffic, at the risk of somewhat lower performance during periods of higher traffic. Reducing this interval below the default may produce larger journal files during periods of low traffic, since AMPS may write more partial batches to the transaction log.
60East recommends leaving this option at the default setting unless the application is intended to optimize for low data rates and testing at production volumes on the storage that will be used for production demonstrates a performance advantage from reducing the interval.
Default: 100ms
Maximum: 1000ms
Minimum: 1us
O_DIRECT
Where supported, O_DIRECT will perform DMA directly from/to physical memory to a user space buffer. Having this enabled can improve AMPS performance, however not all devices support O_DIRECT.
When O_DIRECT is disabled, data loss can occur because the operating system can acknowledge that data has been written to the device before the actual write has happened.
60East does not recommend disabling O_DIRECT unless the device that holds the transaction log does not support O_DIRECT.
Default: enabled
InactiveClientAckExpiration
Sets the amount of time to retain records for an inactive publisher. On recovery, AMPS will remove the clients.ack entry for any publisher that has not published a message for longer than the interval set in this option.
This option can help reduce clients.ack growth for installations that have large numbers of short-lived publishers. If this option is specified, the interval should be longer than the expected time that a given publisher would be inactive and longer than the time that messages from that publisher would be retained.
For example, if an application has publishers that are active at the end of every week and the installation retains journals for 14 days, the interval should be longer than 14 days.
Default: Retain records indefinitely.
CompressedJournalCacheMemoryLimit
Sets the maximum amount of server memory to use for caching compressed journal files when one or more bookmark replays is using a compressed journal file.
Increasing this setting can improve performance when replaying bookmark subscriptions from compressed journals, in exchange for temporarily consuming more memory to service those replays.
Default: 10% of server memory or 10GB, whichever is lower.
If Topic is included in the TransactionLog configuration, it must contain the following elements. Expand each item for more details.
Name (required)
The name of the topic to record.
This element can be a literal name or a regular expression.
This element is required and there is no default.
MessageType (required)
The message type of the topic.
This must be one of the message types loaded by default or a message type declared in the configuration file.
This element is required and there is no default.
The following example demonstrates a transaction log where the journal file will be written to ./amps/journal. When AMPS starts, a single journal file will be pre-allocated as noted by the PreallocatedJournalFiles setting; and when the first journal file is completely full, a new journal file will be created.
This journal will contain those messages which match the topic orders and have a message type of nvfix and also, any messages published to a topic that contains the string LOGGED_ of a message type of json.
The journal will contain messages published to the virtual topics within the physical SOW topic named bucket, even though the topics for those messages do not contain the word bucket.
<!-- All transaction log definitions are contained within the TransactionLog block.
The following global settings apply to all Topic blocks defined within the
TransactionLog: JournalDirectory, PreallocatedJournalFiles, and JournalSize. -->
<TransactionLog>
<!-- The JournalDirectory is the filesystem location where journal files and
journal index files will be stored. -->
<JournalDirectory>./amps/journal/</JournalDirectory>
<!-- The JournalArchiveDirectory is the filesystem location to which AMPS will
archive journals. Notice that AMPS does not archive files by default. You
configure an action to archive journal files -->
<JournalArchiveDirectory>/mnt/somedev0/amps/journal</JournalArchiveDirectory>
<!-- PreallocatedJournalFiles defines the number of journal files AMPS will
create as part of the server startup. Default: 2 Minimum: 1 -->
<PreallocatedJournalFiles>1</PreallocatedJournalFiles>
<!-- The JournalSize is the approximate size of the journal files that AMPS
will create. -->
<JournalSize>10MB</JournalSize> <!-- Suitable for development work on
a local development system.
A production deployment would
typically use 250MB to 2GB
depending on message size,
message volume and retention
policy. -->
<!-- When a Topic is specified, then all messages which match exactly the
specified topic or regular expression will be included in the transaction
log. Otherwise, AMPS initializes the transaction logging, but does not
record any messages to the transaction log. -->
<Topic>
<Name>orders</Name>
<MessageType>nvfix</MessageType>
</Topic>
<Topic>
<Name>LOGGED_.*</Name>
<MessageType>json</MessageType>
</Topic>
<Topic>
<Name>bucket</Name>
<MessageType>json</MessageType>
</Topic>
</TransactionLog>
<SOW>
... other configuration here ...
<Topic>
<Name>bucket</Name>
<MessageType>json</MessageType>
<Pattern>.*-cached-values</Pattern>
<Key>/msgId</Key>
<FileName>./sow/%n.sow</FileName>
</Topic>
</SOW>
