Dump journal File

The AMPS journal dump utility is used in examining the contents of an AMPS journal file for debugging and program tuning. The amps_journal_dump utility is most commonly used as a tool to debug the forensic lifespan of messages that have previously been published to AMPS. The amps_journal_dump tool is used to show that messages exist in a journaled topic, and to show the order the message was received in, and the timestamp associated with the message.

Command Line Options

The amps_journal_dump program has the following options available. These can also be printed to the screen by typing amps_journal_dump -help.

Option	Description
filename (required)	Filename of the AMPS journal file.
-h, --help	Show the program help message and quit.
-l LIMIT	Limit range of output to entries N:M where N is the first entry and M is the last entry. Passing in a single value, M, will return the first M results.
--localtime	Display ISO 8601 timestamp in local time.
--extents	Add local and replication extents information at the end of the journal dump.
--no-data	Do not display data values in the journal dump output.
--include-noops	Include noops in the journal dump.

Looking at the Output

In this section will examine some sample output from running amps_journal_dump. We will then go over what each of the entries emitted by the program means.

File Name       : AMPS-Sample.0000000000.journal
File Size       : 10485760
Version         : amps::txlog/v8
Extents         : [1623783047000000005:1623783047000002563]
First Timestamp : 20210615T185233.126619Z
Last Timestamp  : 20210615T185234.032520Z
______________________________________________________________
entry                         :                              0
crc32                         :                     2427361060
type                          :                        publish
flags                         :                           none
file offset                   :                           4096
tx byte count                 :                            256
msg byte count                :                             21
msg type                      :                           json
local txid                    :            1711058949000000003
previous local txid           :                              0
source txid                   :           14124379235191657289
source name hash              :                              0
client name hash              :           15749960451245477525
client seq                    :            1711058949000000003
topic hash                    :           10912813577604266571
sow expiration time           :                              0
iso8601 timestamp             :        20240321T220952.535075Z
timestamp                     :             212577862192535075
previous byte count           :                              0
topic byte count              :                             13
topic                         : [sample-record]
data                          : [{"message":"example"}]
correlation id byte count     :                              0
correlation id                : []
auth id byte count            :                              0
auth id                       : []
rep path byte count           :                              0
rep path                      : []
_____________________________________________________

...

Total Entries    : 2559
Total Bytes      : 10485760
Remaining Bytes  : 0

As is apparent in the listing, the output from amps_journal_dump is split into three sections, a header, a listing of the contents of the journal file and a footer.

The header contains general information about the journal file as it is represented in the filesystem, and state data about the journal file.

Field	Description
File Name	The name of the file as it appears on the local filesystem.
File Size	Number of total bytes allocated by the journal file.
Version	This is the version of the formatting used to write the data in the journal file.
Extents	A pair of numbers where the first is the number of extents used by the journal file, and the second is the number of blocks allocated for the journal file.

The second section of the amps_journal_dump lists each of the entries contained in the journal file, along with all of the meta-data used to track and describe the entry. For the sake of simplicity, the example only shows a single journal entry. In an actual installation, amps_journal_dump will produce a record for every message in the journal.

The exact options present depend on the version of the journal file.

Field	Description
entry	A monotonically increasing value representing the order in which the record was inserted into the transaction log file.
crc32	The cyclic redundancy check used for error checking the transaction log entry.
type	The AMPS command used in the original message or the type of entry in the journal file.
file offset	Offset for this entry within the journal file.
tx byte count	Size of this entry within the journal file.
msg byte count	Number of bytes in the message payload.
msg type	Message type of the message.
local txid	The monotonically increasing identifier used across all records local transaction log journal files.
previous local txid	The monotonically increasing identifier used across all records local transaction log journal files.
source txid	The monotonically increasing identifier used across all records local transaction log journal files. If a publish from a client does not include a sequence number, and the publish was made directly to this instance, this field can be used to hold the original client name hash for a publish or delete. In that case, the client name hash used for message identification purposes is generated by AMPS (to avoid message identity conflicts).
source name hash	For replicated messages, the hash of the instance from which the message was received.
client name hash	The hash of the publisher client name. The combination of the client name hash and the client seq are used as the identity of the message. If a publish or delete does not include a sequence number, and the publish was made directly to this instance, this field may be a hash generated by AMPS while the original client name hash is stored in the source txid field.
client seq	The sequence number of the message from this publisher.
topic hash	The identifier for the topic name.
sow expiration time	The expiration set on this individual message when it is used in a SOW topic or queue.
iso8601 timestamp	The time the message was processed, represented as an ISO-8601 timestamp.
timestamp	The raw timestamp stored in the AMPS transaction log. This is a microsecond-precision timestamp.
previous byte count	The size, in bytes, of the previous record in the transaction log.
topic byte count	The length, in bytes, of the topic name.
topic	The name of the topic this command applies to.
data	The data for this entry in the transaction log. For published messages, this will be the message as published. For other commands, this could be data used internally by AMPS.
correlation id byte count	The number of bytes in the correlation ID for the command.
correlation id	The correlation ID for the command, if one was set on the command.
auth id byte count	The length of the authenticated ID that submitted the command.
auth id	The authenticated ID that submitted the command.
rep path byte count	The length of the replication path. This path records the route that this message took to reach this instance of AMPS.
rep path	The replication path. This path records the route that the message took to reach this instance of AMPS.

As seen in the listing, the final section contains general usage information about the data contained in the journal file.

Field	Description
Total Entries	Total number of journal entries entered into the journal file.
Total Bytes	The number of reserved bytes consumed by the journal file.
Remaining Bytes	The number of unused bytes available out of the total reserved file size.

Timestamp Formatting

The timestamp format used in amps_journal_dump is formatted by default using the system timezone for its location. To display the time in another timezone, the TZ environment variable can be configured to modify the output.

%> TZ='America/New_York' ./amps_journal_dump A.000000000.journal

%> TZ='Asia/Tokyo' ./amps_journal_dump A.000000000.journal

%> TZ='Europe/London' ./amps_journal_dump A.000000000.journal