Dump journal File

The AMPS journal dump utility is used in examining the contents of an AMPS journal file for debugging and program tuning. The amps_journal_dump utility is most commonly used as a tool to debug the forensic lifespan of messages that have previously been published to AMPS. The amps_journal_dump tool is used to show that messages exist in a journaled topic, and to show the order the message was received in, and the timestamp associated with the message.

Command Line Options

The amps_journal_dump program has the following options available. These can also be printed to the screen by typing amps_journal_dump -help.

Option

Description

filename (required)

Filename of the AMPS journal file.

-h, --help

Show the program help message and quit.

-l LIMIT

Limit range of output to entries N:M where N is the first entry and M is the last entry.

Passing in a single value, M, will return the first M results.

--localtime

Display ISO 8601 timestamp in local time.

--extents

Add local and replication extents information at the end of the journal dump.

--no-data

Do not display data values in the journal dump output.

--include-noops

Include noops in the journal dump.

Looking at the Output

In this section will examine some sample output from running amps_journal_dump. We will then go over what each of the entries emitted by the program means.

File Name       : AMPS-Sample.0000000000.journal
File Size       : 10485760
Version         : amps::txlog/v8
Extents         : [1623783047000000005:1623783047000002563]
First Timestamp : 20210615T185233.126619Z
Last Timestamp  : 20210615T185234.032520Z
______________________________________________________________
entry                         :                              0
crc32                         :                     2427361060
type                          :                        publish
flags                         :                           none
file offset                   :                           4096
tx byte count                 :                            256
msg byte count                :                             21
msg type                      :                           json
local txid                    :            1711058949000000003
previous local txid           :                              0
source txid                   :           14124379235191657289
source name hash              :                              0
client name hash              :           15749960451245477525
client seq                    :            1711058949000000003
topic hash                    :           10912813577604266571
sow expiration time           :                              0
iso8601 timestamp             :        20240321T220952.535075Z
timestamp                     :             212577862192535075
previous byte count           :                              0
topic byte count              :                             13
topic                         : [sample-record]
data                          : [{"message":"example"}]
correlation id byte count     :                              0
correlation id                : []
auth id byte count            :                              0
auth id                       : []
rep path byte count           :                              0
rep path                      : []
_____________________________________________________

...

Total Entries    : 2559
Total Bytes      : 10485760
Remaining Bytes  : 0

As is apparent in the listing, the output from amps_journal_dump is split into three sections, a header, a listing of the contents of the journal file and a footer.

The header contains general information about the journal file as it is represented in the filesystem, and state data about the journal file.

Field

Description

File Name

The name of the file as it appears on the local filesystem.

File Size

Number of total bytes allocated by the journal file.

Version

This is the version of the formatting used to write the data in the journal file.

Extents

A pair of numbers where the first is the number of extents used by the journal file, and the second is the number of blocks allocated for the journal file.

The second section of the amps_journal_dump lists each of the entries contained in the journal file, along with all of the meta-data used to track and describe the entry. For the sake of simplicity, the example only shows a single journal entry. In an actual installation, amps_journal_dump will produce a record for every message in the journal.

The exact options present depend on the version of the journal file.

Field

Description

entry

A monotonically increasing value representing the order in which the record was inserted into the transaction log file.

crc32

The cyclic redundancy check used for error checking the transaction log entry.

type

The AMPS command used in the original message or the type of entry in the journal file.

file offset

Offset for this entry within the journal file.

tx byte count

Size of this entry within the journal file.

msg byte count

Number of bytes in the message payload.

msg type

Message type of the message.

local txid

The monotonically increasing identifier used across all records local transaction log journal files.

previous local txid

The monotonically increasing identifier used across all records local transaction log journal files.

source txid

The monotonically increasing identifier used across all records local transaction log journal files. If a publish from a client does not include a sequence number, and the publish was made directly to this instance, this field can be used to hold the original client name hash for a publish or delete. In that case, the client name hash used for message identification purposes is generated by AMPS (to avoid message identity conflicts).

source name hash

For replicated messages, the hash of the instance from which the message was received.

client name hash

The hash of the publisher client name. The combination of the client name hash and the client seq are used as the identity of the message. If a publish or delete does not include a sequence number, and the publish was made directly to this instance, this field may be a hash generated by AMPS while the original client name hash is stored in the source txid field.

client seq

The sequence number of the message from this publisher.

topic hash

The identifier for the topic name.

sow expiration time

The expiration set on this individual message when it is used in a SOW topic or queue.

iso8601 timestamp

The time the message was processed, represented as an ISO-8601 timestamp.

timestamp

The raw timestamp stored in the AMPS transaction log. This is a microsecond-precision timestamp.

previous byte count

The size, in bytes, of the previous record in the transaction log.

topic byte count

The length, in bytes, of the topic name.

topic

The name of the topic this command applies to.

data

The data for this entry in the transaction log.

For published messages, this will be the message as published. For other commands, this could be data used internally by AMPS.

correlation id byte count

The number of bytes in the correlation ID for the command.

correlation id

The correlation ID for the command, if one was set on the command.

auth id byte count

The length of the authenticated ID that submitted the command.

auth id

The authenticated ID that submitted the command.

rep path byte count

The length of the replication path. This path records the route that this message took to reach this instance of AMPS.

rep path

The replication path. This path records the route that the message took to reach this instance of AMPS.

As seen in the listing, the final section contains general usage information about the data contained in the journal file.

Field

Description

Total Entries

Total number of journal entries entered into the journal file.

Total Bytes

The number of reserved bytes consumed by the journal file.

Remaining Bytes

The number of unused bytes available out of the total reserved file size.

Timestamp Formatting

The timestamp format used in amps_journal_dump is formatted by default using the system timezone for its location. To display the time in another timezone, the TZ environment variable can be configured to modify the output.

%> TZ='America/New_York' ./amps_journal_dump A.000000000.journal
%> TZ='Asia/Tokyo' ./amps_journal_dump A.000000000.journal
%> TZ='Europe/London' ./amps_journal_dump A.000000000.journal

Last updated

Copyright 2013-2024 60East Technologies, Inc.