Dump journal File
The AMPS journal dump utility is used in examining the contents of an AMPS journal file for debugging and program tuning. The amps_journal_dump utility is most commonly used as a tool to debug the forensic lifespan of messages that have previously been published to AMPS. The amps_journal_dump tool is used to show that messages exist in a journaled topic, and to show the order the message was received in, and the timestamp associated with the message.
Command Line Options
The amps_journal_dump program has the following options available. These can also be printed to the screen by typing amps_journal_dump -help.
| Option | Description |
|---|---|
| Filename of the AMPS journal file. |
-h, --help | Show the program help message and quit. |
-l LIMIT | Limit range of output to entries N:M where N is the first entry and M is the last entry. Passing in a single value, M, will return the first M results. |
--localtime | Display ISO 8601 timestamp in local time. |
--extents | Add local and replication extents information at the end of the journal dump. |
--no-data | Do not display data values in the journal dump output. |
--include-noops | Include noops in the journal dump. |
Looking at the Output
In this section will examine some sample output from running amps_journal_dump. We will then go over what each of the entries emitted by the program means.
File Name : AMPS-Sample.0000000000.journal
File Size : 10485760
Version : amps::txlog/v8
Extents : [1623783047000000005:1623783047000002563]
First Timestamp : 20210615T185233.126619Z
Last Timestamp : 20210615T185234.032520Z
______________________________________________________________
entry : 0
crc32 : 2427361060
type : publish
flags : none
file offset : 4096
tx byte count : 256
msg byte count : 21
msg type : json
local txid : 1711058949000000003
previous local txid : 0
source txid : 14124379235191657289
source name hash : 0
client name hash : 15749960451245477525
client seq : 1711058949000000003
topic hash : 10912813577604266571
sow expiration time : 0
iso8601 timestamp : 20240321T220952.535075Z
timestamp : 212577862192535075
previous byte count : 0
topic byte count : 13
topic : [sample-record]
data : [{"message":"example"}]
correlation id byte count : 0
correlation id : []
auth id byte count : 0
auth id : []
rep path byte count : 0
rep path : []
_____________________________________________________
...
Total Entries : 2559
Total Bytes : 10485760
Remaining Bytes : 0
As is apparent in the listing, the output from amps_journal_dump is split into three sections, a header, a listing of the contents of the journal file and a footer.
The header contains general information about the journal file as it is represented in the filesystem, and state data about the journal file.
| Field | Description |
|---|---|
| File Name | The name of the file as it appears on the local filesystem. |
| File Size | Number of total bytes allocated by the journal file. |
| Version | This is the version of the formatting used to write the data in the journal file. |
| Extents | A pair of numbers where the first is the number of extents used by the journal file, and the second is the number of blocks allocated for the journal file. |
The second section of the amps_journal_dump lists each of the entries contained in the journal file, along with all of the meta-data used to track and describe the entry. For the sake of simplicity, the example only shows a single journal entry. In an actual installation, amps_journal_dump will produce a record for every message in the journal.
The exact options present depend on the version of the journal file.
| Field | Description |
|---|---|
| entry | A monotonically increasing value representing the order in which the record was inserted into the transaction log file. |
| crc32 | The cyclic redundancy check used for error checking the transaction log entry. |
| type | The AMPS command used in the original message or the type of entry in the journal file. |
| file offset | Offset for this entry within the journal file. |
| tx byte count | Size of this entry within the journal file. |
| msg byte count | Number of bytes in the message payload. |
| msg type | Message type of the message. |
| local txid | The monotonically increasing identifier used across all records local transaction log journal files. |
| previous local txid | The monotonically increasing identifier used across all records local transaction log journal files. |
| source txid | The monotonically increasing identifier used across all records local transaction log journal files. |
| source name hash | For replicated messages, the hash of the instance from which the message was received. |
| client name hash | The hash of the publisher client name. The combination of the client name hash and the client seq are used as the identity of the message. |
| client seq | The sequence number of the message from this publisher. |
| topic hash | The identifier for the topic name. |
| sow expiration time | The expiration set on this individual message when it is used in a SOW topic or queue. |
| iso8601 timestamp | The time the message was processed, represented as an ISO-8601 timestamp. |
| timestamp | The raw timestamp stored in the AMPS transaction log. This is a microsecond-precision timestamp. |
| previous byte count | The size, in bytes, of the previous record in the transaction log. |
| topic byte count | The length, in bytes, of the topic name. |
| topic | The name of the topic this command applies to. |
| data | The data for this entry in the transaction log. For published messages, this will be the message as published. For other commands, this could be data used internally by AMPS. |
| correlation id byte count | The number of bytes in the correlation ID for the command. |
| correlation id | The correlation ID for the command, if one was set on the command. |
| auth id byte count | The length of the authenticated ID that submitted the command. |
| auth id | The authenticated ID that submitted the command. |
| rep path byte count | The length of the replication path. This path records the route that this message took to reach this instance of AMPS. |
| rep path | The replication path. This path records the route that the message took to reach this instance of AMPS. |
As seen in the listing, the final section contains general usage information about the data contained in the journal file.
| Field | Description |
|---|---|
| Total Entries | Total number of journal entries entered into the journal file. |
| Total Bytes | The number of reserved bytes consumed by the journal file. |
| Remaining Bytes | The number of unused bytes available out of the total reserved file size. |
Timestamp Formatting
The timestamp format used in amps_journal_dump is formatted by default using the system timezone for its location. To display the time in another timezone, the TZ environment variable can be configured to modify the output.
%> TZ='America/New_York' ./amps_journal_dump A.000000000.journal
%> TZ='Asia/Tokyo' ./amps_journal_dump A.000000000.journal
%> TZ='Europe/London' ./amps_journal_dump A.000000000.journal