(logging-finding-messages)=

Using amps-grep to Find Information in Logs

The AMPS logs contain a record of events in the instance. Log messages are intended to be read as a part of that sequence of events. While an individual message is useful for showing that a particular event happened, the other messages in the log will show what sequence of events led up to that event and what the results of that event were.

At the same time, an active AMPS instance with dozens or hundreds of active clients generates a high volume of events, which can make it difficult to locate and correlate the events that are relevant to a specific problem. To help with this, the AMPS distribution contains the amps-grep tool to make it easy to find information in the logs.

This section presents some of the most useful techniques for locating information in the logs using amps-grep.

Finding Information for a Specific Client

When troubleshooting problems with a specific client (or connection), it's often helpful to be able to see the full sequence of events for that client. For example, if AMPS is returning an invalid options message to a client, finding the command that the client is sending to AMPS and the detailed messages that AMPS logs in response, can be very helpful in understanding the details of the error.

To find information about a specific client, use the following general pattern:

<pre>$ amps-grep <em>client_name</em> <em>log_files</em> &gt; out.txt </pre>

The amps-grep command extracts every event that contains the client name from the log files, and then the Linux shell writes those events to the out.txt file. Since the amps-grep tool is aware of the structure of multi-line AMPS event records, it captures the full event message, not just the individual lines that contain the name of the client.

For example, if the client has a client name of queue-processor-compute-host-39, and the logs are stored in files with the suffix .log, the command to extract the events with that client name would be along the lines of:

$ amps-grep 'queue-processor-compute-host-39' *.log > out.txt

This command will write every event message that contains a reference to the client queue-processor-compute-host-39 to the out.txt file.

Finding Information for a Specific Thread

Another situation where amps-grep can be useful is when it is important to get information about a specific thread in AMPS.

For example, in a situation where AMPS has produced a minidump, it is typically more useful to see the activity on the thread that produced the dump, than it is to see the activity of other threads near the time that the minidump was produced.

To track the sequence of events for a thread that produces a minidump, the first step is to find the AMPS identifier for the thread. This is the identifier used in the event log messages, which is different from the operating system assigned identifier for the thread. Then use that identifier to extract messages from the log.

For example, given a minidump message like the following:

2020-04-25T07:27:59.1355850-07:00 [6] critical: 01-0022 AMPS has
detected that it may not be running correctly and wrote a minidump to:
/tmp/1516e4d1-8bca-b14b-17853753-45dba87b.dmp

AMPS has assigned thread ID 6 to this thread. For convenience in searching, AMPS always puts the thread ID in brackets at the beginning of the message (since 6, by itself, is so common in the logs as to not be useful).

Extracting every message recorded by thread ID 6 in the log can be done with a command like the following:

$ amps-grep ' [6] ' *.log > out.txt

With this command line, amps-grep locates all messages for that thread ID, and the Linux shell writes the results to the out.txt file.

Tips for Using amps-grep

When using amps-grep, there are a few things to be aware of:

1. Unlike regular grep, by default amps-grep uses exact matching rather than regular expression matching. To use a regular expression, provide the -E option to amps-grep.

2. If you are searching multiple files and piping the output of one amps-grep command to another amps-grep command, use the -h flag to the first amps-grep to suppress the file name on matching lines. If you do not provide -h, the presence of the file name can interfere with amps-grep correctly identifying the start and end of each AMPS log message.

3. If you want to find more than one search term, you can use the -e flag to specify multiple search terms, for example:

   $ amps-grep -e 'error' -e 'warning' *.log

   Although it is possible to use a regular expression for a search like this, it is not necessary to do so.

4. The amps-grep utility provides a usage message with more details on the available options and usage.