Storage Performance Testing

amps_bio_perf_test

AMPS contains a utility, amps_bio_perf_test, that measures the raw performance of storage devices for the type of sequential writes (and, optionally, reads) that are used by the AMPS transaction log.

The utility can be used to get a sense of the maximum throughput that a given device offers, and to compare one device to another.

Options and Parameters

Option	Description
-b,--batch_size	The number of messages to be batched (1 to 256).
-m,--message_size	The message size in bytes (256 to 128K, must be a multiple of 256).
-f,--file_size	Size of the data file to write to, in MB (max 8192).
-r,--read_mix	Read mix specified as a percentage (0 - 99).
-f,--file	Name of the data file that is created (this should be on the device to be tested). The file will be allocated before the test begins running, and will contain nonsense data.

60East recommends running the amps_bio_perf_test tool on a system that is otherwise quiet. The --file specified is the file that the tool creates for writing data, so that file must be on the device to be measured.

The amps_bio_perf_test uses the O_DIRECT flag for writes (as does the AMPS transaction log writer), bypassing caching at the operating system level.

The amps_bio_perf_test tool attempts to fully use the I/O capacity of the target device. Do not run this tool on a system that is being used for another purpose, since the tool may impact the performance of any other process using the device being tested.

Output

The amps_bio_perf_test tool produces the following metrics:

Metric	Explanation
elapsed time	Time during which the tool was actively writing/reading data.
writer thread count	Number of threads writing data (always 1).
batch size	Number of messages (submissions) in a batch.
submit size	Size of each message (submission), in bytes.
write size	Total size of each batch written, in bytes.
submit count	Number of messages (submissions) during the test.
read count	Number of read operations during the test.
write count	Number of write operations during the test.
submit rate	Average number of messages (submissions) per second while the tool was actively writing/reading data.
write rate	Average number of writes (batches written) per second while the tool was actively writing/reading data.
actual read mix	Proportion of reads to writes during the test (this may differ from the target due to thread scheduling, etc.).
mean write latency	Mean latency for a write, in microseconds.

The tool also writes csv files with the raw data used to calculate these metrics, which can be helpful for more in-depth analysis (for example, calculating the 99th percentile latency rather than the mean). One file contains the full set of latency samples collected, the other file contains the full set of throughput samples collected. If the files do not exist when the tool is run, it will create them. Otherwise, the tool will append to the files so that results can be tracked over multiple runs.

Tips on Evaluating Overall Performance

The amps_bio_perf_test tool tries to simulate the AMPS transaction log workload under ideal conditions for maximum throughput. The tool preallocates the file that it writes to, and also pregenerates all of the data it will write and the data structures that it will use to record results. During the active part of the test, the tool itself does not allocate memory, but instead measures the time taken for writes and records that data into the preallocated records for the results.

The submit rate is the absolute maximum number of messages that could be written per second by that device. To achieve these rates, the payload and metadata size for every message would have to exactly equal the submit size, and the ingress rate from the network would need to exactly equal the rate at which the device can write messages (that is, no gaps in message flow or pushback from the device).

The mean write latency (measured in microseconds) is the average amount of time that each write takes to complete. This affects the overall latency of the system, and is strongly correlated with performance (as reflected in the write rate).

For example, consider a result that shows a "submit rate" of 2500000 for 512 byte messages. This means that this device could (in principle, under absolutely ideal conditions), support a maximum write throughput of:

512 bytes	2,500,000 per second
1024 bytes (1KB)	1,250,000 per second
10KB	125,000 per second
100KB	12,500 per second

These are, of course, approximations of the expected maximum throughput. In practice, message sizes rarely fit perfectly into a batch, application activity sees bursts and lulls, the number of threads reading from the transaction log varies based on activity (AMPS tries to consolidate replays from the transaction log into the minimum number of threads to save CPU and I/O bandwidth), and the transaction log device may be used for other traffic (for example, SOW topic storage, statistics, or error and event logging) that may share I/O bandwidth. These factors (and others) can affect the actual performance in deployment.

Usage

60East typically runs the amps_bio_perf_test tool with a command line similar to the following:

$ amps_bio_perf_test -b 128 -m 512 -s 8192 -r 0 -f /mnt/fastdrive/ampsdir/bio.data

This provides a good baseline for the maximum write performance of the system.