Programmatically Deleting Records from the Topic State

AMPS allows applications to explicitly remove records from a SOW topic using the sow_delete command.

When removing records from a SOW, there are three different ways to indicate which message, or messages, will be deleted:

1. Using a content filter. AMPS will delete all messages in the SOW that match the content filter. To delete every message in the SOW, use the special filter 1=1 to indicate that the filter is true for every message, regardless of the contents of the message. (In essence, AMPS runs a query to locate the records to be deleted, and then deletes the matching records.)

2. Using the SOW key assigned to the message. AMPS accepts a list of SOW keys, and will remove the messages indicated by those SOW keys.

3. Using message data. The application provides message data with the sow_delete command. AMPS parses the message data to determine the SOW key for the record that would be updated if the command were a publish, and deletes that record (if one exists). Notice that if the topic is configured so that publishers must provide the SOW key, the key cannot be derived from the data, which means that using message data to delete messages may not produce the expected results.

When a record is removed from the SOW, AMPS sends an out-of-focus (OOF) message to any subscriptions that have requested OOF notifications. AMPS also updates any views that use the SOW topic, and the record will be removed from conflated topics at the next conflation interval.

When the SOW is configured with the History option to enable historical queries, the sow_delete command removes the message from the current set of messages in the SOW. The command does not remove previously saved versions of the message: the historical state of the SOW is unaffected by the sow_delete.

The most efficient way to delete a specific message or specific set of messages is to use the SOW key that AMPS assigns, when that key is available. You can provide these keys in the SowKeys header (a delete by keys), or by providing a filter expression that will be evaluated as a query on the primary key or a hash index. See the Indexing SOW Topics section for details on how AMPS determines if a hash index or primary key can be used for a filter.

When the SOW delete provides an example message to be deleted, AMPS parses that message to determine the SOW key and then uses that to key to delete the message, which is also relatively efficient.

Deleting a message from the SOW means that AMPS can reuse the space that the message consumed, but AMPS does not reduce the size of the storage for the topic when a message is removed. Typically, SOW topics in production reach a steady state based on the number of messages that are typically present at any given time: it is most efficient to simply make the space available for new messages.

To reduce the size of a file used to persist a topic in the SOW after messages are removed, use the Compact SOW Topic action. Running this operation will typically reduce throughput to the topic being compacted during the process of compacting the topic, so this should only be done during a maintenance window or when reducing (or pausing) throughput to the topic would have less impact on the application than leaving the SOW file at its current size.

Removing a message from a Topic in the State of the World removes the message from that Topic and notifies any View or ConflatedTopic that depends on this topic that the message has been removed (see Aggregation and Analytics for details on creating a View, see Conflated Topics for details on creating a conflated topic). Removing a message from a Topic adds the delete command to the transaction log, but does not remove messages stored in the transaction log (see Record and Replay Messages).

If the Topic contains History, the sow_delete affects the current value of the Topic but does not remove previous state. AMPS will remove records that have not been current for longer than the retention Window, as described in the Historical SOW Topic Queries section.