Historical SOW Topic Queries

Topics in the State of the World can also be configured to include historical snapshots of messages, which allows subscribers to retrieve the contents of the topic at a particular point in time.

As with simple queries, a client can issue a query by sending AMPS a sow command and specifying an AMPS topic. For a historical query, the client also adds a timestamp that includes the point in time for the query in the Bookmark header of the command. A filter can be used to further refine the query results based on the message content.

When to Use a Historical SOW Query

Use a historical SOW query when it is important to get a snapshot of the state of messages in a topic as they existed at a specific point in time (that is, if it is important for an application to be able to query the state of the world at a point in time).

If an application needs to replay the exact sequence of messages delivered to a topic, but does not need to be able to query the values that were current at a specific point in time, record the topic in the transaction log and replay from the transaction log.

If an application needs to both retrieve a snapshot of the values that were current at a specific point in time and replay the exact sequence of messages from that point forward, use a historical SOW query and record the topic in the transaction log.

Configuring the Topic: Window and Granularity

By default, AMPS does not maintain history for a topic in the State of the World. To enable history (and historical query) for the topic, add the History element to the Topic configuration. This element configures how much information AMPS stores for enabling historical queries.

There are two options that control how AMPS stores data for historical queries:

1. The Window option sets the amount of time that AMPS will retain historical versions of messages. AMPS will remove the historical state of the message from the SOW topic once that historical state is older than the specified window. (If the message has been deleted, and the delete command is older than the specified window, AMPS may remove the message from the SOW topic entirely). AMPS always retains the most current state of a message, even if that state was published earlier than the specified Window.

   In other words, a given version of a message is eligible for removal after it has no longer been the most current update to that message for longer than the specified Window.

2. The Granularity option sets the interval at which AMPS retains a historical copy of a message in the SOW. For example, if the Granularity is set to 10m, AMPS stores a historical copy of the message no more frequently than every 10 minutes, regardless of how many times the message is updated in that 10 minute interval. AMPS stores the copies when a new message arrives to update the SOW. This means that AMPS always returns a valid SOW state that reflects a published message, but -- as with a conflated topic -- the SOW may not reflect all of the states that a message passes through. This also means that AMPS uses SOW space efficiently. If no updates have arrived for a message, since the last time a historical message was saved, AMPS has no need to save another copy of the message.

When a message is deleted from a topic that maintains history, AMPS saves the fact that the message has been deleted, and queries as of that point in time will not return the message. However, previously saved states of the message within the Window are still present and can still be queried.

Likewise, if an application queries at a point in time earlier than the Window, AMPS will return an empty result set (even if messages had actually been present in the topic at that point), since the SOW state is only retained for the period in the Window.

The Granularity for a topic is always specified as a duration. If your application requires that a query be able to return the exact state of the SOW exactly as AMPS would have represented it at that time (with no tolerance for the granularity), you can specify that AMPS keep every message during the Window by specifying a Granularity of 0s. Notice that this is not required to replay every message after a point-in-time query, since replay is delivered from the transaction log rather than the stored State of the World.

When a historical SOW and Subscribe query is entered, and the topic is covered by a transaction log, AMPS returns the state of the SOW adjusted to the next oldest granularity, then replays messages from that point. In other words, AMPS returns the same results as a historical SOW query, then replays the full sequence of messages from that point forward.

The transaction log and the SOW topic are maintained separately and have separate views of history. When a version of the message is removed from the SOW topic (because it is older than the specified Window), the message remains in the transaction log, but will not be returned by a SOW query.

The length of time that messages remain in the topic is specified by the Window. A SOW topic that retains history does not support sow expiration.

If it is necessary to delete messages after they have been active for a certain period of time in a topic that maintains history, use an explicit delete from an application or use a scheduled action, as described in Creating a Maintenance Schedule for a Topic.

Message Sequence Flow

The message sequence flow is the same as a simple SOW query flow. Once AMPS has transmitted the messages that were in the SOW as of the timestamp of the query, the query ends. Notice that the query will include messages that have been subsequently deleted from the SOW, but which were the current state of the message as of that timestamp.

Pagination with Historical SOW Queries

Topics that maintain History in the SOW support paginated queries from a point in time. When the topic is also covered by the transaction log, the sow_and_subscribe command also supports paginated subscriptions from a point in time. See Paginated SOW and Subscribe for details.