State of the World Message Enrichment
Topics recorded to the State of the World (SOW) can provide inline message enrichment for messages published to the topic. This capability is especially useful for applications that do consistent, simple transformations on incoming data. For example, you can use this capability to automatically add a calculated price to an incoming order, to map abbreviated data such as status codes to easier-to-understand values, or even to compute the value of a field used for a SOW key.
AMPS provides two distinct stages of message enrichment: preprocessing and enrichment. The preprocessing stage occurs before AMPS calculates the SOW key for the message. Fields that are added or updated in the preprocessing stage can be used as the SOW key for the message. Given that this stage occurs before the SOW key is generated, this stage does not have access to the previous state of the message in the SOW. The enrichment stage occurs after AMPS calculates the SOW key. Enrichment performed at this stage has access to the previous state of the SOW.
If entitlement for the instance uses content filters for publish entitlements, these filters are applied to the incoming message before either enrichment stage runs. For more details on the steps involved in enrichment, see the sequence of operations in SOW Update and Enrichment Processing.
Message enrichment only affects the message data, not the metadata on the message. In other words, while enrichment can change any field in the data, you cannot change metadata properties such as the topic the message was published to, the acknowledgments requested on the message, or the authenticated username for the publish command.
Message enrichment rewrites the message before the messages are stored in AMPS or delivered to publishers. AMPS also provides the ability to aggregate or analyze messages while preserving the original state of the message, as described in the chapter on Aggregation and Analysis. If a subscriber only needs a subset of data in a message, AMPS provides the ability for that subscriber to provide a select list to retrieve only the needed data.
Preprocessing Messages
The preprocessing stage of AMPS enrichment allows you to alter a message before the SOW key is calculated. This gives you the ability to easily add or transform fields that are used in the SOW key. Use this stage to enrich messages when the enriched field should be used as part of the SOW key. To specify preprocessing for a topic, you add a Preprocessing directive to the Topic configuration for the SOW topic.
Use Preprocessing when you need to change the value of a field that is part of the Key for the topic. Otherwise, use Enrichment.
Preprocessing field directives operate on a single message and construct fields based on that message. The results of the preprocessing expression are merged into the incoming message. Any field in the source message that is not changed or removed during preprocessing is left unchanged, so it is not necessary to include all fields in the message in the Preprocessing block.
Since preprocessing fields apply to a specific message, preprocessing fields cannot specify the topic or message type in an XPath identifier.
By default, AMPS serializes fields with a NULL value in the preprocessing result. Preprocessing fields can include a directive that specifies that if a field contains a NULL value, it should be removed from the set of fields rather than serialized. The directive HINT OPTIONAL applied to the XPath identifier specifies that if the result of the source expression is NULL, AMPS does not provide the value for the message type to serialize. For example, use the following directive to remove a /source field if the value provided is not in a specific list of values:
For more information on constructing preprocessing fields, see Constructing Preprocessing Fields.
Enriching Messages
AMPS enrichment operates on a message after the SOW key is computed, but before an incoming delta publish is merged to an existing message, or the incoming message is written to the transaction log, stored to the SOW, used to update views, or delivered to subscribers. Use this enrichment stage when the enrichment process depends on the previous values of the message, or when the updated fields will not be used in the SOW key. To specify enrichment for a topic, you add an Enrichment directive to the configuration for the SOW topic.
Enrichment field directives operate on a single message and construct fields based on that message. Enrichment expressions operate on the current message and change the current message. The results of the enrichment directives are merged into the incoming message. Any field in the source message that is not changed or removed during enrichment is left unchanged, so it is not necessary to include all fields in the message in the Enrichment directive.
Since enrichment fields apply to a specific message, enrichment fields cannot specify the topic or message type in an XPath identifier.
Within an enrichment expression, AMPS provides two special modifiers for XPath identifiers that specify whether an XPath identifier refers to the current incoming message or the previous state of the message. These modifiers apply only to the source expression, and cannot be used in special modifiers. They are:
OF CURRENT
Specify that the XPath identifier refers to the incoming message.
OF PREVIOUS
Specify that the XPath identifier refers to the previous state of the message in the SOW.
If there is no record in the SOW for this message, all identifiers that specify OF PREVIOUS return NULL.
By default, AMPS serializes fields with a NULL value during enrichment. Enrichment fields can include a directive that specifies that if a field contains a NULL value, it should be removed from the set of fields rather than serialized. The directive HINT OPTIONAL applied to the XPath identifier specifies that if the result of the source expression is NULL, AMPS does not include the value in the set of XPath identifiers for the message type to serialize. For example, use the following directive to use remove a /source field if the value provided is not in a specific list of values:
For more information on constructing enrichment fields, see Constructing Enrichment Fields.
SOW Update and Enrichment Processing
The following diagram presents a simplified, high-level view of the update process for an individual message. For the purposes of this diagram, views and conflated topics can be considered listeners on the SOW topic, while applications that connect to AMPS and the on-publish and on-deliver actions can be considered subscribers.
It's important to keep in mind the following aspects of the SOW update sequence:
If the publish is disallowed due to topic-based entitlements or the publish filter specified for entitlements, there is no change to the state of the SOW. The entitlement filter (if one exists), is applied to the incoming message before preprocessing, enrichment, or delta merge occurs.
AMPS records the enriched message in the transaction log and SOW file. When AMPS is configured for enrichment or your application performs a delta publish, the transaction log and SOW do not preserve a record of the original message received by AMPS. Instead, they record the enriched and merged message.
Content filtering for subscriptions, views, and so forth is done on the final enriched and merged message, not on the original message as published.
Last updated
