Understanding SOW Keys

This section describes AMPS SOW keys in detail, including information on how AMPS generates SOW keys and considerations for applications that generate SOW keys. An individual SOW topic may use either AMPS-generated SOW keys or user-generated SOW keys. Every message in the SOW must use the same type of key generation.

Regardless of how the SOW key is generated, AMPS creates an opaque value from the SOW key and uses this value for efficient lookup internally. For SOW keys that AMPS generates, this opaque value is returned in the message header for SOW messages and is used in commands that reference SOW keys. When the SOW key is provided with a message, AMPS returns the original value in the SOW key header, and the original value is used in commands that reference SOW keys.

For topics that have a SOW key (including views and conflated topics), commands that directly use the SOW for a topic (for example, sow, sow_and_subscribe, sow_delete) can provide a SOW key, or a set of SOW keys with the command. When a set of SOW keys is provided with one of these commands, the command will only operate on messages that have a SOW key in the provided set.

AMPS-Generated SOW Keys

AMPS-generated SOW keys are often the easiest and most reliable way to define the SOW key for a message. The advantages of this approach are that AMPS handles all of the mechanics of generating the key, the key will always match the data in the message, and there is no need for a publisher to be concerned with how AMPS assigns the key. The publisher simply publishes messages, and AMPS handles all of the details.

AMPS generates SOW keys based on the message content when you define one or more Key fields in the SOW configuration. For example, if your SOW tracks unique orders that are identified by an orderId field in the message, you could provide the following Key element in your SOW configuration:

<Key>/orderId</Key>

This configuration item tells AMPS to use that field of the message to generate SOW keys. AMPS supports composite SOW keys when multiple Key elements are provided. For example, the following configuration specifies that every unique combination of /orderId and /customerId is a unique record in the SOW:

<Key>/orderId</Key>
<Key>/customerId</Key>

When AMPS generates a key, it creates the key based on the key domain (which is the name of the topic by default) and the values of the fields specified as SOW keys. AMPS concatenates these values together with a unique separator and then calculates a checksum over the value. This ensures that different values create different keys, and ensures that records in different topics have different keys.

In some cases, you may need AMPS to calculate consistent SOW key values for identical messages even when the messages are published to different topics. The SOW topic definition allows you to set an explicit key domain in the configuration, which AMPS will use instead of the topic name when generating SOW keys. For example, if your application uses the orderId field of a message as a SOW key in both a ShippingStatus topic and an OpenOrders topic, having AMPS generate a consistent key for the same orderId value may make it easier to correlate messages from those topics in your application. By setting the same KeyDomain value in the Topic configuration for those SOW topics, you can ensure that AMPS generates consistent SOW keys for the same order ID across topics.

An application should treat SOW keys generated with the AMPS default SOW key generator as opaque tokens. The value of a generated SOW key is guaranteed to be consistent for the same fields, values, and key domain. However, an application should not make assumptions as to the specific value that the AMPS default key generator will produce from a given set of values. If an application requires a specific value for the SOW key, the application should generate a SOW key, as described in the following section.

Using Enrichment with SOW Keys

The preprocessor phase of AMPS enrichment occurs before AMPS generates SOW keys for a message. You can use this phase of enrichment to construct fields that are then used to generate the SOW key for a message.

Customizing AMPS-Generated SOW Keys

AMPS allows you to customize how the server generates SOW keys for a topic. To customize SOW key generation, you implement a SOW key generator module and specify that the module should be used to generate keys for that SOW topic.

To use a custom SOW key generator, you first load the module in the Modules section of the configuration file, then specify the module as the KeyGenerator for the SOW topic.

<AMPSConfig>
    ...

    <!-- load the module -->
    <Modules>
        <Module>
            <Name>key-generator</Name>
            <Library>libmy_key_generator.so</Library>
        </Module>
    </Modules>

    <!-- use the module to generate keys -->
    <SOW>
        <Topic>
            <Name>custom-keyed-sow</Name>
            <FileName>./sow/%n.sow</FileName>
            <KeyGenerator>
                <Module>key-generator</Module>
                <Options>
                    <OptionOne>module-specific-option</OptionOne>
                    <OptionTwo>another-specific-option</OptionTwo>
                </Options>
            </KeyGenerator>
        </Topic>
    </SOW>

    ...
</AMPSConfig>

For information on implementing a custom SOW key generator, contact 60East support for the AMPS Server SDK.

User-Generated SOW Keys

AMPS allows applications to explicitly generate and assign SOW keys. In this case, the publisher calculates the SOW key for the message and includes that key in the message when it is published. AMPS does not interpret the data in the message to decide whether the message is unique: AMPS uses only the value of the SOW key.

When using a user-generated SOW key, applications should consider the following:

• All publishers should use a consistent method for generating SOW keys.

• SOW keys must contain only characters that are valid in Base64 encoding.

• The application must ensure that messages intended to be logically different do not receive the same SOW key.

User-generated SOW keys are particularly useful for the binary message type. For this message type, AMPS does not parse the message, so providing an explicit SOW key allows you to create a SOW that contains only binary messages.

To specify that AMPS will require publishers to this topic to submit the SOW key, the Topic configuration does not specify any Key fields and does not specify a KeyGenerator for the topic.