Chaining Key Generator

The AMPS distribution includes a module that can generate a SOW key for a set of chained messages.

Message chains are most frequently used in FIX order processing systems to track a set of updates to an original order from a set of systems that use unique local identifiers for the order. As messages arrive, AMPS must update the record for the original order, regardless of whether the identifier on the current message is the original order, or is an order chained to the original order.

A message chain allows an application to treat any update to an identifier in the chain as an update to the original message in the chain. The libamps_id_chaining_key_generator module supports this by generating the same SOW key for any message in the chain. To use this module, messages must have a field that identifies the current message and a field that identifies the previous message in the message chain, if one exists.

Chained Message Sample Case

Consider a message processing scheme that uses two fields to identify related messages. Each message contains a DocumentNumber field that indicates the current document. If the message updates or extends an existing document, the message contains a ParentDocument that, when present, refers to the DocumentNumber of the document that the message updates or extends.

With the default SOW key generator, each of the following messages would be a distinct message in the SOW topic:

delta_publish: {"DocumentNumber":1, "Status":"Started"}
delta_publish: {"DocumentNumber":2, "ParentDocument":1, "Order":"Antivenom"}
delta_publish: {"DocumentNumber":3, "ParentDocument":2, "Order":"Sandwich"}
delta_publish: {"DocumentNumber":4, "ParentDocument":1, "Status":"Pending"}

With the default SOW key generator, at the end of the publishing process, the SOW contains four distinct records:

{"DocumentNumber":1, "Status":"Started"}
{"DocumentNumber":2, "ParentDocument":1, "Order":"Antivenom"}
{"DocumentNumber":3, "ParentDocument":2, "Order":"Sandwich"}
{"DocumentNumber":4, "ParentDocument":1, "Status":"Pending"}

However, with the chaining key generator, AMPS is able to combine these messages into a single chain and produces the following single record:

{"DocumentNumber":4, "ParentDocument":1 , "Order":"Sandwich", "Status":"Pending"}

The sequence of events for producing this message is as follows:

• When the first message arrives with a /DocumentNumber of 1, the module begins a new chain (since there is no /ParentDocument present).

• When the second message arrives, the module knows that it is an update to the same message since the message contains a /ParentDocument value. In this case, because the value is 1, the update is to the first message received. The module also adds a /DocumentNumber of 2 to the chain, so that subsequent messages that refer to a /ParentDocument of 2 are a part of the chain and update the same message.

• The same process occurs for the third message: the module looks up the message that should be updated when the /ParentDocument is 2, and traces the chain back to the original underlying message. The module adds a /DocumentNumber of 3 to the chain, so that updates with a /ParentDocument of 3 will update the same message.

• When the last message arrives, the module knows that a /ParentDocument of 1 is still an update to the same message, since this is the original value. The module adds the value 4 to the chain.

In each case, rather than simply using the fields in the message directly, the module creates a chain of linked identifiers: each identifier in the chain produces the same SOW key as the first identifier in the chain, so each update in the chain updates the same message.

It is an error for a publisher to publish a message that resolves to two different message chains. If the module receives such a message, the module will not generate a SOW key, and the message is not processed by AMPS.

Configuring the Chaining Key Generator

To load the module in AMPS, add the following configuration item to the Modules block of the AMPS configuration file:

<AMPSConfig>
    ...

    <Modules>
         ...

        <Module>
            <Name>key-chaining</Name>
            <Library>libamps_id_chaining_key_generator.so</Library>
        </Module>

    </Modules>

</AMPSConfig>

You then use the module as the KeyGenerator for each topic in the SOW that will use chaining key generation.

The module accepts the following options:

Parameter	Description
Key	A field to use in chaining. AMPS supports any number of Key fields. The first Key field specified in the configuration is the primary field to use in chaining. When the primary field is present on a message, and the value of the field is not in an existing chain of values, the module creates a new chain. When the message contains the primary field and there is no previous entry for the value of that field, this message is the head of the chain and is used to generate the SOW key. Key parameters specified after the first Key are secondary fields. When a secondary field is present on the message, the module generates a SOW key for this message as though the message contained a primary field with this value. In addition, the module stores the value of the primary field in the current, if any, message as equivalent to this value, enabling subsequent messages to be chained to this message. There is no default for this parameter. The parameter requires an AMPS field identifier, such as /11 or /Order/ClOrdID. The module requires a primary field and at least one secondary field to be defined.
FileName	Sets the name of the file that the module uses to store chaining data. This module persists existing chains between restarts of the AMPS server. If a file with the given name exists when AMPS starts, the module reads chaining data from the file. Otherwise, the module creates a new file.
Primary	A synonym for Key that explicitly specifies that this field is the primary field. When this configuration element is present, AMPS uses the field specified in this element as the primary field and considers any field specified in a Key element to be a secondary field.
Secondary	A synonym for Key that explicitly specifies that this field is a secondary field.
Validation	Specifies whether the module validates that incoming messages are properly chained. When set to true or 1, the module records extra data to attempt to detect errors in the sequencing of the chain. The module will consider it an error when it detects that two or more distinct chains share identifiers and would have been combined into a single chain had messages arrived in a different order. In some systems, this indicates an error in message processing. Default: This option defaults to false.

Example

The example configuration file below shows one way to use the chaining key generator module.

<Modules>
   ...

    <Module>
        <Name>key-chaining</Name>
        <Library>libamps_id_chaining_key_generator.so</Library>
    </Module>
</Modules>

<SOW>
    ...

    <Topic>
        <Name>Orders</Name>
        <MessageType>json</MessageType>
        <KeyGenerator>
            <Module>key-chaining</Module>
            <Options>
                <Primary>/DocumentNumber</Primary>
                <Key>/ParentDocument</Key>
                <Key>/RelatedDocument</Key>
                <FileName>./sow/Orders.chain</FileName>
            </Options>
        </KeyGenerator>
        <FileName>./sow/%n.sow</FileName>
    </Topic>

    <Topic>
        <Name>ExternalOrders</Name>
        <MessageType>fix</MessageType>
        <KeyGenerator>
            <Module>key-chaining</Module>
            <Options>
                <!-- /11 is the primary field -->
                <Key>/11</Key>
                <Key>/41</Key>
                <FileName>./sow/ExternalOrders.chain</FileName>
            </Options>
        </KeyGenerator>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
</SOW>

Notice that once the module is loaded, it can be used for any message type, and can accept different configuration values for each topic in the SOW that uses the generator.