Composite Messages

Sometimes, applications only need to filter on a small subset of the fields in a message. Sometimes applications need to send and receive messages that cannot be meaningfully parsed by AMPS, such as images or audio files. For these cases, AMPS provides a composite message type that lets you create a new message type by combining existing message types.

For example, you might create a message type that includes three parts: the metadata for an image as a json document, a small JPG thumbnail as a binary message part, and a full size PNG image as another binary message part.

Composite messages can also be useful when the message itself is large or resource-intensive to parse. In this case, you can create a message type that includes the information needed to filter messages in a JSON or NVFIX part, and include the full message in the unparsed payload of the composite message, as described below.

AMPS provides two different types of composite messages. Messages created using the composite-local module preserve information about the individual parts for filtering, aggregation, and projection. Messages created using the composite-global module treat the individual parts as elements of a single document.

Configuring Composite Message Types

To use a composite message type, you must first configure the type by declaring it in the MessageTypes section of the AMPS configuration file. The declaration contains the name of the new composite message type, specifies that the new type is composite, and lists the parts of the composite message type.

For example, the MessageType element below declares a new composite message type named images. The new type contains a json document at the beginning of the message, followed by two uninterpreted binary message parts. AMPS will combine the XPath identifiers for all message parts into a single set of identifiers. Notice that, because only one part of the message type is parsable, using composite-global simplifies the identifiers for the message.

<MessageTypes>
    ...

    <MessageType>
        <Name>images</Name>
        <Module>composite-global</Module>
        <MessageType>json</MessageType>
        <MessageType>binary</MessageType>
        <MessageType>binary</MessageType>
    </MessageType>

    ...

</MessageTypes>

The MessageType entries for the composite message can be any AMPS message type, including both the built-in types and any previously defined message type.

Once the new composite message type is created, you can use the new type in the configuration file.

Composite message types have the following restrictions:

• Delta subscribe and delta publish are not supported for message types that use composite-global.

• Views, joins, and aggregation cannot project message types that use composite-global. (However, composite message types that use composite-global can be an UnderlyingTopic or one of the topics in a Join.)

• Composite message types do not support features that automatically construct messages, such as subscriptions to the AMPS/.* topics and stats acks, regardless of the module the type uses.

Unparsed Payload Section

All composite message types, regardless of how they are defined, provide an unparsed payload section. The unparsed payload section does not need to be declared in the MessageType declaration. As the name suggests, AMPS does not parse or interpret this section, so the unparsed payload can contain any content of any type. In an AMPS client, the unparsed payload section is represented as bytes that appear at the end of the message, after the last defined part.

The unparsed payload is included to simplify the common technique where a message type contains a header that is used for filtering followed by an unparsed binary in cases where the only use of the message is subscribe / publish.

If your composite message type contains a single binary part and your use of the message is simple subscribe/publish, consider using the unparsed payload section in your application rather than declaring a binary message part.

Notice, however, that since AMPS does not parse or interpret this section, it may not appear in serialized representations of the message (for example, log messages, Galvanometer display, and so on). Any use of the message that considers the whole message (such as delta publish or delta subscribe) will not consider the unparsed payload.

Content Filtering with Composite Message Types

Composite message types support filtering on the contents of the composite message. There are some simple conventions to remember when constructing expressions to filter on. For more details about content filtering, see Filtering Subscriptions by Content.

These conventions are consistent anywhere that AMPS needs to find a value within the composite message type. That includes content filters for client subscriptions, identifying SOW keys, creating views and aggregates, creating conflated topics, and so on.

composite-global

When using the composite-global message type, AMPS combines all parts of the message into a unified set of XPath identifiers. AMPS creates the set of identifiers for each part of the message. If different parts of the message contain the same identifier, AMPS treats that identifier as though the identifier contained an array of values: AMPS creates an array that contains all of the values in the different parts of the message. Message types that do not support content filtering do not provide XPath identifiers.

For example, consider the message below for a composite-global message type that includes two json parts and a binary part:

{"id":1,"data":"sample","message":"part one message"}
{"message":"another part","customer":"Awesome Amalgamated, Ltd."}
0xDEEA0934DF23A37780934...

AMPS constructs the following set of XPath identifiers and values:

Identifier	Value
/id	1
/data	"sample"
/message	["part one message", "another part"]
/customer	"Awesome Amalgamated, Ltd."

In short, when using composite-global, AMPS combines the parsable parts of the message into a single global set of XPath values, and ignores any part of the message that cannot be parsed.

composite-local

When using the composite-local message type, AMPS creates a distinct set of XPath identifiers for each part of the message. AMPS adds an XPath step with the position of the message part at the beginning of the identifier. Message types that do not support content filtering do not provide XPath identifiers, and AMPS skips over them.

For example, consider the message below for a composite-local message type that includes two json parts and a binary part:

{"id":1,"data":"sample","message":"part one message"}
{"message":"another part","customer":"Awesome Amalgamated, Ltd."}
0xDEEA0934DF23A37780934...

AMPS constructs the following set of XPath identifiers and values:

Identifier	Value
/0/id	1
/0/data	"sample"
/0/message	"part one message"
/1/message	"another part"
/1/customer	"Awesome Amalgamated, Ltd."

In short, when using composite-local, AMPS creates XPath identifiers for each part of the message, using the position of the message part within the composite as the first part of the identifier. AMPS skips over any part of the message that cannot be parsed, and simply produces no values for that part of the message.

Choosing a Composite Type

To choose which composite type best fits your application, consider the following factors:

• If you need to use delta messaging with this message type, use composite-local.

• If there may be redundant field names in the parts of the message, and it is important to be able to filter based on which part contains the field, use composite-local.

• If you need to be able to create views of this type, use composite-local.

Otherwise, composite-global may be easier and more straightforward for client filtering, since clients do not need to know the detailed structure of the message type to be able to filter on the message. Notice, though, that subscribers will need to know how the message type is defined, since that will determine whether subscribers need to include part specifiers (/0, /1, and so on) to their filters.