LogoLogo
AMPS Server Documentation 5.3.5 (Early Access)
AMPS Server Documentation 5.3.5 (Early Access)
  • Welcome to AMPS
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
    • Configuring AMPS
      • Getting Started with AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working with Configuration Files
      • Including External Files
      • Production Configuration
      • Instance-Level Configuration
      • Configuration Options Quick Reference
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • Extracting Matching Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
      • Configuring a SOW
        • Configuring Topics in a SOW
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
      • Configuring Conflated Topics in a SOW
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
      • Configuring Views in a SOW
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
      • Configuring a Transaction Log
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Queue Subscriptions Compared to Bookmark Replays
      • Configuring Queues in a SOW
      • Advanced Queue Configuration
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
      • Configuring Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
      • Configuring Transports
      • Configuring Protocols
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to Files
      • Logging to Syslog
      • Looking up Errors with ampserr
      • Configuring Logging
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Configuring AMPS for Automation with Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Replicating Messages Between Instances
      • Replication Basics
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
      • Configuring Replication
        • Configuring Outgoing Replication Destinations
        • Configuring Incoming Replication Transports
        • Replication Configuration Validation
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
        • Configuring Authentication
      • Entitlement
        • Configuring Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
      • Loadable Authentication/Entitlements Modules
        • Simple Access Entitlements Module
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • OAuth Authentication Module
      • Loadable Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Loadable Function Modules
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Loadable Authentication/Entitlements Modules
      • Loadable Authenticator Modules
      • Loadable SOW Key Generator
        • Chaining Key Generator
      • Configuring Modules
    • AMPS Statistics
    • File Format Versions
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu (host statistics)
      • disks (host statistics)
      • memory (host statistics)
      • name (host statistics)
      • network (host statistics)
    • AMPS Instance Statistics
      • api (instance statistics)
      • clients (instance statistics)
      • config.xml (instance statistics)
      • config_path (instance statistics)
      • conflated_topics (instance statistics)
      • cpu (instance statistics)
      • cwd (instance statistics)
      • description (instance statistics)
      • environment (instance statistics)
      • lifetimes (instance statistics)
      • logging (instance statistics)
      • memory (instance statistics)
      • message_types (instance statistics)
      • name (instance statistics)
      • name_hash (instance statistics)
      • pid (instance statistics)
      • processors (instance statistics)
      • queues (instance statistics)
      • queries (instance statistics)
      • replication (instance statistics)
      • sow (instance statistics)
      • statistics (instance statistics)
      • subscriptions (instance statistics)
      • timestamp (instance statistics)
      • transaction_log (instance statistics)
      • transports (instance statistics)
      • tuning (instance statistics)
      • uptime (instance statistics)
      • user_id (instance statistics)
      • version (instance statistics)
      • views (instance statistics)
  • AMPS Command Reference
    • Commands to AMPS
      • logon command
      • Publishing
        • publish command
        • delta_publish command
      • Subscribing to and Querying Topics
        • subscribe command
        • sow command
        • sow_and_subscribe command
        • unsubscribe command
        • delta_subscribe command
        • sow_and_delta_subscribe command
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat command
      • flush command
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end: Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
  • Glossary
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
  • Message Type Specific Options
  • FIX/NVFIX Options
  • JSON Option
  • Composite-Local and Composite-Global Option
  • Google Protocol Buffer Options
  • Custom Message Types
Export as PDF
  1. AMPS User Guide
  2. Message Types

Configuring Message Types

PreviousStruct Message TypesNextCommand Acknowledgment

Last updated 7 days ago

The MessageTypes tag defines the message types supported by the AMPS instance. A single AMPS instance can support multiple message types.

As mentioned in the section, MessageType definitions for fix, nvfix, xml, json, bflat, msgpack, bson, and binary are automatically loaded by AMPS. You only need to define a new MessageType for these types if the settings for the message type need to be changed (for example, to create a custom FIX-based type that changes the FieldSeparator of the message).

The MessageTypes tag can contain multiple MessageType definitions. To add more than one message type to the message types that are loaded by default, include multiple MessageType tags.

AMPS loads the capability to use Google protocol buffer (protobuf) messages by default. To use protocol buffer messages, you configure one or more message types that use the protobuf module and load the .proto files that define the format of the messages you will be processing with AMPS.

AMPS also supports the ability to create a composite message type by combining a number of existing message types. Composite message types are defined using the MessageType configuration element.

Described below are the configuration items available for a MessageType. Expand each item for more details.

Name (required)

This element defines the name for the message type.

The name is used to specify MessageType in other sections such as Transport, TransactionLog and the elements of the SOW section.

By default, AMPS loads message types for fix, nvfix, json, bflat, msgpack, bson, xml and binary. It is typically not necessary to configure these types for use.

Other message types, such as Google protocol buffers, are available by default, but require configuration to be used.

Module

This element specifies the name of the module that will be loaded for this message type.

By default, AMPS loads the modules that implement the following message types: fix, nvfix, json, bflat, msgpack, bson, xml, protobuf, and binary.

AMPS supports creating composite message types out of existing message types using the composite-global and composite-local modules, which are loaded by default.

AMPSVersionCompliance

Sets the version compatibility for FIX messages that AMPS sends to the /AMPS/SOWStats topic.

AMPS accepts three values for this option:

  • 2 creates messages that use the FIX field tags used by AMPS 2.X.

  • 4 creates messages that use the default FIX field tags (the values used in AMPS 4.X). With this version, FIX messages use different field numbering for /AMPS/SOWStats and /AMPS/ClientStatus messages.

  • 5 creates messages that use a unified set of FIX tags. When this option is set to 5, AMPS uses consistent field numbering between /AMPS/SOWStats and /AMPS/ClientStatus messages (which is only available on versions 5.X and later).

For message types other than FIX, there is no difference between 4 and 5.

These message types were not supported in AMPS 2.X. AMPS provides reasonable values for these message types when this value is set to 2, but there is no backward compatibility to enforce.

For most cases, you can leave this option set to the default. If you are using a system that requires consistent FIX tags across messages, set this parameter to 5. If you are using an existing system that expects AMPS 2.X tags, set this parameter to 2.

Default: 4 (for compatibility with the largest number of existing installations)

Options

Options to pass to a custom message type module.

AMPS does not specify the format or type of the items within an Options element. AMPS simply parses the XML and then sends the XML to the module. If you are configuring a custom message type, see the documentation for that message type module for details.

Message Type Specific Options

Below are the configuration items that apply to specific message types.

FIX/NVFIX Options

Described below are the options that apply to fix and nvfix message types. Expand each item for more details.

FieldSeparator

Applies to fix and nvfix message types.

Sequence of characters used to separate field items in a FIX message.

Note: This field is the ASCII value of the char sequence.

HeaderSeparator

Applies to fix and nvfix message types.

Sequence of characters used to separate the header from the body in a FIX message.

Note: This field is the ASCII value of the char sequence.

MessageSeparator

Applies to fix and nvfix message types.

Sequence of characters used to separate message items in the body in a FIX message.

Note: This field is the ASCII value of the char sequence.

The example below defines a FIX-based message type with custom separators.

<MessageTypes>
    <MessageType>
        <Name>fix-custom</Name>
        <Module>fix</Module>
        <!-- The following are FIX specific options. -->
        <FieldSeparator>1</FieldSeparator>
        <HeaderSeparator>2</HeaderSeparator>
        <MessageSeparator>5</MessageSeparator>
    </MessageType>
</MessageTypes>

JSON Option

Described below is the option that applies to the json message type. Expand the item for more details.

EarlyTerminationOptimization

Applies to the json message type.

By default, AMPS includes an optimization to allow the server to only partially parse JSON messages. This may result in unexpected behavior for some messages.

For example, given a message such as { "code" : 1, "data" : "some data", "code" : 2 }, AMPS will report the value of code as 1 when this optimization is active. To ensure consistent results, in this mode AMPS always reports the first value for a field even when AMPS fully parses the message.

When set to false, the optimization is disabled. AMPS will fully parse all JSON messages and report the last value for a field. For the message above, AMPS would report the value of code as 2.

Default: true

The example below disables the optimization:

    <MessageType>
        <Name>json-custom</Name>
        <Module>json</Module>
        <EarlyTerminationOptimization>false</EarlyTerminationOptimization>
    </MessageType>

Composite-Local and Composite-Global Option

The MessageType entries for the composite message can be any AMPS message type, including both the built-in types and any previously defined message type.

Once the new composite message type is created, you can use the new type in the configuration file.

Described below is the option that applies to composite-local or composite-global message types. Expand the item for more details.

MessageType (required)

Applies to message types that use the composite-local or composite-global modules.

For composite message types, the MessageType definition must contain one or more message type declarations that specify the types that the composite message type contains.

For example, the MessageType element below declares a new composite message type named images. The new type contains a json document at the beginning of the message, followed by two uninterpreted binary message parts. AMPS will combine the XPath identifiers for all message parts into a single set of identifiers. Notice that, because only one part of the message type is parsable, using composite-global simplifies the identifiers for the message.

<MessageTypes>
    <MessageType>
        <Name>images</Name>
        <Module>composite-global</Module>
        <MessageType>json</MessageType>
        <MessageType>binary</MessageType>
        <MessageType>binary</MessageType>
    </MessageType>
</MessageTypes>

The example below defines a composite message type that combines a json message and a custom-payload message:

<MessageTypes>
    <MessageType>
        <Name>custom-composite</Name>
        <Module>composite-local</Module>
        <MessageType>json</MessageType>
        <MessageType>custom-payload</MessageType>
    </MessageType>
</MessageTypes>

Google Protocol Buffer Options

To use a protobuf message, you must first edit the configuration file to include a new MessageType. Then, specify the path to the protobuf file and the name of the protobuf file itself inside the MessageType.

Each message type references a ProtoFile, and specifies a single top-level type from the file. The ProtoFile may include other files through the standard protocol buffer include mechanism. Likewise, the top-level type may be any valid protocol buffer definition, including definitions that contain other types.

Once the protocol buffer MessageType is created as described above, you must either create a Transport that specifies that message type exactly, or you must create a Transport that can accept any known message type and ensure that the client specifies the new message type (in the example case, my-protobuf-message) in the connect string.

Described below are the options that apply to protobuf message types. Expand each item for more details.

Type (required)

Applies to message types that use the protobuf module.

The name of the type within the .proto file to use for this message type. The name must be package-qualified (for example, my.package.Message would load the type Message within the package my.package).

Obsolete usage - A previous meaning of this element was made obsolete in AMPS 4.0 and later versions. That usage was replaced by the Module directive.

ProtoPath (required)

Applies to message types that use the protobuf module.

The path in which to search for .proto files. The content of this element has the following syntax:

alias ; full-path

The alias provides a short identifier to use when searching for .proto files. The full-path is the path that is substituted for that identifier.

A configuration may omit the alias, and simply provide the path. For example, to use the path /mnt/repository/protodefs when no alias is provided, you would declare a ProtoPath of:

 /mnt/repository/protodefs

or

 ;/mnt/repository/protodefs

The following ProtoPath declaration sets proto-archive as an alias for /mnt/shared/protofiles:

proto-archive;/mnt/shared/protofiles

AMPS uses the aliases provided in this configuration item when processing import statements within the loaded .proto files, with the empty alias used for import statements that do not specify a path alias.

For example, given the definitions above, this import statement:

import "proto-archive/AType.proto";

will load the file at /mnt/shared/protofiles/AType.proto, while the import statement:

import "MyFavoriteType.proto";

will load the file at /mnt/repository/protodefs/MyFavoriteType.proto.

If no ProtoPath declaration sets an empty alias, all imports processed by AMPS must use one of the aliases set for the instance, or AMPS will fail to find the specified file.

Unless your existing definitions use an aliasing scheme, it is most convenient to set the empty alias.

You may specify any number of ProtoPath declarations.

ProtoFile (required)

Applies to message types that use the protobuf module.

The name of the .proto file to use for this message type. To use an alias, prefix the name of the file with the alias.

For example, if your ProtoPath declarations have created the proto-archive alias for the directory where your .proto files are stored, you could use the following to access the my-messages.proto file in that directory.

proto-archive/my-messages.proto

Below is a sample configuration of a protobuf message type:

<MessageTypes>
    <MessageType>
        <Name>my-protobuf-messages</Name>
        <Module>protobuf</Module>
        <ProtoPath>proto-archive;/mnt/shared/protofiles</ProtoPath>
        <ProtoFile>proto-archive/person.proto</ProtoFile>
        <Type>MyNamespace.Message</Type>
    </MessageType>
</MessageTypes>

Custom Message Types

The example below defines a custom message type. The Module in this case - type-module, must be the Name of a Module specified in the Modules section of the configuration file.

<MessageTypes>
    <MessageType>
        <Name>custom-payload</Name>
        <Module>type-module</Module>
    </MessageType>
</MessageTypes>
Default Message Types