LogoLogo
AMPS Server Documentation 5.3.4
AMPS Server Documentation 5.3.4
  • Welcome to AMPS 5.3.4
  • Introduction to AMPS
    • Overview of AMPS
    • Getting Started With AMPS
      • Installing AMPS
      • Starting AMPS
      • JSON Messages - A Quick Primer
      • spark: the AMPS command-line client
      • Evaluating AMPS on Windows or MacOS
      • Galvanometer and RESTful Statistics
    • AMPS Basics: Subscribe and Publish to Topics
    • State of the World (SOW): The Message Database
      • When Should I Store a Topic in the SOW?
      • How Does the SOW Work?
      • Configuration
      • Queries
      • Atomic Query and Subscribe
      • Advanced Messaging and the SOW
    • Record and Replay Messages with the AMPS Transaction Log
    • Message Queues
    • Scenario and Feature Reference
      • Recovery Strategies
    • Getting Support
    • Advanced Topics
    • Next Steps
  • AMPS Evaluation Guide
    • Introduction
    • Evaluation and Development with AMPS
    • Tips on Measuring Performance
    • Next Steps
  • AMPS User Guide
    • Introduction
      • Product Overview
      • Requirements
      • Organization of this Guide
        • Documentation Conventions
      • Technical Support
    • Installing and Starting AMPS
      • Installing AMPS
      • Starting AMPS
      • Production Configuration
    • Subscribe and Publish
      • Topics
      • Filtering Subscriptions by Content
      • Conflated Subscriptions
      • Replacing Subscriptions
      • Messages in AMPS
      • Message Ordering
      • Retrieving Part of a Message
    • AMPS Expressions
      • Syntax
      • Identifiers
      • AMPS Data Types
      • Grouping and Order of Evaluation
      • Logical Operators
      • Arithmetic Operators
      • Comparison Operators
      • LIKE Operator
      • Conditional Operators
      • Working with Arrays
      • Regular Expressions
      • Performance Considerations
    • AMPS Functions
      • AMPS Function Overview
      • String Comparison Functions
      • Concatenating Strings
      • Managing String Case
      • Replacing Text in Strings
      • String Manipulation Functions
      • Date and Time Functions
      • Array Reduce Functions
      • Geospatial Functions
      • Numeric Functions
      • CRC Functions
      • Message Functions
      • Client Functions
      • Coalesce Function
      • AMPS Information Functions
      • Typed Value Creation
      • Constructing Fields
      • Aggregate Functions
    • State of the World (SOW) Topics
      • How Does the SOW Work?
      • Using the State of the World
      • Understanding SOW Keys
      • Indexing SOW Topics
      • Programmatically Deleting Records from the Topic State
      • SOW Maintenance
        • Creating a Maintenance Schedule for a Topic
        • Setting Per-Message Lifetime
      • Storing Multiple Logical Topics in One Physical Topic
    • Querying the State of the World (SOW)
      • Overview of SOW Queries
      • Query and Subscribe
      • Historical SOW Topic Queries
      • Managing Result Sets
      • Batching Query Results
    • Out-of-Focus Messages (OOF)
    • State of the World Message Enrichment
    • Incremental Message Updates
      • Using Delta Publish
      • Understanding Delta Publish
      • Delta Publish Support
    • Receiving Only Updated Fields
      • Using Delta Subscribe
      • Identifying Changed Records
      • Conflated Subscriptions and Delta Subscribe
      • Select List and Delta Subscribe
      • Options for Delta Subscribe
    • Conflated Topics
    • Aggregation and Analytics
      • Understanding Views
      • Defining Views and Aggregations
      • Constructing Field Contents
      • Best Practices for Views
      • View Examples
      • Aggregated Subscriptions
    • Record and Replay Messages
      • Using the Transaction Log and Bookmark Subscriptions
      • Understanding Message Persistence
      • Configuring a Transaction Log
      • Replaying Messages with Bookmark Subscription
      • Managing Journal Files
      • Using amps-grep to Search the Journal
    • Message Queues
      • Getting Started with AMPS Queues
      • Understanding AMPS Queuing
      • Advanced Messaging and Queues
      • Replacing Queue Subscriptions
      • Handling Unprocessed Messages
      • Advanced Queue Configuration
      • Queue Subscriptions Compared to Bookmark Replays
    • Message Types
      • Default Message Types
      • BFlat Messages
      • MessagePack Messages
      • Composite Messages
      • Protobuf Message Types
      • Struct Message Types
    • Command Acknowledgment
      • Requesting Acknowledgments
      • Receiving Acknowledgments
      • Bookmark Subscriptions and Completed Acknowledgments
      • Bookmark Subscriptions and Persisted Acknowledgments
      • Acknowledgment Conflation and Publish Acknowledgements
    • Transports
      • Client Connections
      • Replication Connections
      • Transport Filters
    • Running AMPS as a Linux Service
      • Installing the Service
      • Configuring the Service
      • Managing the Service
      • Uninstalling the Service
    • Logging
      • Configuring Logging
      • Log Message Format
      • Message Levels
      • Message Categories
      • Logging to a File
      • Logging to a Compressed File
      • Logging to Syslog
      • Logging to the Console
      • Looking up Errors with ampserr
    • Event Topics
      • Client Status Events
      • SOW Statistics Events
      • Persisting Event Topics
    • Utilities
      • Command-Line Basic Client
      • Dump clients.ack File
      • Dump journal File
      • Dump queues.ack File
      • Dump SOW File
      • Dump Journal Topic Index File
      • Find Bookmark or Transaction ID in Transaction Log
      • Find Information in Error Log or Transaction Log
      • Identify Type of AMPS File
      • List/Explain Error Codes
      • Query Statistics Database
      • Statistics Database Report
      • Storage Performance Testing
      • Submit Minidump to 60East
      • Obsolete Utility: Upgrade File Formats
    • Monitoring AMPS
      • Statistics Collection
        • Time Range Selection
        • Output Formatting
      • Galvanometer
      • Configuring Monitoring
    • Automating AMPS with Actions
    • Replicating Messages Between Instances
      • Replication Basics
      • Configuring Replication
      • Replication Configuration Validation
      • Replication Resynchronization
      • Replication Compression
      • Destination Server Failover
      • Two-Way Replication
      • PassThrough Replication
      • Guarantees on Ordering
      • Replication Security
      • Understanding Replication Message Routing
      • Replicated Queues
      • Replication Best Practices
    • Highly Available AMPS Installations
      • Overview of High Availability
        • Example: Pair of Instances for Failover
        • Example: Regional Distribution
        • Example: Regional Distribution with HA
        • Example: Hub and Spoke / Expandable Mesh
      • Details of High Availability
      • Slow Client Management and Capacity Limits
      • Message Ordering Considerations
    • Operation and Deployment
      • Capacity Planning
      • Linux OS Settings
      • Upgrading AMPS
      • Using AMPS with a Proxy
      • Operations Best Practices
    • Securing AMPS
      • Authentication
      • Entitlement
      • Providing an Identity for Outbound Connections
      • Protecting Data in Transit Using TLS/SSL
    • Troubleshooting AMPS
      • Planning for Troubleshooting
      • Diagnostic Utilities
      • Finding Information in the Log
      • Reading Replication Log Messages
      • Troubleshooting Disconnected Clients
      • Troubleshooting Regular Expression Subscriptions
    • AMPS Distribution Layout
    • Optionally-Loaded Modules
      • Optional Functions
        • Legacy Messaging Functions
        • Special-Purpose Functions
      • Optional SOW Key Generator
        • Chaining Key Generator
      • Optional Authentication/Entitlements Modules
        • RESTful Authentication and Entitlements
        • Multimethod Authentication Module
        • Simple Access Entitlements Module
      • Optional Authenticator Modules
        • Multimethod Authenticator
        • Command Execution Authenticator
    • AMPS Statistics
    • File Format Versions
  • AMPS Configuration Guide
    • AMPS Configuration Basics
      • Getting Started With AMPS Configuration
      • Units, Intervals, and Environment Variables
      • Working With Configuration Files
      • Including External Files
    • Instance Level Configuration
    • Admin Server and Statistics
    • Modules
    • Message Types
    • Transports
    • Logging
    • State of the World (SOW)
      • SOW/Topic
      • SOW/*Queue
      • SOW/ConflatedTopic
      • SOW/View
    • Replication
      • Replication Validation
    • Transaction Log
    • Authentication
    • Entitlement
    • Actions
      • Configuration for Actions
      • Choosing When an Action Runs
        • On a Schedule
        • On AMPS Startup or Shutdown
        • On a Linux Signal
        • On a REST Request
        • On Minidump Creation
        • On Client Connect or Disconnect
        • On Client Logon
        • On Client Offline Message Buffering
        • On Subscribe or Unsubscribe
        • On Incoming Replication Connections
        • On Outgoing Replication Connections
        • On Message Published to AMPS
        • On Message Delivered to Subscriber
        • On Message Affinity
        • On SOW Message Expiration
        • On SOW Message Delete
        • On OOF Message
        • On Message Condition Timeout
        • On Message State Change
        • On a Custom Event
      • Choosing What an Action Does
        • Rotate Error/Event Log
        • Compress Files
        • Truncate Statistics
        • Manage Transaction Log Journal Files
        • Remove Files
        • Delete SOW Messages
        • Compact SOW Topic
        • Query SOW Topic
        • Manage Security
        • Enable or Disable Transports
        • Publish Message
        • Manage Replication Acknowledgment
        • Extract Values from a Message
        • Translate Data Within an Action
        • Increment Counter
        • Raise a Custom Event
        • Execute System Command
        • Manage Queue Transfers
        • Create Minidump
        • Shut Down AMPS
        • Debug Action Configuration
      • Conditionally Stopping an Action
        • Based on File System Capacity
        • Based on an Expression
      • Examples of Action Configuration
        • Archive Journals Once a Week
        • Archive Journals On RESTful Command
        • Record Expired Queue Messages to a Dead Letter Topic
        • Copy Messages that Exceed a Timeout to a Different Topic
        • Deactivate and Reactivate Security on Signals
        • Reset Entitlements for a Disconnected Client
        • Extract Values from a Published Message
        • Shut Down AMPS When a Filesystem Is Full
        • Increment a Counter and Echo a Message
    • Protocols
  • AMPS Monitoring Guide
    • Statistics Types
    • Table Reference
    • Administrative Actions
    • Host Statistics
      • cpu
      • disks
      • memory
      • name
      • network
    • AMPS Instance Statistics
      • api
      • clients
      • config.xml
      • config_path
      • conflated_topics
      • cpu
      • cwd
      • description
      • environment
      • lifetimes
      • logging
      • memory
      • message_types
      • name
      • name_hash
      • pid
      • processors
      • queues
      • queries
      • replication
      • sow
      • statistics
      • subscriptions
      • timestamp
      • transaction_log
      • transports
      • tuning
      • uptime
      • user_id
      • version
      • views
  • AMPS Command Reference
    • Commands to AMPS
      • logon
      • Publishing
        • publish
        • delta_publish
      • Subscribing to and Querying Topics
        • subscribe
        • sow
        • sow_and_subscribe
        • unsubscribe
        • delta_subscribe
        • sow_and_delta_subscribe
      • Removing Messages (SOW/Topic or Message Queue)
      • heartbeat
      • flush
    • Responses from AMPS
      • sow: Content from Server
      • publish: Content from Server
      • oof: Content from Server
      • ack: Status from Server
      • group_begin / group_end : Result Set Delimiters
    • Protocol Reference
      • AMPS Protocol
      • Legacy Protocols Reference
    • Command Cookbook
      • Cookbook: Delta Publish
      • Cookbook: Delta Subscribe
      • Cookbook: Publish
      • Cookbook: SOW
      • Cookbook: SOW and Delta Subscribe
      • Cookbook: SOW and Subscribe
      • Cookbook: SOW Delete
      • Cookbook: Subscribe
  • Deployment Checklist
    • Ensure Sufficient Capacity
    • Apply System and AMPS Configuration
    • Create Maintenance Plan
    • Create Monitoring Strategy
    • Create Patch and Upgrade Plan
    • Create and Test Support Process
    • Conclusion
  • AMPS Clients
    • Performance Tips and Best Practices
    • C++
    • C#/.NET
    • Java
    • JavaScript
    • Python
Powered by GitBook

Get Help

  • FAQ
  • Legacy Documentation
  • Support / Contact Us

Get AMPS

  • Evaluate
  • Develop

60East Resources

  • Website
  • Privacy Policy

Copyright 2013-2024 60East Technologies, Inc.

On this page
  • Simple Aggregate View Example
  • Multiple Topic Aggregate Example
  • View Projected Into Different Message Type
Export as PDF
  1. AMPS User Guide
  2. Aggregation and Analytics

View Examples

The following sections provide examples of practical scenarios that demonstrate AMPS views and how they can be used to aggregate and analyze data.

Simple Aggregate View Example

For a potential usage scenario, imagine the topic ORDERS which includes the following NVFIX message schema:

NVFIX Tag

Description

OrderID

Unique order identifier

Tick

Symbol

ClientId

Unique client identifier

Shares

Currently executed shares for the chain of orders

Price

Average price for the chain of orders

This topic includes information on the current state of executed orders, but may not include all the information we want updated in real-time. For example, we may want to monitor the total value of all orders executed by a client at any moment. If ORDERS was a SQL Table within an RDBMS, the “view” we would want to create would be similar to:

CREATE VIEW TOTAL_VALUE AS
SELECT ClientId, SUM(Shares * Price) AS TotalCost,
                 SUM(Shares * Price)/SUM(Shares) AS WeightedAveragePrice
FROM ORDERS
GROUP BY ClientId

As defined above, the TOTAL_VALUE view would only have two fields:

  1. ClientId: the client identifier

  2. TotalCost: the summation of current order values by client

Views in AMPS are specified in the AMPS configuration file in View sections, which are defined in the SOW section. The example above would be defined as:

<SOW>
    <Topic>
        <Name>ORDERS</Name>
        <MessageType>nvfix</MessageType>
        <Key>/OrderID</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <Name>TOTAL_VALUE</Name>
        <UnderlyingTopic>ORDERS</UnderlyingTopic>
        <MessageType>nvfix</MessageType>
        <Projection>
            <Field>/ClientId</Field>
            <Field>SUM(/Shares * /Price) AS /TotalCost</Field>
            <Field>SUM(/Shares * /Price) / SUM(/Shares) AS /WeightedAveragePrice</Field>
        </Projection>
        <Grouping>
            <Field>/ClientId</Field>
        </Grouping>
    </View>
</SOW>

Views require an underlying topic in the SOW (which includes queues, conflated topics, or other views).

The Topic element is the name of the new topic that is being defined. This Topic value will be the topic that can be used by clients to subscribe for future updates or perform SOW queries against.

The UnderlyingTopic is the SOW topic or topics that the view operates on. That is, the UnderlyingTopic is where the view gets its data from. All XPath references within the Projection fields are references to values within this underlying SOW topic (unless they appear on the right-hand side of the AS keyword.)

Unlike ANSI SQL, AMPS allows you to include fields in the Projection that are not included in the Grouping or used within the aggregate functions. In this case, AMPS uses the last value processed for the value of these fields. AMPS enforces a consistent order of updates to ensure that the value of the field is consistent across recovery and restart.

An unexpected 0 (zero) or null value in an aggregate field within a view usually means that the value is either zero or NaN. Most AMPS message types default to using 0 instead of NaN. However, any numeric aggregate function will result in a NaN if the aggregation includes a field that is not a number.

Finally, the Grouping section is a list of one or more Field tags that define how the records in the UnderlyingTopic will be grouped to form the records in the view. In this example, we grouped by the tag holding the client identifier. However, we could have easily made this the “Symbol” tag /Tick.

In the below example, we group by the /ClientId because we want to count the number of orders for each client that have a value greater than 1,000,000:

<SOW>
    ...

    <View>
        <Name>NUMBER_OF_ORDERS_OVER_ONEMILL</Name>
        <UnderlyingTopic>ORDERS</UnderlyingTopic>
        <Projection>
            <Field>/ClientId</Field>
            <Field><![CDATA[SUM(IF(/shares * /price > 1000000, /shares * /price, NULL)) AS /AggregateValue]]></Field>
            <Field>SUM(IF(/Shares * /Price &gt; 1000000, /Shares * /Price, NULL)) AS /AggregateValue2</Field>
        </Projection>
        <Grouping>
            <Field>/ClientId</Field>
        </Grouping>
        <MessageType>nvfix</MessageType>
    </View>

    ...
</SOW>

Notice that the /AggregateValue and /AggregateValue2 will contain the same value; however /AggregateValue was defined using an XML CDATA block and /AggregateValue2 was defined using the XML > entity reference.

Since the AMPS configuration is XML, special characters in projection expressions must either be escaped with XML entity references or wrapped in a CDATA section.

Multiple Topic Aggregate Example

This example demonstrates how to create an aggregate view that uses more than one topic as a data source. For a potential usage scenario, imagine that another publisher provides a COMPANIES topic which includes the following NVFIX message schema:

NVFIX Tag

Description

CompanyId

Unique identifier for the company

Tick

Symbol

Name

Company name

This topic includes the name of the company, and an identifier used for internal record keeping in the trading system. Using this information, we want to provide a running total of orders for that company, including the company name.

If ORDERS and COMPANIES were a SQL Table within an RDBMS, the “view” we would want to create would be similar to:

CREATE VIEW TOTAL_COMPANY_VOLUME AS
SELECT COMPANIES.CompanyId, COMPANIES.Tick, COMPANIES.Name, SUM(ORDERS.Shares) AS TotalVolume
FROM COMPANIES LEFT OUTER JOIN ORDERS
    ON COMPANIES.Tick = ORDERS.Tick
GROUP BY ORDERS.Tick

As defined above, the TOTAL_COMPANY_VOLUME table would have four columns:

  1. CompanyId: the identifier for the company

  2. Tick: the ticker symbol for the company

  3. Name: the name of the company

  4. TotalVolume: the total number of shares involved in orders

To create this view, use the following definition in the AMPS configuration file:

<SOW>
    <Topic>
        <Name>ORDERS</Name>
        <MessageType>nvfix</MessageType>
        <Key>/OrderID</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <Topic>
        <Name>COMPANIES</Name>
        <MessageType>nvfix</MessageType>
        <Key>/CompanyId</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <Name>TOTAL_COMPANY_VOLUME</Name>
        <UnderlyingTopic>
            <Join>[ORDERS]./Tick = [COMPANIES]./Tick</Join>
        </UnderlyingTopic>
        <MessageType>nvfix</MessageType>
        <Projection>
            <Field>[COMPANIES]./CompanyId</Field>
            <Field>[COMPANIES]./Tick</Field>
            <Field>[COMPANIES]./Name</Field>
            <Field>SUM([ORDERS]./Shares) AS /TotalVolume</Field>
        </Projection>
        <Grouping>
            <Field>[ORDERS]./Tick</Field>
        </Grouping>
    </View>
</SOW>

As with the single topic example, first define the underlying topics as SOW topics. Next, the view defines the underlying topic that is the source of the data. In this case, the underlying topic is a join between two topics in the instance. The definition next declares the message type of the projected messages. The message types that you join can be different types, and the projected messages can be a different type than the underlying message types. The projection uses three fields from the COMPANIES topic and one field that is aggregated from messages in the ORDERS topic. The projection groups results by the Tick symbols that appear in messages in the ORDERS topic.

View Projected Into Different Message Type

This example shows how to project an underlying topic of one message type into a topic of a different message type.

There is very little difference between this example and the single topic view example above. The main difference is that, because the destination view has a different message type than the underlying topic, every reference to a field from the underlying topic must be fully-qualified with the message type.

As before, imagine the topic ORDERS which includes the following NVFIX message schema:

NVFIX Tag

Description

OrderID

Unique order identifier

Tick

Symbol

ClientId

Unique client identifier

Shares

Currently executed shares for the chain of orders

Price

Average price for the chain of orders

As before, we want to project the summation of current order values by client. The TOTAL_VALUE view will have two fields:

  1. ClientId: the client identifier

  2. TotalCost: the summation of current order values by client

However, in this case, we want to project the summary into a JSON document. To do this we simply specify that the final view will be in JSON format, and fully qualify all references to the underlying topic in the view definition.

The example above would be defined as:

<SOW>
    <Topic>
        <Name>ORDERS</Name>
        <MessageType>nvfix</MessageType>
        <Key>/OrderID</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <Name>TOTAL_VALUE</Name>
        <UnderlyingTopic>[nvfix].[ORDERS]</UnderlyingTopic>
        <MessageType>json</MessageType>
        <Projection>
            <Field>[nvfix].[ORDERS]./ClientId AS /ClientId</Field>
            <Field>SUM([nvfix].[ORDERS]./Shares * [nvfix].[ORDERS]./Price) AS /TotalCost</Field>
        </Projection>
        <Grouping>
            <Field>[nvfix].[ORDERS]./ClientId</Field>
        </Grouping>
    </View>
</SOW>

This example uses an underlying topic in NVFIX format, computes an aggregation by ClientId, and then produces output in JSON format.

PreviousBest Practices for ViewsNextAggregated Subscriptions

Last updated 1 year ago

The Projection section is a list of 1 or more Field tags that define what the view will contain. The field specifications can contain either a raw XPath value, as in /ClientId above, which is a straight copy of the value found in the underlying topic into the view topic using the same target XPath or an expression as described in the section on . In the case of ClientId, if we had wanted to translate the tag into a different tag, such as CID, then we could have used the AS keyword to do the translation as in /ClientId AS /CID.

Updates to underlying topics can potentially cause many more updates to downstream views, which can create stress on downstream clients subscribed to the view. If any underlying topic has frequent updates to the same records and/or a real-time view is not required, as in a GUI, then a replica of the topic may be a good solution to reduce the frequency of the updates and conserve bandwidth. For more on conflated topics, please see .

Conflated Topics
Constructing View Fields