Example: Hub and Spoke / Expandable Mesh

For more complex replication topologies, or in a situation where an installation may want to scale out to accommodate an ever-increasing number of subscribers, consider using a "hub and spoke" topology.

This topology is particularly useful in cases where a large number of applications need to operate over the same data, but the applications themselves are largely independent of each other, where data is consumed in a different region or different organization than where the data originates, or in cases where given applications require intensive CPU or memory resources to work with the data, whereas other applications using the same data do not require these resources. For example, if two applications have different CPU-intensive views over the same data, isolating those applications into separate application instances can help to reduce the resources required for any one instance.

In this topology, replication is handled by AMPS instances dedicated to managing replication, as shown in the diagram below. In this strategy, each instance has one of three distinct roles:

• An ingestion instance accepts messages from a publisher into the AMPS replication fabric. All ingestion instances replicate to each other (using sync acknowledgment) and replicate to the hub (also using sync acknowledgment).

  The ingestion instances do not define a state of the world.

• One or more hub instances that accept messages from the ingestion instances and replicate those messages to the application instances.

  The hub instances do not replicate back to the ingestion instances, and they do not define a state of the world.

• The application instances provide messages to applications that use the messages.

  These instances do not replicate back to the hub instances. If an application will use multiple instances, these instances replicate to each other using sync acknowledgment.

  The application instances define the state of the world as needed -- any Topics, Views, ConflatedTopics, LocalQueues, or GroupLocalQueues that the application will use.

  Different applications may use different application instances: each application instance only needs to define the state of the world that the applications that use that instance need.

This architecture provides decoupling between publishers and applications, and decoupling between different applications that use the same message stream.

This topology also reduces the risk and expense of adding more instances for application use. Only the "hub" instances need to be updated to add or remove application instances. Since the "hub" maintains only messages for replication (no state of the world is defined on the "hub" instances), adding or removing a destination at the hub instance is very efficient. Recovery times for the hub are very quick since the only state that needs to be recovered is the state of the transaction log itself.

In the simplest configuration, the "hub" instance or instances simply pass through all messages and all topics to all downstream instances, leaving the application instances to determine what topics should be replicated. In more sophisticated configurations, the "hub" instances can direct topics for specific applications to a specific set of instances.

The hub and spoke topology has the following advantages:

• Easy to add and remove instances to a replication fabric.

• Allows the ability to create autonomous groups of instances servicing a given application.

• High resilience to failures within the application instances.

• In many situations, reduces the bandwidth required to keep a large number of instances up to date (as compared with direct replication between the instances).

The hub and spoke topology has the following limitations:

• In some topologies, may have higher latency for active publishes.

• Does not support fully distributed queues (use local queues or group local queues on a subset of instances instead).

• Requires an instance of AMPS (or two, for HA) that does not have client activity.

• Requires exclusion of replication validation to the hub instance.

The diagram below shows an example of the hub and spoke topology:

Hub and spoke topology