Message Broker

The Message Broker is a long-running process that queues messages in both directions between the manager and the workers. Mediating the message transfers via queues means that it is not necessary to maintain direct connections between the manager and any active workers. This improves the robustness of the system with respect to:

  • fatal errors or bugs in the manager or worker code

  • network latency and reliability

  • operation across a distributed collection of computers

  • the occasional need in the maintenance of a nowcast system to stop and restart the manager

  • etc.

When the broker is started it binds to a workers port to listen for messages from workers, and a manager port to listen for messages from the manager. After that, the broker simply listens for messages and queues them in both directions between the workers and manager ports.

../_images/MessageBroker.png

Schematic of a nowcast system message broker ports and message queues.

The server on which the broker is running, and the workers and manager port numbers that the system uses are defined in the ZeroMQ Server and Ports section of the Nowcast Configuration File.

Note

If the manager or some of the workers run on different machines than the message broker it is necessary to ensure that the appropriate firewall rules are in place to allow traffic to pass between those machines via the worker and/or manager port(s).

The nowcast messaging system is based on the ZeroMQ distributed messaging framework. Message Broker is a wrapper around a ZeroMQ QUEUE device. It provides configuration, logging, and signal handling for the QUEUE device.

The recommended way to launch the message broker is to put it under the control of a process manager like Supervisor. Please see Nowcast Process Management for details.

Note

Since manager/worker communication, and distributed logging all use ZeroMQ ports, it is crucial to ensure that all port numbers used are unique.