The Message Broker is a long-running process that queues messages in both directions between the manager and the workers. Mediating the message transfers via queues means that it is not necessary to maintain direct connections between the manager and any active workers. This improves the robustness of the system with respect to:
fatal errors or bugs in the manager or worker code
network latency and reliability
operation across a distributed collection of computers
the occasional need in the maintenance of a nowcast system to stop and restart the manager
When the broker is started it binds to a workers port to listen for messages from workers, and a manager port to listen for messages from the manager. After that, the broker simply listens for messages and queues them in both directions between the workers and manager ports.
The server on which the broker is running, and the workers and manager port numbers that the system uses are defined in the ZeroMQ Server and Ports section of the Nowcast Configuration File.
If the manager or some of the workers run on different machines than the message broker it is necessary to ensure that the appropriate firewall rules are in place to allow traffic to pass between those machines via the worker and/or manager port(s).
The nowcast messaging system is based on the ZeroMQ distributed messaging framework. Message Broker is a wrapper around a ZeroMQ QUEUE device. It provides configuration, logging, and signal handling for the QUEUE device.
The recommended way to launch the message broker is to put it under the control of a process manager like Supervisor. Please see Nowcast Process Management for details.
Since manager/worker communication, and distributed logging all use ZeroMQ ports, it is crucial to ensure that all port numbers used are unique.