Built-in Workers
The framework provides a few worker modules for tasks that are generic enough that they are likely to be required in most nowcast systems.
rotate_logs
Worker
The nemo_nowcast.workers.rotate_logs
worker iterates through the nowcast system logging handlers,
calling the doRollover()
method on any that are instances of
logging.handlers.RotatingFileHandler
.
This worker is normally launched in automation at the end of a nowcast processing cycle (e.g. end of the day).
It can also be launched from the command-line by the nowcast administrator as necessary for system maintenance.
Nowcast systems typically use rotating log files to avoid log files that grow without limit, and to split the log information into logical pieces. Please see Logging for more details of logging in nowcast systems, and Logging Configuration for information about how to configure rotating log files.
Rotating the log files at the end of a day’s nowcast processing is the recommended pattern. To implement that in your nowcast system:
Add a rotate_logs section to the workers section of the Message Registry section of your Nowcast Configuration File:
rotate_logs: checklist key: log rotation success: log files rotated failure: log file rotation failed crash: rotate_logs worker crashed
Add a
after_rotate_logs()
function to yournowcast.next_workers
module:def after_rotate_logs(msg, config, checklist): """Calculate the list of workers to launch after the rotate_logs worker ends. :arg msg: Nowcast system message. :type msg: :py:class:`collections.namedtuple` :arg config: :py:class:`dict`-like object that holds the nowcast system configuration that is loaded from the system configuration file. :type config: :py:class:`nemo_nowcast.config.Config` :arg dict checklist: System checklist: data structure containing the present state of the nowcast system. :returns: Worker(s) to launch next :rtype: list """ return []
Since log file rotation is generally the last thing to happen in a nowcast’s daily cycle of operations we simply return an empty list; i.e. there are no next workers.
Add a:
NextWorker('nemo_nowcast.workers.rotate_logs')
object to the list of next workers returned by the
after_worker_name()
function for the worker that you want the log file rotation operation to follow.The recommended pattern is that the
rotate_logs
worker be launched immediately after successful execution of the clear_checklist Worker.
clear_checklist
Worker
The nemo_nowcast.workers.clear_checklist
worker sends a message to the nowcast system manager requesting that it clear its system state checklist.
This worker is normally launched in automation at the end of a nowcast processing cycle (e.g. end of the day), just prior to launching the rotate_logs Worker.
It can also be launched from the command-line by the nowcast administrator as necessary for system maintenance.
Clearing the checklist just before rotating the log files at the end of a day’s nowcast processing is the recommended pattern. To implement that in your nowcast system:
Add a
clear_checklist
section to the workers section of the Message Registry section of your Nowcast Configuration File:clear_checklist: clear checklist: request that manager clear system state checklist success: system state checklist cleared failure: system state checklist clearance failed crash: clear_checklist worker crashed
Note
No checklist key element is required because the clear_checklist worker is a special case worker that does not return any information to add to the checklist (having just requested that it be cleared).
Add a
after_clear_checklist()
function to yournowcast.next_workers
module:def after_clear_checklist(msg, config, checklist): """Calculate the list of workers to launch after the clear_checklist worker ends. :arg msg: Nowcast system message. :type msg: :py:class:`collections.namedtuple` :arg config: :py:class:`dict`-like object that holds the nowcast system configuration that is loaded from the system configuration file. :type config: :py:class:`nemo_nowcast.config.Config` :arg dict checklist: System checklist: data structure containing the present state of the nowcast system. :returns: Worker(s) to launch next :rtype: list """ next_workers = { 'crash': [], 'failure': [], 'success': [NextWorker('nemo_nowcast.workers.rotate_logs')], } return next_workers[msg.type]
The recommended pattern is to launch the rotate_logs Worker upon success of the
clear_checklist
worker.Add a:
NextWorker('nemo_nowcast.workers.clear_checklist')
object to the list of next workers returned by the
after_worker_name()
function for the worker that you want the log file rotation operation to follow.