Built-in Workers

The framework provides a few worker modules for tasks that are generic enough that they are likely to be required in most nowcast systems.

rotate_logs Worker

The nemo_nowcast.workers.rotate_logs worker iterates through the nowcast system logging handlers, calling the doRollover() method on any that are instances of logging.handlers.RotatingFileHandler.

This worker is normally launched in automation at the end of a nowcast processing cycle (e.g. end of the day).

It can also be launched from the command-line by the nowcast administrator as necessary for system maintenance.

Nowcast systems typically use rotating log files to avoid log files that grow without limit, and to split the log information into logical pieces. Please see Logging for more details of logging in nowcast systems, and Logging Configuration for information about how to configure rotating log files.

Rotating the log files at the end of a day’s nowcast processing is the recommended pattern. To implement that in your nowcast system:

  1. Add a rotate_logs section to the workers section of the Message Registry section of your Nowcast Configuration File:

    rotate_logs:
      checklist key: log rotation
      success: log files rotated
      failure: log file rotation failed
      crash: rotate_logs worker crashed
    
  2. Add a after_rotate_logs() function to your nowcast.next_workers module:

    def after_rotate_logs(msg, config, checklist):
        """Calculate the list of workers to launch after the rotate_logs worker
        ends.
    
        :arg msg: Nowcast system message.
        :type msg: :py:class:`collections.namedtuple`
    
        :arg config: :py:class:`dict`-like object that holds the nowcast system
                     configuration that is loaded from the system configuration
                     file.
        :type config: :py:class:`nemo_nowcast.config.Config`
    
        :arg dict checklist: System checklist: data structure containing the
                             present state of the nowcast system.
    
        :returns: Worker(s) to launch next
        :rtype: list
        """
        return []
    

    Since log file rotation is generally the last thing to happen in a nowcast’s daily cycle of operations we simply return an empty list; i.e. there are no next workers.

  3. Add a:

    NextWorker('nemo_nowcast.workers.rotate_logs')
    

    object to the list of next workers returned by the after_worker_name() function for the worker that you want the log file rotation operation to follow.

    The recommended pattern is that the rotate_logs worker be launched immediately after successful execution of the clear_checklist Worker.

clear_checklist Worker

The nemo_nowcast.workers.clear_checklist worker sends a message to the nowcast system manager requesting that it clear its system state checklist.

This worker is normally launched in automation at the end of a nowcast processing cycle (e.g. end of the day), just prior to launching the rotate_logs Worker.

It can also be launched from the command-line by the nowcast administrator as necessary for system maintenance.

Clearing the checklist just before rotating the log files at the end of a day’s nowcast processing is the recommended pattern. To implement that in your nowcast system:

  1. Add a clear_checklist section to the workers section of the Message Registry section of your Nowcast Configuration File:

    clear_checklist:
      clear checklist: request that manager clear system state checklist
      success: system state checklist cleared
      failure: system state checklist clearance failed
      crash: clear_checklist worker crashed
    

    Note

    No checklist key element is required because the clear_checklist worker is a special case worker that does not return any information to add to the checklist (having just requested that it be cleared).

  2. Add a after_clear_checklist() function to your nowcast.next_workers module:

    def after_clear_checklist(msg, config, checklist):
        """Calculate the list of workers to launch after the clear_checklist worker
        ends.
    
        :arg msg: Nowcast system message.
        :type msg: :py:class:`collections.namedtuple`
    
        :arg config: :py:class:`dict`-like object that holds the nowcast system
                     configuration that is loaded from the system configuration
                     file.
        :type config: :py:class:`nemo_nowcast.config.Config`
    
        :arg dict checklist: System checklist: data structure containing the
                             present state of the nowcast system.
    
        :returns: Worker(s) to launch next
        :rtype: list
        """
        next_workers = {
            'crash': [],
            'failure': [],
            'success': [NextWorker('nemo_nowcast.workers.rotate_logs')],
        }
        return next_workers[msg.type]
    

    The recommended pattern is to launch the rotate_logs Worker upon success of the clear_checklist worker.

  3. Add a:

    NextWorker('nemo_nowcast.workers.clear_checklist')
    

    object to the list of next workers returned by the after_worker_name() function for the worker that you want the log file rotation operation to follow.