utils.logger

Pipeline logging setup: one UT-timestamped log file per invocation.

Implements the WMKO logging requirements (DRP-RUN-07/08/09/12): a user-configurable log directory, all logs under one parent directory, and per-invocation log files that concurrent pipeline instances never share.

Handlers are installed on the root logger so that module loggers (logging.getLogger(__name__)), the py.warnings bridge, and third-party libraries all reach the same file. Two sibling entry points install them, never at import time:

  • setup_logging – the single-recipe leaf runner (scripts/processing/reduce.py, the kpfpipe run entry) calls it once per reduction, writing that recipe’s per-unit log; its console echo defaults to stderr.

  • setup_batch_logging – the fan-out orchestrators (masters.py/science.py) call it once per invocation, writing a batch-summary log of the dispatch’s own decision points; its console echo is pinned to stdout so an operator can watch batch progress live. It is a thin wrapper over setup_logging.

Library code only ever calls logging.getLogger(__name__); with no handlers installed (e.g. recipes driven directly by tests) records are simply dropped.

kpfpipe.utils.logger.build_log_path(log_dir, recipe_name, target, start_time=None)

Build the unique per-invocation log path (does not create the file).

The layout is {log_dir}/{YYYYMMDD}/kpf_{recipe_name}_{target}_ {YYYYMMDDTHHMMSS}.log with both date components in UT. The date subdirectory keeps every log under the one configured parent directory (DRP-RUN-09) while the per-invocation filename records what ran and when.

Parameters:

  • log_dir (str) – The configured parent log directory (DRP-RUN-07).

  • recipe_name (str) – Short recipe identifier, e.g. ‘science’ or ‘masters’.

  • target (str) – The reduction target: obs_id, datecode, or ‘run’ when neither applies.

  • start_time (time.struct_time or None) – UT start time of the invocation; None means time.gmtime() now.

Returns:

The absolute log-file path.

Return type:

str

kpfpipe.utils.logger.get_level(name)

Map a level name (‘debug’ … ‘critical’, any case) to its logging int; raises ValueError on an unknown name.

kpfpipe.utils.logger.setup_batch_logging(log_dir, label, level='INFO', console=True)

Install per-invocation handlers for a batch driver.

Called by the masters/science orchestrators and the timeseries wrapper (label is the stage name). Sibling to setup_logging: same root-handler machinery and file layout, but for the fan-out drivers rather than one recipe. Writes {log_dir}/{YYYYMMDD}/kpf_{label}_batch_{stamp}.log, recording the batch’s own decision points – units dispatched, canary result, per-unit ok/failed, and the failure sentinels – alongside (not replacing) each unit’s per-reduction log. The console echo is pinned to sys.stdout so an operator can watch batch progress live (parity with the print()``s this replaces), while each record is also persisted to the batch log file. The stdout echo is filtered (``_BatchConsoleFilter) so that below WARNING only the driver’s own scripts.*/__main__ narration reaches the terminal – library INFO chatter is kept out of the live view but still written to the batch log file. Called once at the top of an orchestrator’s main(); setup_logging stays the leaf-only, per-recipe entry.

Parameters:

  • log_dir (str) – The configured parent log directory (DRP-RUN-07).

  • label (str) – Short orchestrator identifier, e.g. ‘masters’ or ‘science’.

  • level (str) – Logging level name; INFO is the production level.

  • console (bool) – Also mirror records to stdout via a StreamHandler.

Returns:

The absolute path of the created batch log file.

Return type:

str

kpfpipe.utils.logger.setup_logging(log_dir, recipe_name, target, level='INFO', console=True, stream=None, console_filter=None)

Install per-invocation file (+ optional console) handlers on root.

  • Tears down any handlers a previous setup_logging installed, so repeated calls never duplicate handlers.

  • Creates the {log_dir}/{YYYYMMDD}/ directory as needed.

  • Opens the log file with exclusive create; on a name collision (two instances starting the same second) it retries with a numeric suffix (.1, .2, …) so concurrent instances never share a file (DRP-RUN-12).

  • Formats records with UT timestamps (time.gmtime).

  • Sets the root logger level, pins chatty third-party loggers to WARNING, and calls logging.captureWarnings(True) so every warnings.warn lands in the log at WARNING with its file:lineno source identification (DRP-RUN-08).

Parameters:

  • log_dir (str) – The configured parent log directory (DRP-RUN-07).

  • recipe_name (str) – Short recipe identifier, e.g. ‘science’ or ‘masters’.

  • target (str) – The reduction target: obs_id, datecode, or ‘run’ when neither applies.

  • level (str) – Logging level name; INFO is the production level.

  • console (bool) – Also mirror records to a console via a StreamHandler.

  • stream (file-like or None) – Console destination when console is true; None means stderr (logging.StreamHandler’s default). The batch orchestrators pass sys.stdout so their live progress stays on stdout.

  • console_filter (logging.Filter or None) – Optional filter attached to the console handler only (never the file handler), to trim what the terminal echoes. setup_batch_logging passes one to quiet sub-WARNING library chatter on the batch console; None (the leaf default) echoes every record at the root level.

Returns:

The absolute path of the created log file.

Return type:

str

kpfpipe.utils.logger.teardown_logging()

Remove and close every handler setup_logging installed.

Also restores the default warnings.showwarning via logging.captureWarnings(False). Safe to call when nothing is installed. Primarily for tests; the CLI relies on process exit.