utils.io

KPF data-file discovery: the FileHandler class plus product-path builders.

class kpfpipe.utils.io.FileHandler(data_dirs)

Bases: object

Discover KPF data files across the L0-input and masters-output trees.

Resolves the pipeline’s data-tree roots once and exposes file discovery as methods keyed by datecode/cal_type, so recipes and scripts never assemble data paths by hand.

Not thread-safe: build_mini_database stores the scanned night on self._mini_db and returns it, so concurrent calls on one instance race. Use one instance per thread (the on-disk cache is shared safely, keyed by datecode).

Parameters:

data_dirs (dict) – The already-extracted [DATA_DIRS] mapping, holding the roots KPF_DATA_INPUT (the L0 input tree) and KPF_MASTERS_OUTPUT (the masters output tree). Required: this is a util, not a pipeline module, so it has no config defaults to fall back on. Callers with a ConfigHandler pass config.get_params(["DATA_DIRS"]) (this class deliberately does not import ConfigHandler, keeping construction light). Either root may be absent – fine for an instance that only calls methods needing the other root; a method whose root is unset raises ValueError. Pass {} for an instance that touches neither root.

build_calibration_stacks(cal_type, *, min_stack_size=1, cluster_gap_seconds=7200, groupby='time_of_day', exclude_junk=True)

Return sorted file lists for all calibration stacks of the requested type, grouped from the mini database carried on the instance (self._mini_db, set by build_mini_database).

Drops observer-flagged junk frames (unless exclude_junk=False), filters by OBJECT, then groups the surviving frames into stacks according to groupby:

  • 'time_of_day' (default) – one stack per observing session, split wherever consecutive frames are more than cluster_gap_seconds apart (the morn/eve/night sessions). The split is purely temporal, so the morn/eve/night OBJECT suffix does not partition it and a mislabeled frame stacks with its true session. Used for bias and thar.

  • 'hst_day' – one stack per HST (Hawaii) calendar day.

  • 'obs_night' – one stack for the whole loaded night (the UTC-keyed datecode), spanning HST midnight. Used for darks, whose sparse sequences routinely straddle HST midnight and belong in a single nightly stack.

Every returned stack has at least min_stack_size files; undersized stacks are dropped, and it raises when none meets the threshold. Reads the mini database off the instance, so the recipe never handles the DataFrame itself.

Parameters:

  • cal_type (str) – Calibration frame type. One of ‘bias’, ‘dark’, ‘flat’, ‘thar’.

  • min_stack_size (int, default 1) – Minimum number of files required per stack; undersized stacks are dropped. The default of 1 keeps every cluster (a no-op filter); the masters recipe passes the configured per-cal-type value.

  • cluster_gap_seconds (int, default 7200) – Gap [s] between consecutive frames that splits one session from the next. The 2-hour default separates KPF morning vs. evening sessions. Applies only to groupby='time_of_day'; ignored otherwise.

  • groupby ({'time_of_day', 'hst_day', 'obs_night'}, default 'time_of_day') – How to group frames into stacks (see summary).

  • exclude_junk (bool, default True) – Drop junk frames (the ISJUNK column) before grouping. Rarely disabled.

Returns:

Sorted file lists, one per stack.

Return type:

list of list of str

Raises:

ValueError – If groupby or cal_type is not recognized, if no mini database is loaded on the instance, if no calibration frames of the requested type are found, or if no stack meets min_stack_size.

build_mini_database(datecode, cache=False)

Scan the PRIMARY header of every L0 FITS file for one observing night, cache the resulting DataFrame on the instance, and return it.

Reads {KPF_DATA_INPUT}/L0/{datecode}/*.fits and stores the result as self._mini_db, so a subsequent build_calibration_stacks needs only the calibration type; callers rarely need the return value directly.

The on-disk CSV cache lives at {KPF_DATA_INPUT}/vNext/mini_db/{datecode}_L0.csv. cache selects which side of it to use, read and write being independent:

  • read ("r") – load a current cache instead of re-scanning the L0 directory. A cache is current only when it passes the count and freshness guardrails in _read_mini_db_cache; a frame added, removed, replaced, or rewritten since the cache was built forces a rescan.

  • write ("w") – after scanning, (re)write the cache (directory created as needed) for next time. A cache hit on read short-circuits the scan, so nothing is rewritten.

Parameters:

  • datecode (str) – Observing-night datecode ‘YYYYMMDD’.

  • cache ({False, "r", "w", "rw", "wr"}, default False) – Which side(s) of the on-disk cache to use. False (default) always scans fresh and never touches the cache. "r" reads a current cache (else scans); "w" writes the scan result; "rw"/"wr" do both (the read-then-write behavior a plain cache once had).

Returns:

One row per readable frame. Columns: the header keys FILENAME (absolute path), TARGNAME, IMTYPE, OBJECT, EXPTIME, ELAPSED; plus derived UTC and HST (KPF-format timestamps from the filename, HST = UTC-10) and ISJUNK (frame is on the observer junk list – flagged, not dropped). A missing header key gives None for that column. An unreadable frame is warned and omitted from this DataFrame (useless for stacking), but is still recorded – with null header columns – in the on-disk cache, so the cache mirrors the L0 directory exactly.

Return type:

pandas.DataFrame

Raises:

ValueError – If cache is not one of the allowed modes, if KPF_DATA_INPUT is unset, or if the L0 directory holds no FITS files.

find_masters(cal_type, level, datecode)

Sorted list of the master files matching cal_type/level written under KPF_MASTERS_OUTPUT for datecode – everything matching {root}/masters/{datecode}/*_master_{cal_type}_{level}.fits (the KOAID prefix wildcarded).

The reader counterpart to a kpf_filepath master path; TestFindMasters.test_finds_kpf_filepath_output guards that the two stay in step.

Raises:

ValueError – If KPF_MASTERS_OUTPUT is unset.

kpfpipe.utils.io.datecode_dirs_in_range(root, start, end)

Sorted datecode subdirs of root within the inclusive [start, end] range.

Used by the masters orchestrator to expand a --date_range against the L0 tree. Non-datecode names and plain files are skipped; datecodes sort lexicographically, which coincides with chronological order for YYYYMMDD.

kpfpipe.utils.io.kpf_directory(kind, *, data_root, level=None, obs_id=None, datecode=None)

Output directory for a KPF product tree.

The single authority for the on-disk output layout; kpf_filepath and the recipes go through it rather than re-deriving os.path.join(data_root, ...) by hand. Pure path construction – it does not touch the filesystem.

The datecode comes from exactly one of obs_id (parsed) or datecode (given directly). QLP needs the full obs_id (it names a path component); science/masters only need the datecode, so either source works – masters in particular is often built from a bare datecode (e.g. the CLI --datecode, before any frame obs_id is in hand).

kind

directory

science

{data_root}/{level}/{datecode} – L0/L1/L2/L4 products

masters

{data_root}/masters/{datecode} (level unused)

QLP

{data_root}/QLP/{datecode}/{obs_id}/{level} – quicklook
Parameters:

  • kind (str) – Which output tree: ‘science’, ‘masters’, or ‘QLP’.

  • data_root (str) – Root data directory (e.g. ‘/data/kpf/’); a non-empty string.

  • level (str or None, optional) – Data level ‘L0’/’L1’/’L2’/’L4’. Required for science and QLP; unused for masters.

  • obs_id (str or None, optional) – Observation ID (e.g. ‘KP.20240405.49597.71’); the datecode source and, for QLP, a path component. Mutually exclusive with datecode.

  • datecode (str or None, optional) – Datecode ‘YYYYMMDD’ directly. Mutually exclusive with obs_id; not accepted for QLP (which needs the full obs_id).

Returns:

The directory path (no trailing separator).

Return type:

str

Raises:

ValueError – If data_root is not a non-empty string, kind is unrecognized, the obs_id/datecode source is missing/ambiguous/invalid, datecode is given for QLP, or level is missing/invalid for a kind that needs it.

kpfpipe.utils.io.kpf_filename(obs_id, level, *, master=None)

Base filename for a KPF data product (no directory). The single source of the naming rule; <model>.generate_standard_filename() and kpf_filepath both delegate here.

Science basenames by level:

L0: {obs_id}.fits (KPF-native) L1: kpf_L1_{YYYYMMDD}T{HHmmss}.fits (no EPRV “S”: no L1 standard) L2/L4: kpf_SL{N}_{YYYYMMDD}T{HHmmss}.fits (EPRV standard)

Master basename: {obs_id}_master_{master}_{level}.fits.

Parameters:

  • obs_id (str) – Observation ID (e.g. ‘KP.20240405.49597.71’). For master products this is the obs_id of the first frame in the stack.

  • level (str) – Data level ‘L0’/’L1’/’L2’/’L4’ (masters: ‘L1’/’L2’/’L4’).

  • master (str or None, optional) – Master calibration type (‘bias’/’dark’/’flat’/’thar’) for a master product, or None (the default) for a science product.

Returns:

The base filename.

Return type:

str

Raises:

ValueError – If obs_id is not a valid observation ID, level is unrecognized, or master is an unrecognized type or paired with an invalid level.

kpfpipe.utils.io.kpf_filepath(obs_id, level, *, data_root=None, master=None)

Build a filepath for a KPF data product: the product’s directory (kpf_directory) joined with its basename (kpf_filename).

The pipeline’s authoritative path builder: constructs the output path from an obs_id string (before/without a populated data object), as every recipe does when deciding where to write. Its basename twin, <model>.generate_standard_filename(), builds the same name from a populated object; TestFilenameConsistency enforces they agree per level.

Parameters:

  • obs_id (str) – Observation ID (e.g. ‘KP.20240405.49597.71’). For master products this should be the obs_id of the first frame in the stack.

  • level (str) – Data level string, one of ‘L0’, ‘L1’, ‘L2’, ‘L4’.

  • data_root (str or None, optional) – Root data directory (e.g. ‘/data/kpf/’). When None (the default), returns the bare filename (kpf_filename). Otherwise must be a non-empty string and a full path is returned.

  • master (str or None, optional) – Master calibration type, one of ‘bias’, ‘dark’, ‘flat’, ‘thar’. If provided, builds a master calibration path. If omitted, builds a science data path.

Returns:

Filepath (full path if data_root is set, bare filename if data_root is None).

Return type:

str

Raises:

ValueError – If level is unrecognized, if obs_id is not a valid observation ID, if master type is unrecognized, or if data_root is not None and not a non-empty string.

kpfpipe.utils.io.load_junk_obs_ids(data_input)

Set of observer-flagged junk obs_ids, to be stored with L0 data at {data_input}/vNext/reference/junk_obs.csv (a title line, an observation_id header, then one obs_id per row). An absent file yields the empty set, so exclusion becomes a no-op.

kpfpipe.utils.io.masters_stack_subdir(masters, kind, level)

Subdirectory of a masters night dir holding the per-frame stacked inputs (and diagnostics) for a kind master at level – e.g. thar/L2 gives {masters}/thar_L2 for the WLS per-frame L2s. Single source of the name, shared by wls.py (writer) and reduce.py (stale-clear); future kinds: flat_L2, thar_L4, and the other wls inputs (lfc, etalon).

kpfpipe.utils.io.read_token_file(path)

Read whitespace-stripped, non-blank lines from a text file.

Backs the --dates (masters) and --obs_ids (science) flags, which each accept a reference file listing one unit – a datecode or an obs_id – per line instead of inline values. Whitespace is stripped from each line and blank lines are dropped; validating that each token is a real datecode/obs_id is left to the caller.