Example of processing data from a single night

The part of the pipeline that generates stacks for master calibration files is not yet optimized to run on personal computers due to extremely high RAM requirements. It is easiest to download a directory of master calibration files for the night that you would like to process. The master calibration directory should be named by it’s date (YYYYMMDD) and placed in a subdirectory called masters within the KPFPIPE_DATA directory (e.g. $KPFPIPE_DATA/masters/20230527/).

Once you have the master calibration files in place you’ll need to edit the following variables in configs/kpf_drp_local.cfg:

masterbias # path to a master bias L0 file (e.g. ``$KPFPIPE_DATA/masters/20230527/kpf_20230527_master_bias.fits``)
masterdark # path to a master dark L0 file (e.g. ``$KPFPIPE_DATA/masters/20230527/kpf_20230527_master_dark.fits``)
masterflat # path to a master flat L0 file used for the flat field correction (e.g. ``$KPFPIPE_DATA/masters/20230527/kpf_20230527_master_flat.fits``)
flat_file  # path to a master flat L0 file used to find the order locations (e.g. ``$KPFPIPE_DATA/masters/20230527/kpf_20230527_master_flat.fits``)
wls_file   # path to a wavelength solution L1 file (e.g. ``$KPFPIPE_DATA/masters/20230527/kpf_20230527_master_WLS_cal-LFC-eve_L1.fits``)

Now download the data from a given night and place it under $KPFPIPE_DATA/L0/ (e.g. $KPFPIPE_DATA/L0/20230527/).

For single-threaded processing you can launch the pipeline with:

kpf --date 20230527 -r recipes/kpf_drp.recipe -c configs/kpf_drp_local.cfg

or for parallel processing use a combination of the --watch, --ncpus, and --reprocess flags:

kpf --reprocess --watch /data/L0/20230527/ --ncpus=8 -r recipes/kpf_drp.recipe -c configs/kpf_drp_local.cfg

In this mode the pipeline will continue to monitor the input directory for new incoming files forever so you’ll need to exit with Ctrl-C once you see that there are no more messages being reported.

In both single-threaded and parallel modes the outputs will be saved under the appropriate date directores in $KPFPIPE_DATA/2D/, $KPFPIPE_DATA/L1/, and $KPFPIPE_DATA/L2/.