Create photometric catalogs

This notebooks demonstrates how to use the AGILE truth catalog in order to reduce the simulated images and create photometric catalogs using the LSST Science pipelines

Below, we provide helpful links and resources to further understand the LSST Science Pipelines and the butler:

The following notebook may be accessed here

[ ]:

import os

if not os.path.exists("src/lsst_inaf_agile"):
    os.chdir("../../")
    os.getcwd()
dirname = "data/tests/test_agile"

Make a butler repository

[ ]:

repo = f"{dirname}/repo"
print(f"Name of repository is '{repo}'")

[ ]:

%%file src/scripts/make_repo.sh
#!/bin/bash -ex
if [ $# -ne 1 ] ; then
    echo "Initialize a repository"
    echo "Usage: $0 [repo]"
    exit 1
fi

REPO=$1
REPO_ASSET_DIR=/beegfs/AGILE/Pleiadi_at_IRA/pleia15/lsst/repo_assets
CALIBS=${REPO_ASSET_DIR}/calib_imSim2.0_w_2023_04

butler create ${REPO}
butler register-instrument ${REPO} lsst.obs.lsst.LsstCam
butler import --export-file "${CALIBS}/export.yaml" \
       --skip-dimensions 'instrument,detector,physical_filter,band' \
       --transfer direct ${REPO} ${CALIBS}
butler write-curated-calibrations ${REPO} 'LSSTCam'
butler register-skymap --config-file ${REPO_ASSET_DIR}/DC2_skymap_config.py ${REPO}
cp ${REPO}/gen3.sqlite3 ${REPO}/gen3.sqlite3_orig

[ ]:

os.system(f"sh src/scripts/make_repo.sh {repo}")

Ingest files into the repository

[ ]:

%%file src/scripts/ingest.sh
#!/bin/bash -x

if [ $# -lt 2 ] ; then
    echo "Usage: $0 [repo] (raws)"
    exit 1
fi

repo=$1
shift
raws="$@"

butler ingest-raws --transfer direct ${repo} ${raws}
butler define-visits --collections LSSTCam/raw/all ${repo} LSSTCam
butler collection-chain ${repo} LSSTCam/defaults 'LSSTCam/raw/all,LSSTCam/calib,u/jchiang/calib_imSim2.0_w_2023_04,refcats,skymaps'

[ ]:

import glob

raws = " ".join(glob.glob(f"{dirname}/imsim/**/amp*.fits.fz", recursive=True))
print(raws)

[ ]:

os.system(f"sh src/scripts/ingest.sh {repo} {raws}")

Run a pipeline processing task

[ ]:

%%file src/scripts/pipetask.sh
#!/bin/sh -ex

###############################################################################
# Run an LSST pipeline task
###############################################################################

if [ $# -lt 2 ] ; then
    echo "Usage: $0 [repo] [task] (task) ..."
    exit 1
fi

# Handles input arguments
REPO=$1
shift
TASKS="$@"
FLAGS="-j 10 -b $REPO -i LSSTCam/defaults -o u/agile/output --register-dataset-types --clobber-outputs --skip-existing-in u/agile/output"

for TASK in $TASKS ; do
    FLAGS1=$FLAGS
    [ $TASK = "calibrate" ] || [ $TASK = "step1" ] && FLAGS1="$FLAGS -C calibrate:etc/cmodel.py"
    pipetask --log-level=DEBUG run $FLAGS1 -p "etc/my_DRP2.yaml#$TASK"
done
exit $?

[ ]:

# run step1 from etc/my_DRP2.yaml. See the file for the definition of this step.
os.system(f"sh src/scripts/pipetask.sh {repo} step1")

Examine at the output products

[ ]:

from lsst.daf.butler import Butler

butler = Butler(f"{repo}", collections="u/agile/output")
# do a butler query

A note on large-scale production

Butler repositories have a limitation of working only in single-threaded applications. To circumvent this, it is suggested that any large-scale production does not call pipetask and/or butler directly, but through some driver software instead. BPS is one solution for these, but others exist, as well. Refer to: https://pipelines.lsst.io/modules/lsst.ctrl.bps/quickstart.html and references therein.