Metadata-Version: 2.1
Name: lstosa
Version: 0.9.5
Summary: Onsite analysis pipeline for the CTA LST-1
Home-page: https://github.com/cta-observatory/lstosa
Author: lstosa developers, CTA-LST project
Author-email: dmorcuen@ucm.es
License: BSD 3-Clause License
Project-URL: Bug Tracker, https://github.com/cta-observatory/lstosa/issues
Project-URL: Documentation, https://lstosa.readthedocs.io/
Project-URL: Source Code, https://github.com/cta-observatory/lstosa
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: BSD License
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Provides-Extra: setup
Provides-Extra: test
Provides-Extra: docs
License-File: LICENSE

# lstosa

  [![ci](https://github.com/cta-observatory/lstosa/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/cta-observatory/lstosa/actions/workflows/ci.yml)
  [![Documentation Status](https://readthedocs.org/projects/lstosa/badge/?version=latest)](https://lstosa.readthedocs.io/en/latest/?badge=latest)
  [![coverage](https://codecov.io/gh/cta-observatory/lstosa/branch/main/graph/badge.svg?token=Zjk1U1ytaG)](https://codecov.io/gh/cta-observatory/lstosa)
  [![quality](https://app.codacy.com/project/badge/Grade/a8743a706e7c45fc989d5ebc4d61d54f)](https://www.codacy.com/gh/cta-observatory/lstosa/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=cta-observatory/lstosa&amp;utm_campaign=Badge_Grade)
  [![pypi](https://img.shields.io/pypi/v/lstosa)](https://pypi.org/project/lstosa/)
  [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6567234.svg)](https://doi.org/10.5281/zenodo.6567234)


Prototype onsite processing pipeline for the Large Size Telescope prototype (LST-1) of [CTA](https://www.cta-observatory.org/) (Cherenkov Telescope Array) based on [cta-lstchain](https://github.com/cta-observatory/cta-lstchain) running on the LST-1 IT onsite center at ORM (La Palma, Spain). It automatically carries out the next-day analysis of observed data using cron jobs, parallelizing the processing using the job scheduler SLURM. It provides data quality monitoring and tracking of analysis products' provenance.
Moreover, it also massively reprocesses the entire LST-1 dataset with each cta-lstchain major release:
 - Code: <https://github.com/cta-observatory/lstosa>
 - Docs: <https://lstosa.readthedocs.io/>
 - License: [BSD-3-Clause](https://github.com/cta-observatory/lstosa/blob/main/LICENSE)

## Install
We recommend using an isolated conda environment.
 - Install miniconda first.
 - Clone the repository, create and activate the conda environment using the `environment.yml` file:

    ```bash
    git clone https://github.com/cta-observatory/lstosa.git
    cd lstosa
    conda env create -n osa -f environment.yml
    conda activate osa
    ```

Then install `lstosa` as a **user** with: `pip install lstosa`, or as a **developer** with: `pip install -e .`.

In case you want to install the lstchain development version instead of a fixed tag, you can run inside the `osa` environment:

```bash
pip install git+https://github.com/cta-observatory/cta-lstchain
```

To update the environment (provided dependencies get updated), use:

```bash
conda env update -n osa -f environment.yml
```

> **Note** to developers: to enforce a unique code convention, please install pre-commit (pre-commit install) after cloning the repository and creating the conda environment. This will black the committed files automatically.

## Workflow management
`lstosa` workflow is handled daily by the `sequencer` script, which identifies which observations are to be processed, generates the analysis workflow, and submits the jobs. A first calibration job produces the daily calibration coefficients. Subsequently, data reconstruction jobs are scheduled on a subrun-wise basis (1 job corresponds to around 10 seconds of observed data, and its processing up to DL2 takes about 30-40 mins).

```mermaid

flowchart LR

    daq --> osa_seq
    osa_seq --> slurm --> osa_closer
    daq[DAQ]


    subgraph osa_seq [sequencer]
        direction TB
    A(Daily observation summary)
    B(Generate workflow)
    C(Submit jobs)
    A --> B --> C
    end

    subgraph slurm [SLURM parallel processing]
        direction TB
    H(Calibration sequence)
    I(Reconstruction sequences)
    H --> I
    end

    subgraph osa_closer [autocloser]
        direction TB
    D(Check job completion)
    E(Move files to final directories)
    F(Merge files)
    G(Parse provenance logs)
    D --> E --> F --> G
    end
```

## Usage
To use lstosa, you will first need to symlink some auxiliary files in a similar directory tree structure to the standard data production and set the paths correctly in your lstosa configuration file. Then to process all the runs from a given date, you can run the following command (use first the `--simulate` option to dry-run without actually submitting jobs):

```bash
sequencer --config your_osa_config.cfg --date YYYY-MM-DD LST1
```

Once all jobs finish, the `autocloser` script checks job completion, merges files, moves them to their final directories, and parses provenance logs.

```bash
autocloser --config your_osa_config.cfg --date YYYY-MM-DD LST1
```

## Dataflow

```mermaid
graph LR

    subgraph DAQ
    D1[R0]
    D2[DRS4 calib run]
    D3[Pedestal calib run]
    D4[Pointing log]
    end

    D2 --> C1
    D3 --> C2

    subgraph Calibration
    C1[DRS4 baseline correction]
    C2[Calibration charge coeffitiens]
    C1 --> C2
    end

    subgraph lstMCpipe
    M1[gamma DL2 MC]
    M2[RF models]
    end

    subgraph Sky-data reconstruction
    S1[DL1a]
    S2[DL1b]
    S3[muons]
    S4[DL1 check]
    S5[DL2]
    S6[DL3]
    S7[IRF]
    D1 & D4 & C1 & C2 --> S1
    S1 --> S3
    S1 --> S2
    S2 & S3 --> S4
    S2 ---> S5
    S5 --> S6
    S7 --> S6
    M2 --> S5
    M1 --> S7
    end

    subgraph High-level Gammapy
    DL4
    DL5
    S6 --> DL4
    DL4 --> DL5 --> ...
    end
```

> **Warning**: standard production of DL3 data and higher-level results is still under development.
