Metadata-Version: 2.1
Name: pysigma-backend-loki
Version: 0.7.0
Summary: pySigma Loki backend
Home-page: https://github.com/grafana/pySigma-backend-loki
License: AGPL-3.0-only
Author: Nick Moore
Author-email: nicholas.moore@grafana.com
Requires-Python: >=3.8,<4.0
Classifier: License :: OSI Approved :: GNU Affero General Public License v3
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: pysigma (>=0.9.5,<0.10.0)
Requires-Dist: pysigma-pipeline-sysmon (>=1.0.1,<2.0.0)
Project-URL: Repository, https://github.com/grafana/pySigma-backend-loki
Description-Content-Type: text/markdown

![PyPI](https://img.shields.io/pypi/v/pysigma-backend-loki)
![Tests](https://github.com/grafana/pySigma-backend-loki/actions/workflows/test.yml/badge.svg)
[![Coverage Status](https://coveralls.io/repos/github/grafana/pySigma-backend-loki/badge.svg?branch=main&t=lvM1Ns)](https://coveralls.io/github/grafana/pySigma-backend-loki?branch=main)

# pySigma Loki Backend

This is the Loki backend for pySigma. It provides the package `sigma.backends.loki` with the `LogQLBackend` class.

It supports the following output formats:

* `default`: plain Loki LogQL queries
* `ruler`: creates Loki LogQL queries in the ruler (YAML) format for generating alerts

It includes the following pipeline transformations in `sigma.pipelines.loki`:

* `SetCustomAttributeTransformation`: adds a specified custom attribute to a rule, which can be used to introduce a [stream selector](https://grafana.com/docs/loki/latest/logql/log_queries/#log-stream-selector) or [parser expression](https://grafana.com/docs/loki/latest/logql/log_queries/#parser-expression) into the generated query
  * The `LokiCustomAttributes` enum contains the relevant custom attribute names used by the backend

Further, it contains the processing pipelines in `sigma.pipelines.loki`:

* `loki_log_parser`: converts field names to logfmt labels used by Grafana
* `loki_promtail_sysmon_message`: parse and adjust field names for Windows sysmon data produced by promtail
  * Note: most rules lack the `sysmon` service tag, and hence this pipeline should be used in combination with the [generic sysmon pipeline](https://github.com/SigmaHQ/pySigma-pipeline-sysmon)

This backend is currently maintained by:

* [Nick Moore](https://github.com/kelnage)
* [Mostafa Moradian](https://github.com/mostafa)

## Installation

To get started developing/testing pySigma-backend-loki, these steps may help you get started:

1. [Install poetry](https://python-poetry.org/docs/#installation)
2. Clone this repository and open a terminal/shell in the top-level directory
3. Run `poetry install` to install the Python dependencies
4. Run `poetry shell` to activate the poetry environment
5. Check it all works by running `poetry run pytest`
6. (Optional) If you wish to validate the generated rules using sigma\_backend\_tester.py, install
   [LogCLI](https://grafana.com/docs/loki/latest/tools/logcli/)

## Releasing

To release new versions of pySigma-backend-loki, we use GitHub actions to update PyPI. When the main branch is in state that is ready to release, the process is as follows:

1. Determine the correct version number using the [Semantic Versioning](https://semver.org/) methodology. All version numbers should be in the format `\d+\.\d+\.\d+(-[0-9A-Za-z-]+)?`
2. Update [pyproject.toml](https://github.com/grafana/pySigma-backend-loki/blob/main/pyproject.toml) with the new version number
3. Commit and push the change to GitHub, and validate that the GitHub actions tests pass
4. Create a signed tag for the release, named the version number prefixed with a v, e.g., `git tag --sign --message="Release vX.X.X" vX.X.X`
5. Push the tag to GitHub, e.g., `git push --tags`, and validate that the release to the test instance of PyPI is successful
6. Run `poetry build` to produce distributable versions in `dist/`
7. Create a release in GitHub against the appropriate tag. If the version number starts with `v0`, or ends with `-alpha/beta` etc., mark it as a pre-release, and attach the distributable files to the release
8. Validate that the release to PyPI GitHub action is successful

## Work in progress

These features are currently either WIP or are planned to be implemented in the near future.

* Various processing pipelines for other applications and log sources
* Generating more accurate log stream selectors based on logsource
* Translate field names in Sigma signatures into relevant labels for Loki using pipelines

## Won't implement (probably)

These features are not easily supported by the backend, and hence are unlikely to be implemented.

* More complex keyword/line filter searches than ANDs of ORs

