Metadata-Version: 2.1
Name: pramen-py
Version: 1.1.0
Summary: Pramen transformations written in python
Home-page: https://github.com/AbsaOSS/pramen
Keywords: paramen,pyspark,transformations,metastore
Author: Artem Zhukov
Author-email: iam@zhukovgreen.pro
Maintainer: Artem Zhukov
Maintainer-email: iam@zhukovgreen.pro
Requires-Python: >=3.6.8,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Requires-Dist: PyYAML (>=6.0,<7.0)
Requires-Dist: attrs (>=21.4.0,<22.0.0)
Requires-Dist: cattrs (==1.0.0); python_full_version >= "3.6.8" and python_version < "3.7"
Requires-Dist: cattrs (>=22.1.0,<23.0.0); python_version >= "3.7" and python_version < "4.0"
Requires-Dist: chispa (>=0.9.2,<0.10.0)
Requires-Dist: click (>=8.0.3,<9.0.0)
Requires-Dist: contextvars (>=2.4,<3.0)
Requires-Dist: environs (>=9.5.0,<10.0.0)
Requires-Dist: loguru (>=0.6.0,<0.7.0)
Requires-Dist: pyspark (==3.1.3)
Requires-Dist: pytest (==6.2.5)
Requires-Dist: pytest-asyncio (==0.16)
Requires-Dist: pytest-cov (==2.12.1)
Requires-Dist: pytest-mock (==3.6.1)
Requires-Dist: pytest-sugar (>=0.9.4,<0.10.0)
Requires-Dist: rich (>=11.1.0,<12.0.0)
Requires-Dist: types-PyYAML (>=6.0.4,<7.0.0)
Requires-Dist: typing-extensions (>=4.1.1,<5.0.0)
Project-URL: Repository, https://github.com/AbsaOSS/pramen
Description-Content-Type: text/markdown

# Pramen-py

Cli application for defining the data transformations for Pramen.

See:
```bash
pramen-py --help
```
for more information.


## Installation

### App settings

Application configuration solved by the environment variables
(see .env.example)

### Add pramen-py as a dependency to your project

In case of poetry:

```bash
# ensure we have valid poetry environment
ls pyproject.toml || poetry init

poetry add pramen-py
```
In case of pip:

```bash
pip install pramen-py
```


## Usage

## Application configuration

In order to configure the pramen-py options you need to set
corresponding environment variables. To see the list of available options run:

```bash
pramen-py list-configuration-options
```

### Developing transformations

pramen-py uses python's
[namespace packages](https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#native-namespace-packages)
for discovery of the transformations.

This mean, that in order to build a new transformer, it should be located
inside a python package with the `transformations` directory inside.

This directory should be declared as a package:
- for poetry
```toml
[tool.poetry]
# ...
packages = [
    { include = "transformations" },
]

```
- for setup.py
```python
from setuptools import setup, find_namespace_packages

setup(
    name='mynamespace-subpackage-a',
    # ...
    packages=find_namespace_packages(include=['transformations.*'])
)
```

Example files structure:
```
❯ tree .
.
├── README.md
├── poetry.lock
├── pyproject.toml
├── tests
│  └── test_identity_transformer.py
└── transformations
    └── identity_transformer
        ├── __init__.py
        └── example_config.yaml
```

In order to make transformer picked up by the pramen-py the following
conditions should be satisfied:
- python package containing the transformers should be installed to the
same python environment as pramen-py
- python package should have defined namespace package `transformations`
- transformers should extend `pramen_py.Transformation` base class

Subclasses created by extending Transformation base class are registered as
a cli command (pramen-py transformations run TransformationSubclassName)
with default options. Check:

```bash
pramen-py transformations run ExampleTransformation1 --help
```

for more details.

You can add your own cli options to your transformations. See example at
[ExampleTransformation2](transformations/example_trasformation_two/some_transformation.py)

### pramen-py pytest plugin

pramen-py also provides pytest plugin with helpful
fixtures to test created transformers.

List of available fixtures:
```bash
#install pramen-py into the environment and activate it
pytest --fixtures
# check under --- fixtures defined from pramen_py.test_utils.fixtures ---
```

pramen-py pytest plugin also loads environment variables from .env
file if it is presented in the root of the repo.

### Running and configuring transformations

Transformations can be run with the following command:
```bash
pramen-py transformations run \
  ExampleTransformation1 \
  --config config.yml \
  --info-date 2022-04-01
```

`--config` is required option for any transformation. See
[config_example.yaml](tests/resources/real_config.yaml) for more information.

To check available options and documentation for a particular transformation,
run:
```bash
pramen-py transformations run TransformationClassName --help
```
where TransformationClassName is the name of the transformation.

## Development

Prerequisites:
- <https://python-poetry.org/docs/#installation>
- python 3.6

Setup steps:

```bash
git clone https://github.com/AbsaOSS/pramen
cd pramen-py
make install  # create virtualenv and install dependencies
make test
make pre-commit

# enable completions
# source <(pramen-py completions zsh)
# source <(pramen-py completions bash)

pramen-py --help
```


### Load environment configuration

Before doing any development step, you have to set your development
environment variables

```bash
make install
```

## Completions

```bash
# enable completions
source <(pramen-py completions zsh)
# or for bash
# source <(pramen-py completions bash)
```


## Deployment

### From the local development environment

```bash
# bump the version
vim pyproject.toml

# deploy to the dev environment (included steps of building and publishing
#   artefacts)
cat .env.ci
make publish
```

