Metadata-Version: 2.1
Name: vdk-lineage
Version: 0.1.6
Summary: VDK Lineage plugin collects lineage (input -> job -> output) information and send it to pre-configured destination.
Home-page: https://github.com/vmware/versatile-data-kit
License: UNKNOWN
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Description-Content-Type: text/markdown

# VDK Lineage

VDK Lineage plugin provides lineage data (input data -> job -> output data) information and send it to pre-configured
destination.

At POC level currently. It collect lineage information for each job run and for each executed query.
Query execution is currently before it's executed (so not query status is logged).


## Usage

```
pip install vdk-lineage
```

And it will start collecting lineage from job and sql queries.

To send data using openlineage specify VDK_OPENLINEAGE_URL. For example:
```
export VDK_OPENLINEAGE_URL=http://localhost:5002
vdk marquez-server --start
vdk run some-job
# check UI for lineage
# stopping the server will delete any lineage data.
vdk marquez-server --stop
```

## Build and testing

In order to build and test a plugin go to the plugin directory and use `../build-plugin.sh` script to build it


