Metadata-Version: 2.1
Name: stdflow
Version: 0.0.66
Summary: Data flow tool that transform your notebooks and python files into pipeline steps by standardizing the data input / output. [for Data science project]
Home-page: https://github.com/CyprienRicque/stdflow
Author: Cyprien Ricque
Author-email: cyprien.ricque@gmail.com
License: Apache Software License 2.0
Keywords: nbdev jupyter notebook python "data science" data flow pipeline
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: dev
License-File: LICENSE

# stdflow

<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

Data flow tool that transform your notebooks and python files into
pipeline steps by standardizing the data input / output. (for data
science projects)

Create clean data flow pipelines just by replacing you `pd.read_csv()`
and `df.to_csv()` by `sf.load()` and `sf.save()`.

## Install

``` sh
pip install stdflow
```

## How to use

### Pipelines

``` python
from stdflow import StepRunner
from stdflow.pipeline import Pipeline

# Pipeline with 2 steps

dm = "../demo_project/notebooks/"

ingestion_ppl = Pipeline([
    StepRunner(dm + "01_ingestion/countries.ipynb"), 
    StepRunner(dm + "01_ingestion/world_happiness.ipynb")
])

# === OR ===
ingestion_ppl = Pipeline(
    StepRunner(dm + "01_ingestion/countries.ipynb"), 
    StepRunner(dm + "01_ingestion/world_happiness.ipynb")
)

# === OR ===
ingestion_ppl = Pipeline()
ingestion_ppl.add_step(StepRunner(dm + "01_ingestion/countries.ipynb"))
# OR
ingestion_ppl.add_step(dm + "01_ingestion/world_happiness.ipynb")


ingestion_ppl
```


    ================================
                PIPELINE            
    ================================

    STEP 1
        path: ../demo_project/notebooks/01_ingestion/countries.ipynb
        vars: {}

    STEP 2
        path: ../demo_project/notebooks/01_ingestion/world_happiness.ipynb
        vars: {}

    ================================

Run the pipeline

``` python
ingestion_ppl.run()
```

    =================================================================================
        01.                ../demo_project/notebooks/01_ingestion/countries.ipynb
    =================================================================================
    Variables: {}
        Path: countries.ipynb
        Duration: 0 days 00:00:00.743971
        Env: {}
    Notebook executed successfully.


    =================================================================================
        02.          ../demo_project/notebooks/01_ingestion/world_happiness.ipynb
    =================================================================================
    Variables: {}
        Path: world_happiness.ipynb
        Duration: 0 days 00:00:00.643122
        Env: {}
    Notebook executed successfully.

## Load and save data

### Option 1: Specify All Parameters

``` python
import stdflow as sf
import pandas as pd

# load data from ../demo_project/data/countries/step_loaded/v_202309212245/countries.csv
df = sf.load(
   root="../demo_project/data/",
   attrs=['countries'],
   step='loaded',
   version='202309212245',  # loads v_202309212245
   file_name='countries.csv',
   method=pd.read_csv  # or method='csv'
)

# export data to ./data/raw/twitter/france/step_processed/v_1/countries.csv
sf.save(
   df,
   root="../demo_project/data/",
   attrs='countries/',
   step='processed',
   version='new',  # creates v_new
   file_name='countries.csv',
   method=pd.DataFrame.to_csv,  # or method='csv'  or any function that takes the object to export as first input
)
```

    attrs=countries/::step_name=processed::version=new::file_name=countries.csv

Each time you perform a save, a metadata.json file is created in the
folder. This keeps track of how your data was created and other
information.

### Option 2: Use default variables

``` python
import stdflow as sf

# use package level default values
sf.root = "../demo_project/data/"
sf.attrs = 'countries'  # if needed use attrs_in and attrs_out
sf.step_in = 'loaded'
sf.step_out = 'processed'

df = sf.load()
# ! root / attrs / step : used from default values set above
# ! version : the last version was automatically used. default: ":last"
# ! file_name : the file, alone in the folder, was automatically found
# ! method : was automatically used from the file extension

sf.save(df)
# ! root / attrs / step : used from default values set above
# ! version: used default %Y%m%d%H%M format
# ! file_name: used from the input (because only one file)
# ! method : inferred from file name
```

    attrs=countries::step_name=processed::version=202309212303::file_name=countries.csv

Note that everything we did at package level can be done with the Step
class

``` python
from stdflow import Step

step = Step(
    root="../demo_project/data/",
    attrs='countries',
    step_in='loaded',
    step_out='processed'
)
# or set after
step.root = "../demo_project/data/"
# ...

df = step.load(version=':last', file_name=":auto", verbose=True)

step.save(df, verbose=True)
```

    INFO:stdflow.step:Loading data from ../demo_project/data/countries/step_loaded/v_202309212302/countries.csv
    INFO:stdflow.step:Data loaded from ../demo_project/data/countries/step_loaded/v_202309212302/countries.csv
    INFO:stdflow.step:Saving data to ../demo_project/data/countries/step_processed/v_202309212304/countries.csv
    INFO:stdflow.step:Data saved to ../demo_project/data/countries/step_processed/v_202309212304/countries.csv
    INFO:stdflow.step:Saving metadata to ../demo_project/data/countries/step_processed/v_202309212304/

    attrs=countries::step_name=processed::version=202309212304::file_name=countries.csv

## Do not

- Save in the same directory from different steps. Because this will
  erase metadata from the previous step.

## Data visualization

``` python
import stdflow as sf

step.save(df, verbose=True, export_viz_tool=True)
```

    INFO:stdflow.step:Saving data to ../demo_project/data/countries/step_processed/v_202309212304/countries.csv
    INFO:stdflow.step:Data saved to ../demo_project/data/countries/step_processed/v_202309212304/countries.csv
    INFO:stdflow.step:Saving metadata to ../demo_project/data/countries/step_processed/v_202309212304/
    INFO:stdflow.step:Exporting viz tool to ../demo_project/data/countries/step_processed/v_202309212304/

    attrs=countries::step_name=processed::version=202309212304::file_name=countries.csv

This command exports a folder `metadata_viz` in the same folder as the
data you exported. The metadata to display is saved in the metadata.json
file.

In order to display it you need to get both the file and the folder on
your local pc (download if you are working on a server)

Then go to the html file in your file explorer and open it. it should
open in your browser and lets you upload the metadata.json file.

Data flow tool that transform your notebooks and python files into
pipeline steps by standardizing the data input / output. (for data
science projects)

Create clean data flow pipelines just by replacing you `pd.read_csv()`
and `df.to_csv()` by `sf.load()` and `sf.save()`.

## Data Organization

### Format

Data folder organization is systematic and used by the function to load
and save. If follows this format:
root_data_folder/attrs_1/attrs_2/…/attrs_n/step_name/version/file_name

where:

- root_data_folder: is the path to the root of your data folder, and is
  not exported in the metadata
- attrs: information to classify your dataset (e.g. country, client, …)
- step_name: name of the step. always starts with `step_`
- version: version of the step. always starts with `v_`
- file_name: name of the file. can be anything

Each folder is the output of a step. It contains a metadata.json file
with information about all files in the folder and how it was generated.
It can also contain a html page (if you set `html_export=True` in
`save()`) that lets you visualize the pipeline and your metadata

## Best Practices:

- Do not use `sf.reset` as part of your final code
- In one step, export only to one path (except the version). meaning for
  one step only one combination of attrs and step_name
- Do not set sub-dirs within the export (i.e. version folder is the last
  depth). if you need similar operation for different datasets, create
  pipelines
