Metadata-Version: 2.1
Name: kfp-tekton
Version: 0.1.0
Summary: Tekton Compiler for Kubeflow Pipelines
Home-page: https://github.com/kubeflow/kfp-tekton/
Author: kubeflow.org
License: Apache 2.0
Description: <h1><a id="compiler-for-tekton">Compiler for Tekton</a></h1>
        
        The Kubeflow Pipelines [SDK](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/) 
        allows data scientists to define end-to-end machine learning and data pipelines.
        The output of the Kubeflow Pipelines SDK compiler is YAML for [Argo](https://github.com/argoproj/argo).
        We are extending the compiler of the Kubeflow Pipelines SDK to generate YAML
        for [Tekton](https://github.com/tektoncd/pipeline).
        
        
        <h2><a id="table-of-contents">Table of Contents</a></h2>
        
        <!-- START of ToC generated by running ./tools/mdtoc.sh sdk/README.md -->
        
          - [Project Prerequisites](#project-prerequisites)
          - [Tested Pipelines](#tested-pipelines)
          - [How to use the KFP-Tekton Compiler](#how-to-use-the-kfp-tekton-compiler)
            - [Installation](#installation)
            - [Compiling a Kubeflow Pipelines DSL script](#compiling-a-kubeflow-pipelines-dsl-script)
            - [Running the compiled pipeline on a Tekton cluster](#running-the-compiled-pipeline-on-a-tekton-cluster)
          - [Build Tekton from Master](#build-tekton-from-master)
          - [Additional Features](#additional-features)
            - [1. Compile Kubeflow Pipelines as a Tekton PipelineRun](#1-compile-kubeflow-pipelines-as-a-tekton-pipelinerun)
            - [2. Compile Kubeflow Pipelines with Artifacts Enabled](#2-compile-kubeflow-pipelines-with-artifacts-enabled)
          - [List of Available Features](#list-of-available-features)
          - [Troubleshooting](#troubleshooting)
        
        <!-- END of ToC generated by running ./tools/mdtoc.sh sdk/README.md -->
        
        
        <h2><a id="project-prerequisites">Project Prerequisites</a></h2>
        
         - Python: `3.7.5`
         - Kubeflow Pipelines: [`0.5.1`](https://github.com/kubeflow/pipelines/releases/tag/0.5.1)
         - Tekton: [`0.13.0`](https://github.com/tektoncd/pipeline/releases/tag/v0.13.0)
         - Tekton CLI: [`0.8.0`](https://github.com/tektoncd/cli/releases/tag/v0.8.0)
        
        Follow the instructions for [installing project prerequisites](https://github.com/kubeflow/kfp-tekton/blob/master/sdk/python/README.md#development-prerequisites)
        and take note of some important caveats.
        
        
        <h2><a id="tested-pipelines">Tested Pipelines</a></h2>
        
        We are [testing the compiler](https://github.com/kubeflow/kfp-tekton/blob/master/sdk/python/tests/README.md) on more than 80 pipelines
        found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compiler
        `testdata` folder, the KFP core samples and the samples contributed by third parties.
        
        A report card of Kubeflow Pipelines samples that are currently supported by the `kfp-tekton`
        compiler can be found [here](https://github.com/kubeflow/kfp-tekton/blob/master/sdk/python/tests/test_kfp_samples_report.txt).
        If you work on a PR that enables another of the missing features please ensure that
        your code changes are improving the number of successfully compiled KFP pipeline samples.
        
        
        <h2><a id="how-to-use-the-kfp-tekton-compiler">How to use the KFP-Tekton Compiler</a></h2>
        
        <h3><a id="installation">Installation</a></h3>
        
        You can install the latest release of the `kfp-tekton` compiler from
        [PyPi](https://pypi.org/project/kfp-tekton/). We recommend to create a Python
        virtual environment first:
        
            python3 -m venv .venv
            source .venv/bin/activate
            
            pip install kfp-tekton
            
        Alternatively you can install the latest version of the `kfp-tekton` compiler
        from source by cloning the repository [https://github.com/kubeflow/kfp-tekton](https://github.com/kubeflow/kfp-tekton):
        
        1. Clone the `kfp-tekton` repo:
        
           ```
           git clone https://github.com/kubeflow/kfp-tekton.git
           cd kfp-tekton
           ```
        
        2. Setup Python environment with Conda or a Python virtual environment:
        
           ```
           python3 -m venv .venv
           source .venv/bin/activate
           ```
        
        3. Build the compiler:
        
           ```
           pip install -e sdk/python
           ```
        
        4. Run the compiler tests (optional):
        
           ```
           make test
           ```
        
        
        <h3><a id="compiling-a-kubeflow-pipelines-dsl-script">Compiling a Kubeflow Pipelines DSL Script</a></h3>
        
        The `kfp-tekton` Python package comes with the `dsl-compile-tekton` command line
        executable, which should be available in your terminal shell environment after
        installing the `kfp-tekton` Python package.
        
        If you cloned the `kfp-tekton` project, you can find example pipelines in the
        `samples` folder or under `sdk/python/tests/compiler/testdata` folder.
        
            dsl-compile-tekton \
                --py sdk/python/tests/compiler/testdata/parallel_join.py \
                --output pipeline.yaml
        
        
        <h3><a id="running-the-pipeline-on-a-tekton-cluster">Running the Pipeline on a Tekton Cluster</a></h3>
        
        After compiling the `sdk/python/tests/compiler/testdata/parallel_join.py` DSL script
        in the step above, we need to deploy the generated Tekton YAML to our Kubernetes
        cluster with `kubectl` and start a pipeline run with `tkn`:
        
            kubectl apply -f pipeline.yaml
            tkn pipeline start parallel-pipeline --showlog
        
        A prompt should be asking for the pipeline arguments. Press `enter` and
        accept the defaults:
        
            ? Value for param `url1` of type `string`? (Default is `gs://ml-pipeline-playground/shakespeare1.txt`) gs://ml-pipeline-playground/shakespeare1.txt
            ? Value for param `url2` of type `string`? (Default is `gs://ml-pipeline-playground/shakespeare2.txt`) gs://ml-pipeline-playground/shakespeare2.txt
            
            Pipelinerun started: parallel-pipeline-run-th4x6
           
        Once the Tekton Pipeline is running, the logs should start streaming:
              
            Waiting for logs to be available...
            
            [gcs-download-2 : gcs-download-2] I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
            [gcs-download : gcs-download] With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
            [echo : echo] Text 1: With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
            [echo : echo] Text 2: I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
        
              
        <h2><a id="build-tekton-from-master">Build Tekton from Master</a></h2>
        
        In order to utilize the latest features and functions of the `kfp-tekton` compiler,
        we suggest to install Tekton from a nightly built or build it from the
        [master](https://github.com/tektoncd/pipeline/blob/master/DEVELOPMENT.md#install-pipeline)
        branch. Features that require a special build, different from the 'Tested Version',
        will be listed below.
        
        - [Exit Handler](https://github.com/kubeflow/kfp-tekton/blob/master/sdk/FEATURES.md#exit-handler)
        
        
        <h2><a id="additional-features">Additional Features</a></h2>
        
        <h3><a id="1-compile-kubeflow-pipelines-as-a-tekton-pipelinerun">1. Compile Kubeflow Pipelines as a Tekton PipelineRun</a></h3>
        
        By default, a Tekton [`PipelineRun`](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#overview)
        is generated by the `tkn` CLI so that users can interactively change their pipeline
        parameters during each execution. 
        However, `tkn` CLI is lagging several important features when generating a `PipelineRun`.
        Therefore, we added support for generating pipelineRun using `dsl-compile-tekton`
        with all the latest `kfp-tekton` compiler features. The comparison between Tekton
        pipeline and Argo workflow is described in our 
        [design docs](https://docs.google.com/document/d/1oXOdiItI4GbEe_qzyBmMAqfLBjfYX1nM94WHY3EPa94/edit#heading=h.f38y0bqkxo87).
        
        Compiling Kubeflow Pipelines into a Tekton `PipelineRun` is currently in the experimental
        stage. [Here](https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md) is
        the list of supported features in `PipelineRun`.
        
        As of today, the below `PipelineRun` features are available within `dsl-compile-tekton`:
         - Affinity
         - Node Selector
         - Tolerations
        
        To compile Kubeflow Pipelines as Tekton pipelineRun, add the `--generate-pipelinerun`
        parameter to the `dsl-compile-tekton` command:
        
            dsl-compile-tekton \
                --py sdk/python/tests/compiler/testdata/tolerations.py \
                --output pipeline.yaml \
                --generate-pipelinerun
        
        
        <h3><a id="2-compile-kubeflow-pipelines-with-artifacts-enabled">2. Compile Kubeflow Pipelines with Artifacts Enabled</a></h3>
        
        **Prerequisites**: Install [Kubeflow Pipelines](https://www.kubeflow.org/docs/pipelines/installation/).
        
        By default, _artifacts_ are disabled because they are dependent on Kubeflow Pipeline's
        [Minio](https://docs.minio.io/) storage. When artifacts are enabled, all the output
        parameters are also treated as artifacts and persisted to the default object storage.
        Enabling artifacts also allows files to be downloaded or stored as artifact inputs/outputs.
        Since artifacts are dependent on the Kubeflow Pipeline's deployment, the generated
        Tekton pipeline must be deployed to the same namespace as Kubeflow Pipelines.
        
        To compile Kubeflow Pipelines as a Tekton `PipelineRun`, add the `--enable-artifacts`
        argument to your `dsl-compile-tekton` commands. Then, run the pipeline in the same
        namespace that is used by Kubeflow Pipelines (typically `kubeflow`) by using the
        `-n` flag. e.g.:
        
            dsl-compile-tekton \
                --py sdk/python/tests/compiler/testdata/parallel_join.py \
                --output pipeline.yaml \
                --enable-artifacts
                
            kubectl apply -f pipeline.yaml -n kubeflow
            
            tkn pipeline start parallel-pipeline --showlog -n kubeflow
        
        
        You should see log messages saying the artifacts were stored in the object storage you specified:
        
            ? Value for param `url1` of type `string`? (Default is `gs://ml-pipeline-playground/shakespeare1.txt`) gs://ml-pipeline-playground/shakespeare1.txt
            ? Value for param `url2` of type `string`? (Default is `gs://ml-pipeline-playground/shakespeare2.txt`) gs://ml-pipeline-playground/shakespeare2.txt
            
            Pipelinerun started: parallel-pipeline-run-g87bs
            
            Waiting for logs to be available...
            [gcs-download : main] With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
            
            [gcs-download : copy-artifacts] Added `storage` successfully.
            [gcs-download : copy-artifacts] tekton/results/data
            [gcs-download : copy-artifacts] tar: removing leading '/' from member names
            [gcs-download : copy-artifacts] `data.tgz` -> `storage/mlpipeline/artifacts/parallel-pipeline-run/gcs-download/data.tgz`
            [gcs-download : copy-artifacts] Total: 0 B, Transferred: 194 B, Speed: 12.07 KiB/s
            
            [gcs-download-2 : main] I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
            
            [gcs-download-2 : copy-artifacts] Added `storage` successfully.
            [gcs-download-2 : copy-artifacts] tar: removing leading '/' from member names
            [gcs-download-2 : copy-artifacts] tekton/results/data
            [gcs-download-2 : copy-artifacts] `data.tgz` -> `storage/mlpipeline/artifacts/parallel-pipeline-run/gcs-download-2/data.tgz`
            [gcs-download-2 : copy-artifacts] Total: 0 B, Transferred: 204 B, Speed: 22.86 KiB/s
            
            [echo : main] Text 1: With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
            [echo : main]
            [echo : main] Text 2: I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
            [echo : main]
        
        
        <h2><a id="list-of-available-features">List of Available Features</a></h2>
        
        To understand how each feature is implemented and its current status, please visit
        the [FEATURES](https://github.com/kubeflow/kfp-tekton/blob/master/sdk/FEATURES.md) doc.
        
        
        <h2><a id="troubleshooting">Troubleshooting</a></h2>
        
        - When you encounter permission issues related to ServiceAccount, refer to
          [Servince Account and RBAC doc](https://github.com/kubeflow/kfp-tekton/blob/master/sdk/sa-and-rbac.md)
          
        - If you run into `bad interpreter: No such file or director` when trying to use
          python's venv, remove the current virtual environment in the `.venv` directory and
          create a new one using `virtualenv .venv`
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.5.3
Description-Content-Type: text/markdown
