Metadata-Version: 2.1
Name: rptf
Version: 0.1.12
Summary: End-to-end pipeline test framework for GitLab
Home-page: https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework
Author: Sri
Author-email: hello@srirangan.net
License: UNKNOWN
Description: ![ICON](https://assets.gitlab-static.net/uploads/-/system/project/avatar/22927059/icon-snek.png)
        
        # Remote Pipeline Test Framework
        
        End-to-end test framework for GitLab pipeline template authors.
        
        Trigger pipelines, make assertions.
        
        ## Why
        
        As a GitLab pipeline author, your pipeline and jobs will be used across several projects under varying configurations. Your pipeline template understands the context (branches, tags, vars, source files etc.) and responds accordingly.
        
        The resultant pipeline's status might vary, the number of stages and jobs might change, statuses of the jobs might vary, artifacts may or may not be produced and environments may or may not be provisioned etc.
        
        Should you not setup proper tests for each of these scenarios? Should these tests not be executed on every change of the pipeline template?
        
        Thus, `rptf`.
        
        ## Table of Contents
        
        1. [Quickstart](#quickstart)
        1. [Using RPTF with GitLab](#using-rptf-with-gitlab)
        1. [Setup GitLab Access Token](#setup-gitlab-access-token)
        1. [Usage With Self-Managed GitLab](#usage-with-self-managed-gitlab)
        1. [Authoring Tests](#authoring-tests)
        1. [Detailed Example](#detailed-example)
        1. [Test Output](#test-output)
        1. [CLI Options](#cli-options)
        1. [JUnit Reports](#junit-reports)
        1. [License](#license)
        1. [Contributing](#contributing)
        1. [Who Uses RPTF](#who-uses-rptf)
        
        ## Quickstart
        
        Being authored in Python, `rptf` is distributed via PyPi: https://pypi.org/project/rptf/
        
        Install `rptf` locally with `pip`.
        
        ```bash
        pip install rptf
        ```
        
        This makes `rptf` available for local execution:
        
        ```bash
        rptf --help
        ```
        
        Test cases are defined in a Yaml file, which by default is called `rptf.yml`.
        
        ```yaml
        # rptf.yml
        
        should_succeed_with_25_jobs:
            trigger:
                project_id: ...
            assertions:
                pipeline_status: success
                job_count: 25
        
        should_fail_with_1_job:
            trigger:
                project_id: ...
            assertions:
                pipeline_status: failed
                job_count: 1
        
        should_fail_with_3_jobs_deploy_skipped:
            trigger:
                project_id: ...
            assertions:
                pipeline_status: failed
                job_count: 3
                job_status:
                    build: success
                    test: failed
                    deploy: skipped
        ```
        
        Each test case contains two sections: `trigger` and `assertions`. The example above shows two test project's pipelines being triggered and asserts being make expect the pipelines have `25` jobs and `1` job respectively.
        
        Pipeline status is also asserted, and the second test case expects a failed pipeline.
        
        ## Using RPTF with GitLab
        
        We can execute `rptf` within a GitLab pipeline, thus, you have a primary test project which contains the `rptf.yml` file and includes a `.gitlab-ci.yml` definition that triggers pipelines in other GitLab projects and makes assertions.
        
        ```yaml
        # .gitlab-ci.yml
        
        stages:
            - test
        
        rptf:
            stage: test
            image: python:3
            before_script:
                - pip install rptf
            script:
                - rptf
        ```
        
        The `rptf` job above triggers pipelines defined in `rptf.yml` as long as the `GITLAB_ACCESS_TOKEN` provided has sufficient permissions. It's advisable to set the token at the project or group level via `GitLab -> Settings -> CICD`.
        
        ## Setup GitLab Access Token
        
        The project containing the `rptf` job needs to provide `GITLAB_ACCESS_TOKEN`. This token must be authorized to execute pipelines across all projects defined in `rptf.yml`.
        
        ## Usage With Self-Managed GitLab
        
        Environment variable `GITLAB_API_URL` is read to determine the GitLab instance where the pipelines are created.
        
        You can set this to get `rptf` to work with your self-managed GitLab instance.
        
        ```shell
        # set GITLAB_API_URL
        export GITLAB_API_URL=https://path-to-your-instance/api/v4
        
        # execute rptf 
        rptf
        ```
        
        ## Authoring Tests
        
        The Yaml specification declares the following attributes that can be used to author test cases:
        
        - `trigger` defines the target project where pipeline should be triggered
            - `project_id` required
            - `ref` defaults to `master`
            - `branch` deprecated in favor of `ref`
            - `variables` defaults to an emppty list `[]'
        - `assertions` defines assertions to make after pipeline finishes execution
            - `pipeline_status` required, expected pipeline status
            - `job_count` expected number of jobs
            - `job_status` dict of expected job status
            - TODO `artifact_count` expected number of artifacts
            - TODO `environment_status` dict of expected environment status
        
        ## Detailed Example
        
        ```yaml
        two_jobs_two_artifacts:
            trigger:
                project_id: 11112222
                ref: feature-branch-abc
                variables:
                    hello: world
                    foo: bar
            assertions:
                pipeline_status: success
                job_count: 2
                job_status:
                    artifact-one: success
                    artifact-two: success
                artifact_count: 2               # TODO
                environment_status:             # TODO
                    success: available          # TODO
                    review_env_one: stopped     # TODO
        ```
        
        ## CLI Options
        
        ```bash
        # help
        rptf --help             # help
        
        # run tests
        rptf                    # by default, executes rptf.yml
        rptf --target FILE      # path to test definition yaml
        
        # toggle JUnit test reporting
        rptf                    # generates JUnit report
        rptf --junit            # generates JUnit report
        rptf --no-junit         # JUnit reporting skipped
        ```
        
        ## Test Output
        
        [![asciicast](https://asciinema.org/a/407815.svg)](https://asciinema.org/a/407815)
        
        ![](https://i.imgur.com/5xm7lKu.png)
        
        ![](https://i.imgur.com/JpZ8RhZ.png)
        
        ## JUnit Reports
        
        By default, all test failures are reported in a JUnit compatible XML file called `rptf-report.junit.xml`.
        
        This can be integrated in the `rptf` job to enable deep integration with GitLab (or any other tool that supports JUnit reporting format).
        
        ```yaml
        # .gitlab-ci.yml
        
        stages:
            - test
        
        rptf:
            stage: test
            image: python:3
            before_script:
                - pip install rptf
            script:
                - rptf
            artifacts:
                reports:
                    junit: rptf-report.junit.xml
        ```
        
        Example of RPTF has failing tests integrated back into a GitLab Merge Request:
        - https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework/-/merge_requests/1
        
        JUnit reporting can be turned off by passing the `rptf --no-junit` flag.
        
        ## License
        
        [MIT](./license)
        
        ## Contributing
        
        Contributions welcome [here](https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/) in the form of feature requests, feature implementations, documentation improvements, examples etc.
        
        ```shell
        # execute locally
        pipenv install
        pipenv run python -m rptf.rptf
        
        ```
        
        ## Who Uses RPTF
        
        In-exhaustive list of RPTF users:
        
        - GitLab Five Minute Production App
        - LeapMotion
        - APNIC
        - Brainbits
        - Everbridge
        - Bolt
        
        [Add](https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework/-/edit/master/readme.md) your organization.
        
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
