Metadata-Version: 2.1
Name: rptf
Version: 0.0.7
Summary: End-to-end pipeline test framework for GitLab
Home-page: https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework
Author: Sri
Author-email: hello@srirangan.net
License: UNKNOWN
Description: ![ICON](https://assets.gitlab-static.net/uploads/-/system/project/avatar/22927059/icon-snek.png)
        
        # Remote Pipeline Test Framework
        
        End-to-end test framework for pipeline template authors. Trigger remote pipelines and make assertions.
        
        ## Why
        
        As a pipeline author, your template will be used across several projects under varying configurations. Your pipeline template will understand these different configurations and respond accordingly.
        
        The pipeline result might vary, the number of stages and jobs might change, artifacts may or may not be produced and environments may or may not be provisioned.
        
        Should you not setup proper tests for each of these scenarios? Should these tests not be executed on every change to the pipeline template?
        
        ![Snek is friend](https://i.imgur.com/4RtN1zw.jpg)
        
        ## Table of Contents
        
        1. [Quickstart](#quickstart)
        1. [Permissions](#permissions)
        1. [Yaml Specification](#yaml-specification)
        1. [Detailed Example](#detailed-example)
        1. [Test Output](#test-output)
        1. [CLI Options](#cli-options)
        1. [JUnit Reports](#junit-reports)
        1. [Install On Host without Docker](#install-on-host-without-docker)
        1. [Execution Stages](#execution-stages)
        
        ## Quickstart
        
        Test cases are defined in a Yaml file, which by default is called `rptf.yml`.
        
        ```yaml
        # rptf.yml
        
        pipeline_big:
            trigger:
                project_id: ...
            assertions:
                pipeline_status: success
                job_count: 25
        
        pipeline_small_fail:
            trigger:
                project_id: ...
            assertions:
                pipeline_status: failed
                job_count: 1
        ```
        
        The example above shows two test cases. Two test projects are configured such that their pipelines have `25` jobs and `1` job respectively. The second case expects a failed pipeline.
        
        We execute `rptf` in a GitLab pipeline, thus, a job to execute the tests is defined in `.gitlab-ci.yml`.
        
        ```yaml
        # .gitlab-ci.yml
        
        stages:
            - test
        
        rptf:
            stage: test
            image: registry.gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework/master
            script: rptf
            artifacts:
                reports:
                    junit: rptf-report.junit.xml
        ```
        
        Then a job is added in the`.gitlab-ci.yml`to execute these tests. It is common have several test-case projects across multiple groups / subgroups. The``rptf``tests can reside in the main project where the pipeline template is authored, although there is no hard requirement.
        
        `rptf` triggers pipelines (regardless of their location) as long as the token provided has sufficient permissions.
        
        ## Permissions
        
        The project containing the `rptf` job needs to provide `GITLAB_ACCESS_TOKEN`. This token must be authorized to execute pipelines across all projects defined in `rptf.yml`.
        
        A project hierarchy pattern that meets the above requirement is:
        ```
        - Project Template Group
            - CICD var {GITLAB_ACCESS_TOKEN}    # with relevant permissions
            - Template Project
                - .gitlab-ci.yml                # with rptf job
                - rptf.yml                      # with test case definitions
            - Test Cases Subgroup
                - Test Project 1
                - Test Project 2
                - Test Project 3
        ```
        
        ## Yaml Specification
        
        - `trigger` defines the target project where pipeline should be triggered
            - `project_id` required
            - `branch` defaults to `master`
        - `assertions` defines assertions to make after pipeline finishes execution
            - `pipeline_status` required, expected pipeline status
            - `job_count` expected number of jobs
            - `job_status` dict of expected job status
            - TODO `artifact_count` expected number of artifacts
            - TODO `environment_status` dict of expected environment status
        
        ## Detailed Example
        
        ```yaml
        two_jobs_two_artifacts:
            trigger:
                project_id: 11112222
            assertions:
                pipeline_status: success
                job_count: 2
                job_status:
                    artifact-one: success
                    artifact-two: success
                artifact_count: 2
                environment_status:
                    success: available
                    review_env_one: stopped
        ```
        
        ## CLI Options
        
        - `rptf --help ` shows help
        - `rptf --target FILE` path to test definitions, by default set to `rptf.yml`
        - `rptf --junit` or `rptf --no-junit` toggles whether JUnit test report must be produced, on by default
        
        ## Test Output
        
        ![](https://i.imgur.com/w516kEv.png)
        
        ## JUnit Reports
        
        By default, all test failures are reported in a JUnit compatible XML file called `rptf-report.junit.xml`.
        
        This can be integrated in the `rptf` job to enable deep integration with GitLab (or any other tool that supports JUnit reporting format).
        
        ```yaml
        # .gitlab-ci.yml
        
        stages:
            - test
        
        rptf:
            stage: test
            image: registry.gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework/master
            script: rptf
            artifacts:
                reports:
                    junit: rptf-report.junit.xml
        ```
        
        JUnit reporting can be turned off by passing the `rptf --no-junit` flag.
        
        ## Install without Docker
        
        TODO
        
        ## Execution Stages
        
        1. Show options selected
        1. Validate test specifications, exit if one or more test definitions invalid
        1. For each test case, trigger pipeline
        1. Wait and watch pipelines for completion
        1. On completion:
            1. Assert pipeline status
            1. Assert job count, if defined
            1. Assert job status, if defined
            1. TODO Assert artifact count, if defined
            1. TODO Assert environment status, if defined
        1. Show result
        1. Produce JUnit report, if enabled
        
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
