Metadata-Version: 2.1
Name: pytest-retry
Version: 1.0.2
Summary: Adds the ability to retry flaky tests in CI environments
Author: str0zzapreti
License: MIT License
        
        Copyright (c) 2022 Silas
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/str0zzapreti/pytest-retry
Keywords: rerun,pytest,flaky
Classifier: License :: OSI Approved :: MIT License
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Framework :: Pytest
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: dev
License-File: LICENSE

![Tests](https://github.com/str0zzapreti/pytest-retry/actions/workflows/tests.yaml/badge.svg)
# pytest-retry

pytest-retry is a plugin for Pytest which adds the ability to retry flaky tests,
thereby improving the consistency of the test suite results. 

## Requirements

pytest-retry is designed for the latest versions of Python and Pytest. Python 3.9+
and pytest 7.0.0 are required. 

## Installation

Use pip to install pytest-retry:
```
$ pip install pytest-retry
```

## Usage

There are two main ways to use pytest-retry:

### 1. Command line

Run Pytest with the command line argument --retries in order to retry every test in 
the event of a failure. The following example will retry each failed up to two times
before proceeding to the next test:

```
$ python -m pytest --retries 2
```

An optional delay can be specified using the --retry-delay argument. This will insert
a fixed delay (in seconds) between each attempt when a test fails. This can be useful
if the test failures are due to intermittent environment issues which clear up after
a few seconds

```
$ python -m pytest --retries 2 --retry-delay 5
```

### 2. Pytest flaky mark

Mark individual tests as 'flaky' to retry them when they fail. If no command line
arguments are passed, only the marked tests will be retried. The default values
are 1 retry attempt with a 0-second delay

```
@pytest.mark.flaky
def test_unreliable_service():
    ...
```

The number of times each test will be retried and/or the delay can be manually
specified as well

```
@pytest.mark.flaky(retries=3, delay=1)
def test_unreliable_service():
    ...
```

The flaky mark will override any command line options if passed when running Pytest.

### Things to consider

- **Currently, failing test fixtures are not retried.** In the future, flaky test setup 
may be retried, although given the undesirability of flaky tests in general, flaky setup 
should be avoided at all costs. Any failures during teardown will immediately halt
further attempts so that they can be addressed immediately. Make sure your teardowns
always work reliably regardless of the number of retries when using this plugin

- When a flaky test is retried, the plugin runs teardown steps for the test as if it 
had passed. This is to ensure that any partial state created by the test is cleaned up 
before the next attempt so that subsequent attempts do not conflict with one another.
Class and module fixtures are included in this teardown with the assumption that false
test failures should be a rare occurrence and the performance hit from re-running 
these potentially expensive fixtures is worth it to ensure clean initial test state. 
With feedback, the option to not re-run class and module fixtures may be added, but 
in general, these types of fixtures should be avoided for known flaky tests.

- Flaky tests are not sustainable. This plugin is designed as an easy short-term
solution while a permanent fix is implemented. Use the reports generated by this plugin
to identify issues with the tests or testing environment and resolve them.

## Reporting

pytest-retry intercepts the standard Pytest report flow in order to retry tests and
update the reports as required. When a test is retried at least once, an R is printed
to the live test output and the counter of retried tests is incremented by 1. After
the test session has completed, an additional report is generated below the standard
output which lists all of the tests which were retried, along with the exceptions
that occurred during each failed attempt. 

```
plugins: retry-1.0.0
collected 1 item

test_retry_passes_after_temporary_test_failure.py R.                     [100%]

======================= the following tests were retried =======================

	test_eventually_passes failed on attempt 1! Retrying!
	Traceback (most recent call last):
	  File "tests/test_example.py", line 4, in test_eventually_passes
	    assert len(a) > 1
	AssertionError: assert 1 > 1
	 +  where 1 = len([1])

=========================== end of test retry report ===========================


========================= 1 passed, 1 retried in 0.01s =========================
```

Tests which have been retried but eventually pass are counted as both retried and
passed, and tests which have been retried but eventually fail are counted as both
retried and failed. Skipped, xfailed, and xpassed tests are never retried.
