Metadata-Version: 2.1
Name: redispatcher
Version: 0.1.4
Summary: Toolset for asynchronous message processing backed by Redis as the message broker
Home-page: https://github.com/rafalstapinski/redispatcher
Keywords: redis,queue,broker,pub sub,asynchronous message
Author: Rafal Stapinski
Author-email: stapinskirafal@gmail.com
Requires-Python: >=3.8,<4.0
Classifier: Environment :: Web Environment
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Software Development :: Libraries
Requires-Dist: aioredis (>=1.3.1,<2.0.0)
Requires-Dist: isort (>=5.8.0,<6.0.0)
Requires-Dist: pydantic (>=1.8.1,<2.0.0)
Project-URL: Repository, https://github.com/rafalstapinski/redispatcher
Description-Content-Type: text/markdown

# redispatcher

redispatcher is a small library that allows you to specify a pool of workers that listen to messages added to queues in Redis. This allows you to execute long running background tasks asynchronously, like sending a welcome email after a user registers.

redispatcher relies on
* `aioredis` to publish to Redis queues and for your consumers to read from Redis
* `pydantic` to validate all messages and make sure they conform to the shape you specify

### You should try redispatcher if you
* Have a redis instance
* Have a web service that needs to process long running tasks in the background
* Don't want to deal with setting up Rabbit and cumbersome libraries like Celery


## Overview

redispatcher can be broken down into three (ish) parts.

#### Consumer
It all begins with a consumer. A consumer is just a class that defines the structure of the mssages it will be listening for and a function that implements the logic for processing that message.

#### Publishing
Every consumer you define will provide you with an easy `publish` method that you can use to queue up messages. Because we use Pydantic, it will validate and ensure that any messages you send/receive have to be formatted correctly. 

#### Consumer Pool
A consumer pool is a separate process that listens for all relevant messages queued up in Redis and dispatches them to the designated consumers to be processed.


## Install
```bash
$ pip install redispatcher
```

### Basic Consumer
```python
# my_consumer.py
from redispatcher import BaseConsumer

class MyConsumer(BaseConsumer):

    QUEUE = "my-queue-key"

    class Message(BaseConsumer.Message):
        email: str
        name: str
        registered: bool
    
    async def process_message(self, message: Message):
        print(f"processing message {message}")
        ...

```

### Running your consumers in a pool

#### Defining your pool
```python
# pool.py
from redispatcher import ConsumerPool, RedispatcherConfig, ConsumerConfig

from my_consumer import MyConsumer

config = RedispatcherConfig(
    redis_dsn="rediss://", # if not provided, will read from env
    consumers=[
        ConsumerConfig(
            consumer_class=MyConsumer
        )
    ]
)

if __name__ == "__main__":
    consumer_pool = ConsumerPool(config)
    consumer_pool.start() 
```

```bash
$ python pool.py
```

### Publishing messages to your pool
```python
# endpoint.py

from my_consumer import MyConsumer
from clients import my_aioredis_client

@app.post("/signup")
async def signup()
    # queue up work to send a welcome email while we continue with the rest of our endpoint logic
    await MyConsumer.publish(MyConsumer.Message(email=..., name=..., registered=True), my_aioredis_client)
```


### Advanced usage

We built redispatcher with a couple of handy utilities, but kept it as minimal as possible for your own consumers to be subclassed and implement any logging/tracing/etc logic yourself. 

Take a look at `examples/nicer_consumer.py` and `examples/example_publisher.py` for some examples of what's possible.


### Contributing

If you have a suggestion on how to improve redispatcher or experience a bug file an issue at <https://github.com/rafalstapinski/redispatcher/issues>.

If you want to contribute, open a PR at <https://github.com/rafalstapinski/redispatcher>.

PyPi: <https://pypi.org/project/redispatcher/>

