Metadata-Version: 2.1
Name: Torrelque
Version: 0.2.0
Summary: Asynchronous Redis-based reliable queue package
Home-page: https://heptapod.host/saajns/torrelque
Author: saaj
Author-email: mail@saaj.me
License: LGPL-3.0-only
Description: .. image:: https://badge.fury.io/py/Torrelque.png
          :target: https://pypi.python.org/pypi/Torrelque
        
        *********
        Torrelque
        *********
        Torrelque is a Python package that provides an asynchronous reliable Redis-backed queues.
        
        Without further ado it's easy to say that the package is an implementation of the queue
        described in this blog post [1]_ with some required changes and improvements. And here's a
        great overview presentation, Redis as a Reliable Work Queue [2]_, from the same engineers.
        
        Prior to version 0.2 Torrelque was Tornado-specific. Since version 5 Tornado runs on
        `asyncio` event loop by default, hence Torrelque can still be used with it.
        
        Install
        =======
        .. sourcecode:: bash
        
            pip install Torrelque
        
        Use
        ===
        .. sourcecode:: python
        
            #!/usr/bin/env python3.7
        
        
            import random
            import logging
            import asyncio
        
            import aredis
            from torrelque import Torrelque
        
        
            logger = logging.getLogger(__name__)
        
        
            async def produce():
                redis = aredis.StrictRedis(decode_responses=True)
                queue = Torrelque(redis)
                while True:
                    for _ in range(5):
                        task = {'value': random.randint(0, 99)}
                        logger.debug('Produced task %s', task)
                        await queue.enqueue(task)
                    await asyncio.sleep(10)
        
        
            async def process(task_data):
                logger.debug('Consmed task %s', task_data)
                await asyncio.sleep(1)
        
            async def consume():
                redis = aredis.StrictRedis(decode_responses=True)
                queue = Torrelque(redis)
                while True:
                    task_id, task_data = await queue.dequeue()
                    if not task_id:
                        continue
                    try:
                        await process(task_data)
                        await queue.release(task_id)
                    except Exception:
                        logger.exception('Job processing has failed')
                        await queue.requeue(task_id, delay = 30)
        
        
            async def main():
                for _ in range(4):
                    asyncio.create_task(consume())
        
                await produce()
        
        
            if __name__ == '__main__':
                logging.basicConfig(level = logging.DEBUG, format = '%(asctime)s %(message)s')
                asyncio.run(main())
        
        
        
        Prior art
        =========
        There's related, synchronous task queue package called *saferedisqueue* [3]_.
        
        
        .. [1] http://blog.bronto.com/engineering/reliable-queueing-in-redis-part-1/
        .. [2] https://www.percona.com/news-and-events/percona-university-smart-data-raleigh/using-redis-reliable-work-queue
        .. [3] https://pypi.org/project/saferedisqueue/
        
Keywords: python redis asynchronous reliable-queue work-queue
Platform: Any
Classifier: Topic :: Software Development :: Libraries
Classifier: Framework :: AsyncIO
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Intended Audience :: Developers
Provides-Extra: test
