Metadata-Version: 2.1
Name: kq
Version: 2.1.0
Summary: Kafka Job Queue for Python
Home-page: https://github.com/joowani/kq
Author: Joohwan Oh
Author-email: joohwan.oh@outlook.com
License: MIT
Description: ## KQ: Kafka Job Queue for Python
        
        ![Build](https://github.com/joowani/kq/workflows/Build/badge.svg)
        ![CodeQL](https://github.com/joowani/kq/workflows/CodeQL/badge.svg)
        [![codecov](https://codecov.io/gh/joowani/kq/branch/master/graph/badge.svg?token=d6ooyuUCl6)](https://codecov.io/gh/joowani/kq)
        
        **KQ (Kafka Queue)** is a lightweight Python library which lets you queue and
        execute jobs asynchronously using [Apache Kafka](https://kafka.apache.org/). It is 
        built on top of [kafka-python](https://github.com/dpkp/kafka-python).
        
        ### Announcements
        
        * Python 3.5 will not be supported from kq v3.0.0.
        * See [releases](https://github.com/joowani/kq/releases) for latest updates.
        
        ### Requirements
        
        * [Apache Kafka](https://kafka.apache.org) 0.9+
        * Python 3.6+
        
        ### Installation
        
        Install using [pip](https://pip.pypa.io):
        
        ```shell
        pip install kq
        ```
        
        ### Usage
        
        Start your Kafka instance. 
        Example using [Kafka Docker](https://github.com/lensesio/fast-data-dev):
        
        ```shell
        docker run -p 9092:9092 -e ADV_HOST=127.0.0.1 lensesio/fast-data-dev
        ```
        
        Define your KQ ``worker.py`` module:
        
        ```python
        import logging
        
        from kafka import KafkaConsumer
        from kq import Worker
        
        # Set up logging.
        formatter = logging.Formatter('[%(levelname)s] %(message)s')
        stream_handler = logging.StreamHandler()
        stream_handler.setFormatter(formatter)
        logger = logging.getLogger('kq.worker')
        logger.setLevel(logging.DEBUG)
        logger.addHandler(stream_handler)
        
        # Set up a Kafka consumer.
        consumer = KafkaConsumer(
            bootstrap_servers='127.0.0.1:9092',
            group_id='group',
            auto_offset_reset='latest'
        )
        
        # Set up a worker.
        worker = Worker(topic='topic', consumer=consumer)
        worker.start()
        ```
        
        Start your worker:
        
        ```shell
        python my_worker.py
        [INFO] Starting Worker(hosts=127.0.0.1:9092 topic=topic, group=group) ...
        ```
        
        Enqueue a function call:
        
        ```python
        import requests
        
        from kafka import KafkaProducer
        from kq import Queue
        
        # Set up a Kafka producer.
        producer = KafkaProducer(bootstrap_servers='127.0.0.1:9092')
        
        # Set up a queue.
        queue = Queue(topic='topic', producer=producer)
        
        # Enqueue a function call.
        job = queue.enqueue(requests.get, 'https://google.com')
        
        # You can also specify the job timeout, Kafka message key and partition.
        job = queue.using(timeout=5, key=b'foo', partition=0).enqueue(requests.get, 'https://google.com')
        ```
        
        The worker executes the job in the background:
        
        ```shell
        python my_worker.py
        [INFO] Starting Worker(hosts=127.0.0.1:9092, topic=topic, group=group) ...
        [INFO] Processing Message(topic=topic, partition=0, offset=0) ...
        [INFO] Executing job c7bf2359: requests.api.get('https://www.google.com')
        [INFO] Job c7bf2359 returned: <Response [200]>
        ```
        
        See [documentation](https://kq.readthedocs.io) for more information.
        
        ### Contributing
        
        Set up dev environment:
        
        ```shell
        cd ~/your/kq/clone          # Activate venv if you have one
        pip install -e .[dev]       # Install dev dependencies (black, mypy, pre-commit etc.)
        pre-commit install          # Install git pre-commit hooks
        py.test                     # Run unit tests
        ```
        
        Run unit tests:
        ```shell
        docker run -p 9092:9092 -e ADV_HOST=127.0.0.1 lensesio/fast-data-dev  # Start Kafka docker
        py.test
        ```
        
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX
Classifier: Operating System :: MacOS
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Provides-Extra: dev
