Metadata-Version: 2.1
Name: kerground
Version: 0.1.0
Summary: Stupid simple background worker based on python.
Home-page: https://github.com/ClimenteA/kerground
Author: Climente Alin
Author-email: climente.alin@gmail.com
License: MIT
Platform: UNKNOWN
Description-Content-Type: text/markdown
License-File: LICENSE

# kerground 
[![Downloads](https://pepy.tech/badge/kerground)](https://pepy.tech/project/kerground) [![PyPI](https://img.shields.io/pypi/v/kerground?color=blue)](https://pypi.org/project/kerground/)


Stupid simple background worker based on python. `Kerground` is super basic - no need for Redis/RabbitMQ. It does not utilize to much RAM because events are saved in plain old json files.


## Quickstart

Install

```py
pip install kerground
```

In your app folder create a new package called `dependencies` (you can add this in `utils` or whatever you consider fit otherwise)

```py
#app/dependencies/kerground.py
from kerground import Kerground

ker = Kerground()
```
You can set on `Kerground` the following params:
- `tasks_path` - path where the `events` will be saved by default in "./.kergroundtasks";
- `pool` - wait in seconds for pending tasks;

Next `register` your background workers like:
```py
#some_module_.py
from app.dependencies import ker

@ker.register(ker.MODE.THREAD, max_retries=3)
def convert_files(event: list[str]):
    pass # some heavy duty stuff here
```
#### **The `event` must be json serializable!** 

There are 3 mode available:
- `ker.MODE.THREAD` - distribute events with `threading.Thread` if you have urls to wait;
- `ker.MODE.PROCESS` - (**default**) distribute events with `multiprocessing.Process` if you have some CPU intensive tasks;
- `ker.MODE.SYNC` - distribute events one by one for the func to process;

By default `max_retries` is `0` you can increase this number if you need to get data from some urls and there is a posibility they will fail.

Now you can send an event to background worker (kerground) like:
```py
#some_other_module_possible_route_handler.py
from app.dependencies import ker

def send_files_for_conversion(files: List[UploadFile]):
    filepaths = [file.filename for file in files]
    msgid = ker.enqueue("convert_files", filepaths)
    return f"Files were sent for conversion with id: {msgid}"
```
Pass to `ker.enqueue` the function name you want to call in background along with the json parsable *args and **kwargs. Function `ker.enqueue` will return an id which you can later inspect for it's status with `ker.check_status(msgid)`.

Prepare the `worker.py` file:
```py
# ./worker.py
from app.dependencies import ker

if __name__ == "__main__":
    ker.listen()
```

You can check the `example` folder which was used for tests.

## Multiple workers?

Just start multiple instances of `worker.py`. You can use [`honcho`](https://honcho.readthedocs.io/en/latest/) to make this easier.
```
# Procfile
web: python main.py
worker1: python worker.py
worker2: python worker.py
etc
```
Then:
```
honcho start
```

**Submit any questions/issues you have! Fell free to fork it and improve it!**


# Roadmap 

- Kerground Dashboard - to show running/pending/failed/done tasks, retrigger running a failed task + maybe some nice vizualizations;
- Benchmarks with Celery;



