Metadata-Version: 2.1
Name: esque
Version: 0.5.0
Summary: esque - an operational kafka tool.
Home-page: https://github.com/real-digital/esque
License: MIT
Keywords: kafka,cli,commandline,administration,operation
Author: real.digital
Author-email: opensource@real-digital.de
Requires-Python: >=3.8,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Software Development
Classifier: Topic :: System :: Systems Administration
Requires-Dist: avro-python3 (>=1.9,<2.0)
Requires-Dist: click (==8.1.3)
Requires-Dist: confluent-kafka[avro] (>=1.6,<2.0)
Requires-Dist: fastavro (>=1.0,<2.0)
Requires-Dist: kafka-python (>=2.0.2,<3.0.0)
Requires-Dist: more-itertools (>=8.10.0,<9.0.0)
Requires-Dist: pendulum (>=2.0,<3.0)
Requires-Dist: pyyaml (>=5.1,<6.0)
Requires-Dist: requests (>=2.22,<3.0)
Requires-Dist: toml (>=0.10.0,<0.11.0)
Requires-Dist: yamale (>=2,<4)
Project-URL: Repository, https://github.com/real-digital/esque
Description-Content-Type: text/markdown

# esque - an operational Kafka tool

[![pypi Version](https://img.shields.io/pypi/v/esque.svg)](https://pypi.org/project/esque/)
[![Python Versions](https://img.shields.io/pypi/pyversions/esque.svg)](https://pypi.org/project/esque/)
![Build Status](https://github.com/real-digital/esque/workflows/Style,%20Unit%20And%20Integration%20Tests/badge.svg)
[![Coverage Status](https://coveralls.io/repos/github/real-digital/esque/badge.svg?branch=master)](https://coveralls.io/github/real-digital/esque?branch=master)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

In the Kafka world nothing is easy, but `esque` (pronounced *esk*) is an attempt at it.

`esque` is a user-centric command line interface for Kafka administration.

## Why should you care?

Some stuff is hard, and that is okay, but listing your kafka topics shouldn't be.

While adopting kafka at real.digital we noticed the immense entry barrier it poses to newcomers.
We can't recount how often we wrote Slack messages asking for the script to check the
status of topics or consumer groups. This is partly (but not only) due to a
fragmented and unclear definition of tooling and APIs for kafka.
In a wide array of administration tools, `esque` distances itself by striving to provide Kafka Ops for Humans, in a usable and natural way.

We feel that the goal of `esque` embodies the principle: “**keep easy things easy, and make hard things possible**”.

## Principles

* batteries included
* feature rich
* robust
* insightful
* by engineers for engineers

## Feature Overview

* Support for any type of Kafka deployment >1.2
* Display Resources (Topics, Consumer Groups, Brokers)
* Get detailed Overviews of Resources (Topics, Consumer Groups, Brokers)
* Create/Delete Topics
* Edit Topic Configurations
* Edit Consumer Offset for Topics
* SASL/SSL Support out of the box
* Consume and Produce to and from Avro and Plaintext Topics (including Avro Schema Resolution from Schema Registry)
* Context Switch (Easily Switch between pre-defined Clusters)
* Kafka Ping (Test roundtrip time to your kafka cluster)

## Command Overview

```bash
$ esque
Usage: esque [OPTIONS] COMMAND [ARGS]...

  esque - an operational kafka tool.

  In the Kafka world nothing is easy, but esque (pronounced esk) is an
  attempt at it.

Options:
  --version      Show the version and exit.
  -v, --verbose  Return stack trace on error.
  --no-verify    Skip all verification dialogs and answer them with yes.
  --help         Show this message and exit.

Commands:
  apply      Apply a set of topic configurations.
  config     Configuration-related options.
  consume    Consume messages from a topic.
  create     Create a new instance of a resource.
  ctx        List contexts and switch between them.
  delete     Delete a resource.
  describe   Get detailed information about a resource.
  edit       Edit a resource.
  get        Get a quick overview of different resources.
  io         Run a message pipeline.
  ping       Test the connection to the kafka cluster.
  produce    Produce messages to a topic.
  set        Set resource attributes.
  urlencode  Url-encode the given value.
```

## Installation and Usage

### Installation

`esque` is available at [pypi.org](https://pypi.org/project/esque/) and can be installed with `pip install esque`. `esque` requires Python 3.6+ to run.

#### Installation on Alpine Linux

There are no wheels for Alpine Linux, so `esque` requires a few extra dependencies to build them during installation.

```bash
apk add python3-dev py3-pip librdkafka librdkafka-dev g++
```

### Autocompletion

The autocompletion scripts for `bash` and `zsh` can be generated by running `esque config autocomplete`.

### Usage

#### Config Definition

When starting `esque` for the first time the following message will appear:

```bash
No config provided in ~/.esque
Should a sample file be created in ~/.esque [y/N]:
```

When answering with `y` `esque` will copy over the [sample config](https://github.com/real-digital/esque/blob/master/esque/config/sample_config.yaml) to `~/.esque/esque_config.yaml`.
Afterwards you can modify that file to fit your cluster definitions.

Alternatively might just provide a config file following the sample config's file in that path.

##### Config Example

```yaml
version: 1
current_context: local
contexts:
  # This context corresponds to a local development cluster
  # created by docker-compose when running esque from the host machine.
  local:
    bootstrap_servers:
      - localhost:9092
    security_protocol: PLAINTEXT
    schema_registry: http://localhost:8081
    default_values:
      num_partitions: 1
      replication_factor: 1
```

#### Config file for "apply" command

The config for the apply command has to be a yaml file and
is given with the option -f or --file.

In the current version only topic configurations can be
changed and specified.

It has to use the same schema, which is used
for the following example:

```yaml
topics:
  - name: topic_one
    replication_factor: 3
    num_partitions: 50
    config:
      cleanup.policy: compact
  - name: topic_two
    replication_factor: 3
    num_partitions: 50
    config:
      cleanup.policy: compact
```

## Development

To setup your development environment, make sure you have at least Python 3.6 & [poetry](https://github.com/sdispater/poetry) installed, then run

```bash
poetry install
poetry shell
```

### Pre Commit Hooks

To install pre commit hooks run:

```bash
pip install pre-commit
pre-commit install
pre-commit install-hooks
```

### Run tests

#### Integration Tests

esque comes with a docker-compose based kafka stack which you can start up with `make test-suite`.

You can then run the integration tests against this stack with `pytest tests/ --integration --local`.

Alternatively you can go the fast way and just run the whole stack + integration tests in docker:

```bash
make integration-test
```

#### Unit Tests

If you only want the unit tests, just run:

```bash
make test
```

## Alternatives

- [LinkedIn KafkaTools](https://github.com/linkedin/kafka-tools)
- [PyKafka Tools](https://github.com/Parsely/pykafka/blob/master/pykafka/cli/kafka_tools.py)
- [Official Kafka Scripts](https://github.com/apache/kafka/tree/trunk/bin)
- [kafkacat](https://github.com/edenhill/kafkacat)
- [kaf](https://github.com/birdayz/kaf)

