Metadata-Version: 2.1
Name: iocingestor
Version: 0.3.0
Summary: Extract and aggregate IOCs from threat feeds.
Home-page: https://github.com/ninoseki/iocingestor
License: GPL-2.0-or-later
Author: Manabu Niseki
Author-email: manabu.niseki@gmail.com
Requires-Python: >=3.6.1,<4.0.0
Classifier: License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Dist: PyYAML (>=5.3.1,<6.0.0)
Requires-Dist: aiocontextvars (>=0.2.2,<0.3.0)
Requires-Dist: async-exit-stack (>=1.0.1,<2.0.0)
Requires-Dist: async-generator (>=1.10,<2.0)
Requires-Dist: beautifulsoup4 (>=4.9.3,<5.0.0)
Requires-Dist: contextvars (>=2.4,<3.0)
Requires-Dist: environs (>=8.0.0,<9.0.0)
Requires-Dist: fastapi (>=0.61.1,<0.62.0)
Requires-Dist: fastapi-utils (>=0.2.1,<0.3.0)
Requires-Dist: feedparser (>=6.0.1,<7.0.0)
Requires-Dist: importlib-metadata (>=1.7.0,<2.0.0)
Requires-Dist: ioc-finder (>=5.0.0,<6.0.0)
Requires-Dist: iocextract (>=1.13.1,<2.0.0)
Requires-Dist: ipaddress (>=1.0.23,<2.0.0)
Requires-Dist: jsonpath-rw (>=1.4.0,<2.0.0)
Requires-Dist: loguru (>=0.5.3,<0.6.0)
Requires-Dist: poetry-version (>=0.1.5,<0.2.0)
Requires-Dist: pydantic (>=1.6.1,<2.0.0)
Requires-Dist: pymisp (>=2.4.131,<3.0.0)
Requires-Dist: pyparsing (>=2.4.7,<3.0.0)
Requires-Dist: requests (>=2.24.0,<3.0.0)
Requires-Dist: sgmllib3k (>=1.0.0,<2.0.0)
Requires-Dist: statsd (>=3.3.0,<4.0.0)
Requires-Dist: twitter (>=1.18.0,<2.0.0)
Requires-Dist: uvicorn (>=0.12.1,<0.13.0)
Project-URL: Repository, https://github.com/ninoseki/iocingestor
Description-Content-Type: text/markdown

# iocingestor

[![PyPI version](https://badge.fury.io/py/iocingestor.svg)](https://badge.fury.io/py/iocingestor)
[![Build Status](https://travis-ci.com/ninoseki/iocingestor.svg?branch=master)](https://travis-ci.com/ninoseki/iocingestor)
[![Coverage Status](https://coveralls.io/repos/github/ninoseki/iocingestor/badge.svg?branch=master)](https://coveralls.io/github/ninoseki/iocingestor?branch=master)
[![CodeFactor](https://www.codefactor.io/repository/github/ninoseki/iocingestor/badge)](https://www.codefactor.io/repository/github/ninoseki/iocingestor)

An extendable tool to extract and aggregate IoCs from threat feeds.

This tool is a forked version of [InQuest](https://inquest.net/)'s [ThreatIngestor](https://github.com/InQuest/ThreatIngestor) focuses on [MISP](https://www.misp-project.org/) integration.

## Key differences

- Better MISP integration.
  - Working with the latest version of MISP.
  - Smart event management based on `reference_link`.
- [MISP warninglist](https://github.com/MISP/misp-warninglists) compatible whitelisting.
- Using [ioc-finder](https://github.com/fhightower/ioc-finder) instead of [iocextract](https://github.com/InQuest/python-iocextract) for IoC extraction.
  - YARA rule extraction is dropped.

## Installation

iocingestor requires Python 3.6+.

Install iocingestor from PyPI:

```bash
pip install iocingestor
```

## Usage

Create a new `config.yml` file, and configure each source and operator module you want to use. (See `config.example.yml` as a reference.)

```bash
iocingestor config.yml
```

By default, it will run forever, polling each configured source every 15 minutes.

## Plugins

iocingestor uses a plugin architecture with "source" (input) and "operator" (output) plugins. The currently supported integrations are:

### Sources

- GitHub repository search
- RSS feeds
- Twitter
- Generic web pages

### Operators

- CSV files
- MISP
- SQLite database

