Metadata-Version: 2.1
Name: TrollHunter
Version: 0.3.3
Summary: TrollHunter
Home-page: https://github.com/StanGirard/TrollHunter
License: GPL
Description: # TrollHunter
        
        TrollHunter is a Twitter Crawler & News Website Indexer.
        It aims at finding Troll Farmers & Fake News on Twitter.
         
        It composed of three parts:
        - Twint API to extract information about a tweet or a user
        - News Indexer which indexes all the articles of a website and extract its keywords
        - Analysis of the tweets and news
        
        ## Installation
        
        ### Docker
        
        TrollHunter requires many services to run
        - ELK ( Elastic Search, Logstash, Kibana)
        - InfluxDb & Grafana
        - RabbitMQ
        
        You can either launch them individually if you already have them setup or use our `docker-compose.yml`
        
        - Install Docker
        - Run `docker-compose up -d`
        
        Change the `.env` with the required values
        
        You can either run
        ```Bash
        pip3 install TrollHunter
        ```
        or clone the project and run 
        ```Bash
        pip3 install -r requirements.txt
        ```
        
        ## Twint API
        
        
        ## News Indexer
        
        The second main part of the project is the crawler and indexer of news.
        
        For this, we use the sitemap xml file of news websites to crawl all the articles. In a sitemap file, we extract the tag
        *sitemap* and *url*.
        
        The *sitemap* tag is a link to a child sitemap xml file for a specific category of articles in the website.
        
        The *url* tag represents an article/news of the website.  
        
        The root url of a sitemap is stored in a postgres database with a trust level of the website (Oriented, Verified,
        Fake News, ...) and headers. The headers are the tag we want to extract from the *url* tag which contains details about
        the article (title, keywords, publication date, ...).
        
        The headers are the list of fields use in the index pattern of ElasticSearch.
         
        In crawling sitemaps, we insert the new child sitemap in the database with the last modification date or update it for
        the ones already in the database. The last modification date is used to crawl only sitemaps which change since the
        last crawling.
        
        The data extracts from the *url* tags are built in a dataframe then sent in ElasticSearch for further utilisation with 
        the request in Twint API.
        
        In the same time, some sitemaps don't provide the keywords for their articles. Hence, from ElasticSearch we retrieve the
        entries without keywords. Then, we download the content of the article and extract the keywords thanks to NLP. Finally,
        we update the entries in ElasticSearch.
        
        #### Run
        For the crawler/indexer:
        
        ```python
        from TrollHunter.news_crawler import scheduler_news
        
        scheduler_news(time_interval)
        ```
        
        For updating keywords:
        ```python
        from TrollHunter.news_crawler import scheduler_keywords
        
        scheduler_keywords(time_interval, max_entry)
        ```
        
        Or see with the [main](https://github.com/StanGirard/TrollHunter/tree/master/docker/news_crawler) use with docker.  
        
        
        
        
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 5 - Production/Stable
Description-Content-Type: text/markdown
