Metadata-Version: 2.1
Name: gerapy-proxy
Version: 0.0.2
Summary: Proxy Components for Scrapy & Gerapy
Home-page: https://github.com/Gerapy/GerapyProxy
Author: Germey
Author-email: cqc@cuiqingcai.com
License: MIT
Description: 
        # Gerapy Proxy
        
        This is a package for supporting proxy in Scrapy, also this
        package is a module in [Gerapy](https://github.com/Gerapy/Gerapy).
        
        ## Installation
        
        ```shell script
        pip3 install gerapy-proxy
        ```
        
        ## Usage
        
        If you have a proxy pool which can provide a random proxy for every request, you can use this package
        to integrate proxy into your Scrapy/Gerapy Project.
        
        For example, there is a [ProxyPool API](https://proxypool.scrape.center/random) which can return a random proxy 
        per time, we can configure `GERAPY_PROXY_POOL_URL` setting provided by this package to enable proxy for every Scrapy Request.
        
        To use this package, firstly install it by this command:
        
        ```shell script
        pip3 install gerapy-proxy
        ```
        
        Then enable it in DownloadMiddleware:
        
        ```python
        DOWNLOADER_MIDDLEWARES = {
            'gerapy_proxy.middlewares.ProxyPoolMiddleware': 543,
        }
        ```
        
        and add proxy url in settings:
        
        ```shell script
        GERAPY_PROXY_POOL_URL = 'https://proxypool.scrape.center/random'
        ```
        
        This ProxyPool is configured based on this [ProxyPool](https://github.com/Python3WebSpider/ProxyPool) repo, you can
        also build your own ProxyPool service.
        
        Now, you've finished it.
        
        The `ProxyPoolMiddleware` will firstly fetch a proxy from `GERAPY_PROXY_POOL_URL` and set `meta.proxy` attribute
        to Scrapy Reqeust.
        
        ## Configuration
        
        ### Basic Auth
        
        If your ProxyPool has Basic Auth, you can enable it by configuring these settings:
        
        ```shell script
        GERAPY_PROXY_POOL_AUTH = True
        GERAPY_PROXY_POOL_USERNAME = <username>
        GERAPY_PROXY_POOL_PASSWORD = <password>
        ```
        
        ### Min Retry Times
        
        If you want to enable Proxy depends on the retry times, you can configure this settings:
        
        ```shell script
        GERAPY_PROXY_POOL_MIN_RETRY_TIMES = 2
        ```
        
        Then proxy will only work if the retry times of Request greater or equal than 2.
        
        ### Random Enabled
        
        If you want to enable the proxy randomly, you can configure the probability of enabling it:
        
        ```shell script
        GERAPY_PROXY_POOL_RANDOM_ENABLE_RATE = 0.8
        ```
        
        Then probability of enabling the proxy is 80%, if you configure it to 1, proxy will always be enabled.
        
        ### Fetch Timeout
        
        You can also configure the max time of fetching proxy from ProxyPool:
        
        ```shell script
        GERAPY_PROXY_POOL_TIMEOUT = 5
        ```
        
        After configuring this, if Proxy Pool does not return result in 5s, proxy will not be used.
        
        ### ProxyPool Response Parser
        
        Your ProxyPool may not return the same format as [this](https://github.com/Python3WebSpider/ProxyPool) in plain text,
        you can also define a parser to extract proxy from your ProxyPool.
        
        For example, if your ProxyPool return this for every request:
        
        ```json
        {
          "host": "111.222.223.224",
          "port": 3128
        }
        ```
        
        You can define a method like:
        
        ```python
        import json
        def parse_result(text):
            data = json.loads(text)
            return f'{data.get("host")}:{data.get("port")}'
          
        GERAPY_PROXY_EXTRACT_FUNC = parse_result 
        ```
        
        Then you will get the proxy with correct format.
        
        ## Example
        
        For more detail, please see [example](./example).
        
        Also you can directly run with Docker:
        
        ```
        docker run germey/gerapy-proxy-example
        ```
        
        Outputs:
        
        ```shell script
        2020-07-13 01:49:13 [scrapy.utils.log] INFO: Scrapy 2.2.0 started (bot: example)
        2020-07-13 01:49:13 [scrapy.utils.log] INFO: Versions: lxml 4.3.3.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.7 (default, May  6 2020, 04:59:01) - [Clang 4.0.1 (tags/RELEASE_401/final)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Darwin-19.4.0-x86_64-i386-64bit
        2020-07-13 01:49:13 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
        2020-07-13 01:49:13 [scrapy.crawler] INFO: Overridden settings:
        {'BOT_NAME': 'example',
         'CONCURRENT_REQUESTS': 3,
         'NEWSPIDER_MODULE': 'example.spiders',
         'RETRY_HTTP_CODES': [403, 500, 502, 503, 504],
         'SPIDER_MODULES': ['example.spiders']}
        2020-07-13 01:49:13 [scrapy.extensions.telnet] INFO: Telnet Password: 83c276fb41754bd0
        2020-07-13 01:49:13 [scrapy.middleware] INFO: Enabled extensions:
        ['scrapy.extensions.corestats.CoreStats',
         'scrapy.extensions.telnet.TelnetConsole',
         'scrapy.extensions.memusage.MemoryUsage',
         'scrapy.extensions.logstats.LogStats']
        2020-07-13 01:49:13 [scrapy.middleware] INFO: Enabled downloader middlewares:
        ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
         'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
         'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
         'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
         'gerapy_pyppeteer.downloadermiddlewares.PyppeteerMiddleware',
         'scrapy.downloadermiddlewares.retry.RetryMiddleware',
         'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
         'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
         'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
         'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
         'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
         'scrapy.downloadermiddlewares.stats.DownloaderStats']
        2020-07-13 01:49:13 [scrapy.middleware] INFO: Enabled spider middlewares:
        ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
         'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
         'scrapy.spidermiddlewares.referer.RefererMiddleware',
         'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
         'scrapy.spidermiddlewares.depth.DepthMiddleware']
        2020-07-13 01:49:13 [scrapy.middleware] INFO: Enabled item pipelines:
        []
        2020-07-13 01:49:13 [scrapy.core.engine] INFO: Spider opened
        2020-07-13 01:49:13 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
        2020-07-13 01:49:13 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
        2020-07-13 01:49:13 [example.spiders.book] INFO: crawling https://dynamic5.scrape.center/page/1
        2020-07-13 01:49:13 [gerapy.pyppeteer] DEBUG: processing request <GET https://dynamic5.scrape.center/page/1>
        2020-07-13 01:49:13 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
        2020-07-13 01:49:14 [gerapy.pyppeteer] DEBUG: crawling https://dynamic5.scrape.center/page/1
        2020-07-13 01:49:19 [gerapy.pyppeteer] DEBUG: waiting for .item .name finished
        2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: wait for .item .name finished
        2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: close pyppeteer
        2020-07-13 01:49:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://dynamic5.scrape.center/page/1> (referer: None)
        2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: processing request <GET https://dynamic5.scrape.center/detail/26898909>
        2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: processing request <GET https://dynamic5.scrape.center/detail/26861389>
        2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: processing request <GET https://dynamic5.scrape.center/detail/26855315>
        2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
        2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
        2020-07-13 01:49:21 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
        2020-07-13 01:49:21 [gerapy.pyppeteer] DEBUG: crawling https://dynamic5.scrape.center/detail/26855315
        2020-07-13 01:49:21 [gerapy.pyppeteer] DEBUG: crawling https://dynamic5.scrape.center/detail/26861389
        2020-07-13 01:49:21 [gerapy.pyppeteer] DEBUG: crawling https://dynamic5.scrape.center/detail/26898909
        2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: waiting for .item .name finished
        2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: wait for .item .name finished
        2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: close pyppeteer
        2020-07-13 01:49:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://dynamic5.scrape.center/detail/26861389> (referer: https://dynamic5.scrape.center/page/1)
        2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: processing request <GET https://dynamic5.scrape.center/page/2>
        2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
        2020-07-13 01:49:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://dynamic5.scrape.center/detail/26861389>
        {'name': '壁穴ヘブンホール',
         'score': '5.6',
         'tags': ['BL漫画', '小基漫', 'BL', '『又腐又基』', 'BLコミック']}
        2020-07-13 01:49:25 [gerapy.pyppeteer] DEBUG: waiting for .item .name finished
        2020-07-13 01:49:25 [gerapy.pyppeteer] DEBUG: crawling https://dynamic5.scrape.center/page/2
        2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: wait for .item .name finished
        2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: close pyppeteer
        2020-07-13 01:49:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://dynamic5.scrape.center/detail/26855315> (referer: https://dynamic5.scrape.center/page/1)
        2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: processing request <GET https://dynamic5.scrape.center/detail/27047626>
        2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
        2020-07-13 01:49:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://dynamic5.scrape.center/detail/26855315>
        {'name': '冒险小虎队', 'score': '9.4', 'tags': ['冒险小虎队', '童年', '冒险', '推理', '小时候读的']}
        2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: waiting for .item .name finished
        2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: crawling https://dynamic5.scrape.center/detail/27047626
        2020-07-13 01:49:27 [gerapy.pyppeteer] DEBUG: wait for .item .name finished
        2020-07-13 01:49:27 [gerapy.pyppeteer] DEBUG: close pyppeteer
        ...
        ```
Platform: UNKNOWN
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.6.0
Description-Content-Type: text/markdown
