Metadata-Version: 2.1
Name: pipelinewise-tap-mongodb
Version: 1.0.0
Summary: Singer.io tap for extracting data from MongoDB - Pipelinewise compatible
Home-page: https://github.com/transferwise/pipelinewise-tap-mongodb
Author: TransferWise
License: UNKNOWN
Description: # pipelinewise-tap-mongodb
        
        This is a [Singer](https://singer.io) tap that produces JSON-formatted data following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md) from a MongoDB source.
        
        ## Set up local dev environment:
        
        ```shell script
        make setup
        ```
        
        ## Activate virtual environment
        
        ```shell script
        . venv/bin/activate
        ```
        
        ## Set up Config file
        
        Create json file called `config.json`, with the following contents: 
        ```json
        {
          "password": "<password>",
          "user": "<username>",
          "host": "<host ip address>",
          "port": "<port>",
          "auth_database": "<database name to authenticate on>",
          "database": "<database name to sync from>"
        }
        ```
        The following parameters are optional for your config file:
        
        | Name | Type | Description |
        | -----|------|------------ |
        | `replica_set` | string | name of replica set |
        | `ssl` | Boolean | can be set to true to connect using ssl, default false |
        | `verify_mode` | Boolean | Default SSL verify mode, default true |
        | `include_schemas_in_destination_stream_name` | Boolean | forces the stream names to take the form `<database_name>-<collection_name>` instead of `<collection_name>`, default false|
        
        All of the above attributes are required by the tap to connect to your mongo instance. 
        here is a [sample configuration file](./sample_config.json).
        
        ## Run in discovery mode
        Run the following command and redirect the output into the catalog file
        ```shell script
        tap-mongodb --config ~/config.json --discover > ~/catalog.json
        ```
        
        Your catalog file should now look like this:
        ```json
        {
          "streams": [
            {
              "table_name": "<table name>",
              "tap_stream_id": "<tap_stream_id>",
              "metadata": [
                {
                  "breadcrumb": [],
                  "metadata": {
                    "row-count":<int>,
                    "is-view": <bool>,
                    "database-name": "<database name>",
                    "table-key-properties": [
                      "_id"
                    ],
                    "valid-replication-keys": [
                      "_id"
                    ]
                  }
                }
              ],
              "stream": "<stream name>",
              "schema": {
                "type": "object"
              }
            }
          ]
        }
        ```
        
        ## Edit Catalog file
        ### Using valid json, edit the config.json file
        To select a stream, enter the following to the stream's metadata:
        ```json
        "selected": true,
        "replication-method": "<replication method>",
        ```
        
        `<replication-method>` must be either `FULL_TABLE`, `INCREMENTAL` or `LOG_BASED`, if it's `INCREMENTAL`, make sure to add a `"replication-key"`.
        
        
        For example, if you were to edit the example stream to select the stream as well as add a projection, config.json should look this:
        ```json
        {
          "streams": [
            {
              "table_name": "<table name>",
              "tap_stream_id": "<tap_stream_id>",
              "metadata": [
                {
                  "breadcrumb": [],
                  "metadata": {
                    "row-count": <int>,
                    "is-view": <bool>,
                    "database-name": "<database name>",
                    "table-key-properties": [
                      "_id"
                    ],
                    "valid-replication-keys": [
                      "_id"
                    ],
                    "selected": true,
                    "replication-method": "<replication method>"
                  }
                }
              ],
              "stream": "<stream name>",
              "schema": {
                "type": "object"
              }
            }
          ]
        }
        
        ```
        ## Run in sync mode:
        ```shell script
        tap-mongodb --config ~/config.json --catalog ~/catalog.json
        ```
        
        The tap will write bookmarks to stdout which can be captured and passed as an optional `--state state.json` parameter to the tap for the next sync.
        
        ## Logging configuration
        The tap uses a predefined logging config if none is provided, however, you can set your own config by setting the environment variable `LOGGING_CONFIG_FILE` as the path to the logging config.
        A sample config is available [here](./sample_logging.conf).
        
        ---
        
        Copyright &copy; 2020 TransferWise
        
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3 :: Only
Description-Content-Type: text/markdown
Provides-Extra: test
Provides-Extra: dev
