Metadata-Version: 2.1
Name: object-clerk
Version: 0.0.16
Summary: A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.
Home-page: https://bitbucket.org/dkistdc/object_clerk/src/master
Author: NSO / AURA
Author-email: "swiant@nso.edu"
License: MIT
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Requires-Python: >=3.6
Provides-Extra: test
License-File: LICENSE

object-clerk
============

|codecov|

A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:

- get_object

- head_object

- upload_fileobj

- copy_object

- delete_object

- list_objects_v2

Features
--------

- Retry connection failures

- Confirm checksum of uploaded and retrieved objects

- Move object

- Constrained interface to support simple CRUD operations for objects in existing buckets

Configuration
-------------
The following environment variables configure the object clerk.

+---------------------+---------------------------------------------------------------------------------------------------------------------------------+--------+-----------------------------------------------------------------------------------------+
| *Variable*          | *Description*                                                                                                                   | *Type* | *Default*                                                                               |
+=====================+=================================================================================================================================+========+=========================================================================================+
| MULTIPART_THRESHOLD | Threshold in bytes at which uploads are broken into multiple parts for upload.  Impacts the checksum stored in the eTag         | STR    | 524288000                                                                               |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------+--------+-----------------------------------------------------------------------------------------+
| S3_CLIENT_CONFIG    | Boto Core Client Configuration                                                                                                  | JSON   | ```json {"connect_timeout": 60, "read_timeout": 60, "retries": {"max_attempts": 0}} ``` |
|                     | https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html                                                |        |                                                                                         |
|                     | Defaults reflect capturing the retry of failed operations at a higher level.                                                    |        |                                                                                         |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------+--------+-----------------------------------------------------------------------------------------+
| S3_UPLOAD_CONFIG    | Transfer Configuration applied to uploads                                                                                       | JSON   | ```json {} ```                                                                          |
|                     | https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig       |        |                                                                                         |
|                     | multipart_threshold and multipart_chunksize are set by the MULTIPART_THRESHOLD environment variable.  Values for these keys in  |        |                                                                                         |
|                     | the S3_UPLOAD_CONFIG will be discarded.                                                                                         |        |                                                                                         |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------+--------+-----------------------------------------------------------------------------------------+
| S3_DOWNLOAD_CONFIG  | Transfer Configuration applied to downloads                                                                                     | JSON   | ```json {} ```                                                                          |
|                     | https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig       |        |                                                                                         |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------+--------+-----------------------------------------------------------------------------------------+

Installation
------------

.. code:: bash

    pip install object-clerk

Examples
--------

**Initialize**

.. code:: python

    clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'

**Get Object**

.. code:: python

    # with checksum verified

    bytes_response = clerk.get_object("bucket", "object_key")

    # without checksum verified

    bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)

**Delete Object**

.. code:: python

    clerk.delete_object("bucket", "object_key")

**Get Object Info**

.. code:: python

    dict_response = clerk.get_object_info("bucket", "object_key")

**Copy Object**

.. code:: python

    # with checksum verified

    clerk.copy_object(
        "source_bucket",
        "source_object_key",
        "destination_bucket",
        "destination_object_key",
    )

    # without checksum verified

    clerk.copy_object(
        "source_bucket",
        "source_object_key",
        "destination_bucket",
        "destination_object_key",
        verify_checksum=False
    )

**Upload Object**

.. code:: python

    # with checksum verified

    with open("file", mode='rb') as f:

        clerk.upload_object(f, "bucket", "object_key")

    # without checksum verified

    with open("file", mode='rb') as f:

        clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)

**Move Object**

.. code:: python

    # with checksum verified

    clerk.move_object(
        "source_bucket",
        "source_object_key",
        "destination_bucket",
        "destination_object_key",
    )

    # without checksum verified

    clerk.move_object(
        "source_bucket",
        "source_object_key",
        "destination_bucket",
        "destination_object_key",
        verify_checksum=False
    )

**List Object**

.. code:: python

    clerk.list_objects(
        "bucket_name",
        1000
    )


Test
----

.. code:: bash

    git clone git@bitbucket.org:swiant/object_store_wrapper.git

    pip install -e .

    export HOST=<host>

    export PORT=<port>

    export ACCESS_KEY=<access_key>

    export SECRET_KEY=<secret_key>

    pytest -v object_clerk



.. |codecov| image:: https://codecov.io/bb/dkistdc/object_store_wrapper/branch/master/graph/badge.svg
   :target: https://codecov.io/bb/dkistdc/object_store_wrapper


