Metadata-Version: 2.1
Name: cloudstorageio
Version: 1.2.7
Home-page: https://github.com/cognaize/cloudstorageio
Author: cognaize
Description-Content-Type: text/markdown
License-File: LICENSE

# cloudstorageio

Storage agnostic IO interface for humans

_Developed and tested on python 3.6+_

[GitHub](https://github.com/VahagnGhaz/cloudstorageio)

[PyPi](https://pypi.org/project/cloudstorageio)

## Getting Started
These instructions will get you a copy of the project up and running on your local machine.

* #### S3 configs
    ```bash
    pip install awscli --upgrade --user
    ```
    set your aws configs
    ```bash
     sudo apt install awscli
    ```

    ```bash
    $ aws configure
    AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
    AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    Default region name [None]: us-west-2
    Default output format [None]: json
     ```
   [click here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#installation) for more info about configuration and installation

* #### Google cloud storage configs
   Provide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS.
   Replace [FILE_NAME] with the file of the JSON file that contains your service account key

    ```bash
   GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"
    ```

   [click here](https://cloud.google.com/storage/docs/reference/libraries) for more info about configuration and installation

* #### Dropbox workspace configs
    Provide app access token by setting the environment variable DROPBOX_TOKEN.
    Replace [TOKEN] with the generated token of your dropbox app

    ```bash
    DROPBOX_TOKEN='TOKEN'
    ```

   [click here](http://99rabbits.com/get-dropbox-access-token/) for more info about configuration and installation


### Installation
```
pip install cloudstorageio
```

### Usage

* Write and read file
```python
from cloudstorageio import CloudInterface

google_file_path = 'gs://bucket-name/path-to-file/sample-file.txt'
s3_file_path = 's3://bucket-name/path-to-file/sample-pic.jpg'
ci = CloudInterface()

# Write text to Google cloud storage file
with ci.open(google_file_path, 'w') as f:
   f.write("Lorem Ipsum is simply dummy text")
# OR
ci.save(google_file_path, "Lorem Ipsum is simply dummy text")

# Read picture from S3 storage
with ci.open(s3_file_path, 'rb') as f:
    s3_output = f.read()  # binary content of picture
# OR
ci.fetch(s3_file_path)

```

* Remove, list, and check folder/file
```python
from cloudstorageio import CloudInterface

dropbox_folder_path = 'dbx://bucket-name/folder'
s3_file_path = 's3://bucket-name/path-to-file/sample-pic.jpg'
ci = CloudInterface()

ci.isfile(s3_file_path) # returns True
ci.isdir(dropbox_folder_path) # returns True
ci.remove(s3_file_path) # removes file
ci.listdir(dropbox_folder_path) # lists folder content
```
* Copy file
```python
from cloudstorageio import CloudInterface

dropbox_file_path = 'dbx://bucket-name/path-to-file/sample-pic.jpg'
gs_file_path = 'gs://bucket-name/path-to-file/sample-file.txt'
ci = CloudInterface()

ci.copy(from_path=dropbox_file_path, to_path=gs_file_path) # copies dropbox file to gs
```

* Copy dir
```python
from cloudstorageio import CloudInterface

s3_dir = 's3://bucket-name/sample_folder'
gs_dir = 'gs://bucket-name/sample_folder'
ci = CloudInterface()

ci.copy_dir(source_dir=s3_dir, dest_dir=gs_dir) # copies s3 folder to gs
```

_Powered by_ ![](/docs/cognaize_logo.png) [Cognaize](https://www.cognaize.com/) 

`# Changelog
All major cloudstorageio changes and pypi releases are listed here 

# Changes 
## [0.0.1] - 2019-01-29
- Created project skeleton
## [0.0.2] - 2019-02-14
- Implemented S3 & Google Cloud Storage Interfaces 
## [0.0.3] - 2019-02-28
- Implemented Cloud Storage Interface
- Implemented Local interface 
## [0.0.4] - 2019-05-31
- Add delete, listdir & isfile/isdir methods for all interfaces
## [0.0.5] - 2019-07-09
- Implemented DropBox interface
## [0.0.6] - 2019-07-24
- Add copy_batch for folder copying 
## [0.0.7] - 2019-08-08
- Add Google Drive interface
## [0.0.8] - 2019-08-30
- Add Async Cloud Interface 
## [0.0.9] - 2019-10-18
- Structure changes (file renames/moves)
- Implemented scalable unittests
## [0.0.10] - 2020-07-16
- Commented Overwriting loggers, 
- add cognaize logo in Readme
## [0.1.0] - 2020-08-05
- Add copy_dir for path to path copy
- Change copy_batch for list to list copy 
- Usages in README
## [0.1.1] - 2020-08-14
- README changes
## [0.1.2] - 2020-11-10
- Using multipart upload when copying from local to S3


# Pypi releases
See CHANGES references  
### [0.12.1] - 2019-03-12
- [0.0.3]

### [1.0.5] - 2019-06-18
- [0.0.4]
### [1.0.8] - 2019-08-01
- [0.0.6]
### [1.0.9] - 2019-10-24
- [0.0.9]
### [1.1.0] - 2020-08-05
- [0.1.0]
### [1.1.1] - 2020-08-14
- [0.1.1]
### [1.1.2] - 2020-11-10
- [0.1.2]
### [1.1.3] - 2020-11-10
- Update listdir for s3
### [1.2.1] - 2020-11-10
- Add AWS session token support
### [1.2.3] - 2020-11-10
- Remove dropbox package dependency (dropbox will not be supported with this version)
### [1.2.4] - 2020-11-10
- Add option include_files for listdir
### [1.2.7] - 2022-10-05
- Updated build to setuptools
