Metadata-Version: 2.1
Name: atlass3hook
Version: 0.0.1
Summary: This atlas s3 hook uses s3fs package to gather the metadata of bucket, pseudo_dir and object, then it inserts these metadata into Atlas instances.
Home-page: https://git.lab.sspcloud.fr/pengfei/atlass3hook
Author: Pengfei Liu
Author-email: pengfei.liu@insee.fr
License: Apache License 2.0
Description: # Apache Atlas S3 hook in Python
        
        This python client uses a s3 client package s3fs to get the metadata of s3 entities such as Bucket, pseudo_dir and 
        object, then it inserts these metadata into Atlas instances.
        
        
        ## Quick start
        
        ### Create a client to connect to an Atlas instance
        ```python
        from atlas_client.client import Atlas
        # login with your token
        hostname = "https://atlas.lab.sspcloud.fr"
        port = 443
        oidc_token = "<your_token>"
        atlas_client = Atlas(hostname, port, oidc_token=oidc_token)
        
        # login with your username and password
        atlas_client = Atlas(hostname, port, username='',password='')
        ```
        ### Create a s3 metadata client to collect metadata of s3 entities
        ```python
        from atlas_s3_hook.S3MetadataClient import S3MetadataClient
        
        s3_end_point = ''
        s3_access_key = ''
        s3_secret_key = ''
        s3_token = ''
        
        s3_client = S3MetadataClient(s3_end_point, s3_access_key, s3_secret_key, s3_token)
        
        
        ```
        
        ### Load a single s3 entity into atlas 
        If you want to load the metadata of a single s3 entity, you can use the following code example
        ```python
        from atlas_s3_hook.S3Hook import S3Hook
        
        # Indicate the path of the entity which you want to 
        path=''
        description=''
        s3_hook = S3Hook(s3_client, atlas_client)
        # Get the class of the s3 entity
        path_class = s3_client.get_class_from_path(path)
        print(path_class)
        
        # Get the metadata of the s3 entity
        meta_data = s3_client.get_path_meta_data(path)
        
        # based on the class of the s3 entity, s3 hook provides different loaders. You need to choose the correct one
        
        # bucket loader 
        s3_hook.create_atlas_bucket(meta_data,description)
        
        # directory loader
        s3_hook.create_atlas_ps_dir(meta_data,description)
        
        # object loader
        s3_hook.create_atlas_object(meta_data,description)
        
        
        ``` 
        
        
        ### Load multiple s3 entities into atlas 
        If you want to load the metadata of multiple s3 entities, you can use the following code example. The S3Scanner class 
        takes a path of s3 entity and load all the metadata of its contents (e.g. sub-directory, objects). 
        
        ```python
        from atlas_s3_hook.S3Scanner import S3Scanner
        
        s3_entity_path=''
        entity_owner=''
        
        minio_scanner = S3Scanner(minio_client, atlas_client, owner=entity_owner)
        minio_scanner.scan_path(s3_entity_path)
        
        ```
        
        ## Prerequisites
        
        This tool only requires python 3.7 or above
        
        ## Supported OS
        
        Windows XP/7/8/10
        
        Linux  
        
        MacOS
        
        
        ## Authors
        
        * **Pengfei Liu** 
        
        
        ## License
        
        This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details
        
        ## Acknowledgement
        
        This package was created by using [s3fs](<https://pypi.org/project/s3fs/>) project
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
