Metadata-Version: 2.1
Name: cognite-cdffs
Version: 0.1.0
Summary: File-System Interface for CDF Files
License: Apache-2.0
Author: Infant Alex
Author-email: infant.alex@cognite.com
Requires-Python: >=3.8.10,<4.0.0
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: cognite-sdk (>=4.11.0,<5.0.0)
Requires-Dist: fsspec (>=2022.10.0,<2023.0.0)
Requires-Dist: requests (>=2.28.1,<3.0.0)
Requires-Dist: twine (>=4.0.2,<5.0.0)
Description-Content-Type: text/markdown

<picture href="https://cognite.com/">
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/cognitedata/cognite-python-docs/master/img/cognite_logo_white.png">
  <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/cognitedata/cognite-python-docs/master/img/cognite_logo_black.png" >
  <img alt="Cognite" src="https://raw.githubusercontent.com/cognitedata/cognite-python-docs/master/img/cognite_logo_black.png" alt="Cognite logo" title="Cognite" align="right" height="80">
</picture>

[![GitHub](https://img.shields.io/github/license/cognitedata/cognite-sdk-python)](https://github.com/cognitedata/cognite-sdk-python/blob/master/LICENSE)
[![Documentation Status](https://readthedocs.org/projects/cdffs/badge/?version=latest)](https://cdffs.readthedocs.io/en/latest/?badge=latest)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)

# cdffs

A file system interface (`cdffs`) to allow users to work with CDF Files using the [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) supported/compatible python packages (`pandas`, `xarray` etc).

`fsspec` provides an abstract file system interface to work with local/cloud storages and based on the protocol name (example, `s3` or `abfs`) provided in the path, `fsspec` translates the incoming requests to storage specific implementations and send the responses back to the upstream package to work with the desired data.

Refer [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/#who-uses-fsspec) to get the list of all supported/compatible python packages.

## Installation

`cdffs` is available on PyPI. Install using, 

```shell
pip install cognite-cdffs
```

## Usage

Three important steps to follow when working with CDF Files using the fsspec supported python packages. 

1) Import `cdffs` package
```python
  from cognite import cdffs
```

2) Create a client config to connect with CDF. 
```python
  # Get TOKEN_URL, CLIENT_ID, CLIENT_SECRET, COGNITE_PROJECT, 
  # CDF_CLUSTER, SCOPES from environment variables.

  oauth_creds = OAuthClientCredentials(
    token_url=TOKEN_URL, client_id=CLIENT_ID, client_secret=CLIENT_SECRET, scopes=SCOPES
  )
  client_cnf = ClientConfig(
    client_name="cdf-client",
    base_url=f"https://{CDF_CLUSTER}.cognitedata.com",
    project=COGNITE_PROJECT,
    credentials=oauth_creds
  )
```

  > Refer [ClientConfig](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/cognite.html#cognite.client.config.ClientConfig) from Cognite Python SDK documentation on how to create a client config.

3) Pass the client config as `connection_config` in `storage_options` when reading/writing the files.

    * Read `zarr` files using using `xarray`.

    ```python
    ds = xarray.open_zarr("cdffs://sample_data/test.zarr", storage_options={"connection_config": client_cnf})
    ```
    * Write `zarr` files using `xarray`.
    
    ```python
    ds.to_zarr("cdffs://sample_data/test.zarr", storage_options={"connection_config": client_cnf, "file_metadata": metadata})
    ```

## Contributing
Want to contribute? Check out [CONTRIBUTING](CONTRIBUTING.md).

