Metadata-Version: 2.1
Name: dlt
Version: 0.4.7
Summary: dlt is an open-source python-first scalable data loading library that does not require any backend to run.
Home-page: https://github.com/dlt-hub
License: Apache-2.0
Keywords: etl
Author: dltHub Inc.
Author-email: services@dlthub.com
Maintainer: Marcin Rudolf
Maintainer-email: marcin@dlthub.com
Requires-Python: >=3.8.1,<3.13
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Provides-Extra: athena
Provides-Extra: az
Provides-Extra: bigquery
Provides-Extra: cli
Provides-Extra: databricks
Provides-Extra: dbt
Provides-Extra: duckdb
Provides-Extra: filesystem
Provides-Extra: gcp
Provides-Extra: gs
Provides-Extra: motherduck
Provides-Extra: mssql
Provides-Extra: parquet
Provides-Extra: postgres
Provides-Extra: qdrant
Provides-Extra: redshift
Provides-Extra: s3
Provides-Extra: snowflake
Provides-Extra: synapse
Provides-Extra: weaviate
Requires-Dist: PyYAML (>=5.4.1)
Requires-Dist: adlfs (>=2022.4.0) ; extra == "az" or extra == "synapse"
Requires-Dist: astunparse (>=1.6.3)
Requires-Dist: botocore (>=1.28) ; extra == "filesystem" or extra == "s3" or extra == "athena"
Requires-Dist: click (>=7.1)
Requires-Dist: cron-descriptor (>=1.2.32) ; extra == "cli"
Requires-Dist: databricks-sql-connector (>=2.9.3,<3.0.0) ; extra == "databricks"
Requires-Dist: dbt-athena-community (>=1.2.0) ; extra == "dbt"
Requires-Dist: dbt-bigquery (>=1.2.0) ; extra == "dbt"
Requires-Dist: dbt-core (>=1.2.0) ; extra == "dbt"
Requires-Dist: dbt-databricks (>=1.7.3,<2.0.0) ; extra == "dbt"
Requires-Dist: dbt-duckdb (>=1.2.0) ; extra == "dbt"
Requires-Dist: dbt-redshift (>=1.2.0) ; extra == "dbt"
Requires-Dist: dbt-snowflake (>=1.2.0) ; extra == "dbt"
Requires-Dist: duckdb (>=0.10.0,<0.11.0) ; (python_version >= "3.12") and (extra == "duckdb" or extra == "motherduck")
Requires-Dist: duckdb (>=0.6.1,<0.10.0) ; (python_version >= "3.8" and python_version < "3.12") and (extra == "duckdb" or extra == "motherduck")
Requires-Dist: fsspec (>=2022.4.0)
Requires-Dist: gcsfs (>=2022.4.0) ; extra == "gcp" or extra == "bigquery" or extra == "gs"
Requires-Dist: gitpython (>=3.1.29)
Requires-Dist: giturlparse (>=0.10.0)
Requires-Dist: google-cloud-bigquery (>=2.26.0) ; extra == "gcp" or extra == "bigquery"
Requires-Dist: grpcio (>=1.50.0) ; extra == "gcp" or extra == "bigquery"
Requires-Dist: hexbytes (>=0.2.2)
Requires-Dist: humanize (>=4.4.0)
Requires-Dist: jsonpath-ng (>=1.5.3)
Requires-Dist: makefun (>=1.15.0)
Requires-Dist: orjson (>=3.6.7,<=3.9.10) ; platform_python_implementation != "PyPy"
Requires-Dist: packaging (>=21.1)
Requires-Dist: pathvalidate (>=2.5.2)
Requires-Dist: pendulum (>=2.1.2)
Requires-Dist: pipdeptree (>=2.9.0,<2.10) ; extra == "cli"
Requires-Dist: psycopg2-binary (>=2.9.1) ; extra == "postgres" or extra == "redshift"
Requires-Dist: psycopg2cffi (>=2.9.0) ; (platform_python_implementation == "PyPy") and (extra == "postgres" or extra == "redshift")
Requires-Dist: pyarrow (>=12.0.0) ; extra == "bigquery" or extra == "parquet" or extra == "motherduck" or extra == "athena" or extra == "synapse"
Requires-Dist: pyathena (>=2.9.6) ; extra == "athena"
Requires-Dist: pyodbc (>=4.0.39,<5.0.0) ; extra == "mssql" or extra == "synapse"
Requires-Dist: pytz (>=2022.6)
Requires-Dist: qdrant-client[fastembed] (>=1.6.4,<2.0.0) ; extra == "qdrant"
Requires-Dist: requests (>=2.26.0)
Requires-Dist: requirements-parser (>=0.5.0)
Requires-Dist: s3fs (>=2022.4.0) ; extra == "filesystem" or extra == "s3" or extra == "athena"
Requires-Dist: semver (>=2.13.0)
Requires-Dist: setuptools (>=65.6.0)
Requires-Dist: simplejson (>=3.17.5)
Requires-Dist: snowflake-connector-python (>=3.5.0) ; extra == "snowflake"
Requires-Dist: tenacity (>=8.0.2)
Requires-Dist: tomlkit (>=0.11.3)
Requires-Dist: typing-extensions (>=4.0.0)
Requires-Dist: tzdata (>=2022.1)
Requires-Dist: weaviate-client (>=3.22) ; extra == "weaviate"
Requires-Dist: win-precise-time (>=1.4.2) ; os_name == "nt"
Project-URL: Repository, https://github.com/dlt-hub/dlt
Description-Content-Type: text/markdown

<h1 align="center">
    <strong>data load tool (dlt) — the open-source Python library for data loading</strong>
</h1>
<p align="center">
Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,<br/>or a GPT-4 assisted development playground—<strong>dlt</strong> can be dropped in anywhere.
</p>


<h3 align="center">

🚀 Join our thriving community of likeminded developers and build the future together!

</h3>

<div align="center">
  <a target="_blank" href="https://dlthub.com/community" style="background:none">
    <img src="https://img.shields.io/badge/slack-join-dlt.svg?labelColor=191937&color=6F6FF7&logo=slack" style="width: 260px;"  />
  </a>
</div>
<div align="center">
  <a target="_blank" href="https://pypi.org/project/dlt/" style="background:none">
    <img src="https://img.shields.io/pypi/v/dlt?labelColor=191937&color=6F6FF7">
  </a>
  <a target="_blank" href="https://pypi.org/project/dlt/" style="background:none">
    <img src="https://img.shields.io/pypi/pyversions/dlt?labelColor=191937&color=6F6FF7">
  </a>
</div>

## Installation

dlt supports Python 3.8+.

```bash
pip install dlt
```

## Quick Start

Load chess game data from chess.com API and save it in DuckDB:

```python
import dlt
from dlt.sources.helpers import requests

# Create a dlt pipeline that will load
# chess player data to the DuckDB destination
pipeline = dlt.pipeline(
    pipeline_name='chess_pipeline',
    destination='duckdb',
    dataset_name='player_data'
)

# Grab some player data from Chess.com API
data = []
for player in ['magnuscarlsen', 'rpragchess']:
    response = requests.get(f'https://api.chess.com/pub/player/{player}')
    response.raise_for_status()
    data.append(response.json())

# Extract, normalize, and load the data
pipeline.run(data, table_name='player')
```


Try it out in our **[Colab Demo](https://colab.research.google.com/drive/1NfSB1DpwbbHX9_t5vlalBTf13utwpMGx?usp=sharing)**

## Features

- **Automatic Schema:** Data structure inspection and schema creation for the destination.
- **Data Normalization:** Consistent and verified data before loading.
- **Seamless Integration:** Colab, AWS Lambda, Airflow, and local environments.
- **Scalable:** Adapts to growing data needs in production.
- **Easy Maintenance:** Clear data pipeline structure for updates.
- **Rapid Exploration:** Quickly explore and gain insights from new data sources.
- **Versatile Usage:** Suitable for ad-hoc exploration to advanced loading infrastructures.
- **Start in Seconds with CLI:** Powerful CLI for managing, deploying and inspecting local pipelines.
- **Incremental Loading:** Load only new or changed data and avoid loading old records again.
- **Open Source:** Free and Apache 2.0 Licensed.

## Ready to use Sources and Destinations

Explore ready to use sources (e.g. Google Sheets) in the [Verified Sources docs](https://dlthub.com/docs/dlt-ecosystem/verified-sources) and supported destinations (e.g. DuckDB) in the [Destinations docs](https://dlthub.com/docs/dlt-ecosystem/destinations).

## Documentation

For detailed usage and configuration, please refer to the [official documentation](https://dlthub.com/docs).

## Examples

You can find examples for various use cases in the [examples](docs/examples) folder.

## Adding as dependency

`dlt` follows the semantic versioning with the [`MAJOR.MINOR.PATCH`](https://peps.python.org/pep-0440/#semantic-versioning) pattern. Currently, we are using **pre-release versioning** with the major version being 0.

- `minor` version change means breaking changes
- `patch` version change means new features that should be backward compatible
- any suffix change, e.g., `post10` -> `post11`, is considered a patch

We suggest that you allow only `patch` level updates automatically:
* Using the [Compatible Release Specifier](https://packaging.python.org/en/latest/specifications/version-specifiers/#compatible-release). For example **dlt~=0.3.10** allows only versions **>=0.3.10** and less than **<0.4**
* Poetry [caret requirements](https://python-poetry.org/docs/dependency-specification/). For example **^0.3.10** allows only versions **>=0.3.10** to **<0.4**
## Get Involved

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

- **Connect with the Community**: Join other dlt users and contributors on our [Slack](https://dlthub.com/community)
- **Report issues and suggest features**: Please use the [GitHub Issues](https://github.com/dlt-hub/dlt/issues) to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
- **Track progress of our work and our plans**: Please check out our [public Github project](https://github.com/orgs/dlt-hub/projects/9)
- **Contribute Verified Sources**: Contribute your custom sources to the [dlt-hub/verified-sources](https://github.com/dlt-hub/verified-sources) to help other folks in handling their data tasks.
- **Contribute code**: Check out our [contributing guidelines](CONTRIBUTING.md) for information on how to make a pull request.
- **Improve documentation**: Help us enhance the dlt documentation.

## License

DLT is released under the [Apache 2.0 License](LICENSE.txt).

