Metadata-Version: 2.1
Name: dlt
Version: 1.6.0a0
Summary: dlt is an open-source python-first scalable data loading library that does not require any backend to run.
Home-page: https://github.com/dlt-hub
License: Apache-2.0
Keywords: etl
Author: dltHub Inc.
Author-email: services@dlthub.com
Maintainer: Marcin Rudolf
Maintainer-email: marcin@dlthub.com
Requires-Python: >=3.9,<3.14
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Provides-Extra: athena
Provides-Extra: az
Provides-Extra: bigquery
Provides-Extra: cli
Provides-Extra: clickhouse
Provides-Extra: databricks
Provides-Extra: deltalake
Provides-Extra: dremio
Provides-Extra: duckdb
Provides-Extra: filesystem
Provides-Extra: gcp
Provides-Extra: gs
Provides-Extra: lancedb
Provides-Extra: motherduck
Provides-Extra: mssql
Provides-Extra: parquet
Provides-Extra: postgis
Provides-Extra: postgres
Provides-Extra: pyiceberg
Provides-Extra: qdrant
Provides-Extra: redshift
Provides-Extra: s3
Provides-Extra: sftp
Provides-Extra: snowflake
Provides-Extra: sql-database
Provides-Extra: sqlalchemy
Provides-Extra: synapse
Provides-Extra: weaviate
Requires-Dist: PyYAML (>=5.4.1)
Requires-Dist: adlfs (>=2024.7.0) ; extra == "az" or extra == "synapse" or extra == "clickhouse"
Requires-Dist: alembic (>1.10.0) ; extra == "sqlalchemy"
Requires-Dist: botocore (>=1.28) ; extra == "filesystem" or extra == "s3" or extra == "athena"
Requires-Dist: click (>=7.1)
Requires-Dist: clickhouse-connect (>=0.7.7) ; extra == "clickhouse"
Requires-Dist: clickhouse-driver (>=0.2.7) ; extra == "clickhouse"
Requires-Dist: cron-descriptor (>=1.2.32) ; extra == "cli"
Requires-Dist: databricks-sdk (>=0.38.0) ; extra == "databricks"
Requires-Dist: databricks-sql-connector (>=2.9.3,<4) ; (python_version <= "3.12") and (extra == "databricks")
Requires-Dist: databricks-sql-connector (>=3.6.0) ; (python_version >= "3.13") and (extra == "databricks")
Requires-Dist: db-dtypes (>=1.2.0) ; extra == "gcp" or extra == "bigquery"
Requires-Dist: deltalake (>=0.21.0) ; extra == "deltalake"
Requires-Dist: dlt-pendulum (==3.0.1a4) ; python_version >= "3.13"
Requires-Dist: duckdb (>=0.9) ; extra == "duckdb" or extra == "motherduck"
Requires-Dist: fsspec (>=2022.4.0)
Requires-Dist: gcsfs (>=2022.4.0) ; extra == "gcp" or extra == "bigquery" or extra == "gs" or extra == "clickhouse"
Requires-Dist: gitpython (>=3.1.29)
Requires-Dist: giturlparse (>=0.10.0)
Requires-Dist: google-cloud-bigquery (>=2.26.0) ; extra == "gcp" or extra == "bigquery"
Requires-Dist: grpcio (>=1.50.0) ; extra == "gcp" or extra == "bigquery"
Requires-Dist: hexbytes (>=0.2.2)
Requires-Dist: humanize (>=4.4.0)
Requires-Dist: jsonpath-ng (>=1.5.3)
Requires-Dist: lancedb (>=0.8.2) ; (python_version < "3.13") and (extra == "lancedb")
Requires-Dist: makefun (>=1.15.0)
Requires-Dist: orjson (>=3.6.7,<4,!=3.9.11,!=3.9.12,!=3.9.13,!=3.9.14,!=3.10.1) ; platform_python_implementation != "PyPy"
Requires-Dist: packaging (>=21.1)
Requires-Dist: paramiko (>=3.3.0) ; extra == "sftp"
Requires-Dist: pathvalidate (>=2.5.2)
Requires-Dist: pendulum (>=2.1.2) ; python_version < "3.13"
Requires-Dist: pipdeptree (>=2.9.0,<2.10) ; extra == "cli"
Requires-Dist: pluggy (>=1.3.0)
Requires-Dist: psycopg2-binary (>=2.9.1) ; extra == "postgres" or extra == "redshift" or extra == "postgis"
Requires-Dist: psycopg2cffi (>=2.9.0) ; (platform_python_implementation == "PyPy") and (extra == "postgres" or extra == "redshift" or extra == "postgis")
Requires-Dist: pyarrow (>=12.0.0,<18) ; (python_version >= "3.9" and python_version < "3.13") and (extra == "bigquery" or extra == "parquet" or extra == "motherduck" or extra == "athena" or extra == "synapse" or extra == "clickhouse" or extra == "dremio" or extra == "lancedb" or extra == "deltalake" or extra == "pyiceberg")
Requires-Dist: pyarrow (>=18.0.0) ; (python_version >= "3.13") and (extra == "bigquery" or extra == "parquet" or extra == "motherduck" or extra == "athena" or extra == "synapse" or extra == "clickhouse" or extra == "dremio" or extra == "lancedb" or extra == "deltalake" or extra == "pyiceberg")
Requires-Dist: pyathena (>=2.9.6) ; extra == "athena"
Requires-Dist: pyiceberg (>=0.8.1) ; extra == "pyiceberg"
Requires-Dist: pyodbc (>=4.0.39) ; extra == "mssql" or extra == "synapse"
Requires-Dist: pytz (>=2022.6)
Requires-Dist: pywin32 (>=306) ; sys_platform == "win32"
Requires-Dist: qdrant-client[fastembed] (>=1.8) ; extra == "qdrant"
Requires-Dist: requests (>=2.26.0)
Requires-Dist: requirements-parser (>=0.5.0)
Requires-Dist: s3fs (>=2022.4.0) ; extra == "filesystem" or extra == "s3" or extra == "athena" or extra == "clickhouse"
Requires-Dist: semver (>=3.0.0)
Requires-Dist: setuptools (>=65.6.0)
Requires-Dist: simplejson (>=3.17.5)
Requires-Dist: snowflake-connector-python (>=3.5.0) ; extra == "snowflake"
Requires-Dist: sqlalchemy (>=1.4) ; extra == "sql-database" or extra == "sqlalchemy" or extra == "pyiceberg"
Requires-Dist: sqlglot (>=20.0.0) ; extra == "filesystem"
Requires-Dist: tantivy (>=0.22.0) ; extra == "lancedb"
Requires-Dist: tenacity (>=8.0.2)
Requires-Dist: tomlkit (>=0.11.3)
Requires-Dist: typing-extensions (>=4.0.0)
Requires-Dist: tzdata (>=2022.1)
Requires-Dist: weaviate-client (>=3.22) ; extra == "weaviate"
Requires-Dist: win-precise-time (>=1.4.2) ; os_name == "nt"
Project-URL: Repository, https://github.com/dlt-hub/dlt
Description-Content-Type: text/markdown

<h1 align="center">
    <strong>data load tool (dlt) — the open-source Python library for data loading</strong>
</h1>
<p align="center">
Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,<br/>or a GPT-4 assisted development playground—<strong>dlt</strong> can be dropped in anywhere.
</p>


<h3 align="center">

🚀 Join our thriving community of likeminded developers and build the future together!

</h3>

<div align="center">
  <a target="_blank" href="https://dlthub.com/community" style="background:none">
    <img src="https://img.shields.io/badge/slack-join-dlt.svg?labelColor=191937&color=6F6FF7&logo=slack" style="width: 260px;"  />
  </a>
</div>
<div align="center">
  <a target="_blank" href="https://pypi.org/project/dlt/" style="background:none">
    <img src="https://img.shields.io/pypi/v/dlt?labelColor=191937&color=6F6FF7">
  </a>
  <a target="_blank" href="https://pypi.org/project/dlt/" style="background:none">
    <img src="https://img.shields.io/pypi/pyversions/dlt?labelColor=191937&color=6F6FF7">
  </a>
  <a target="_blank" href="https://pypi.org/project/dlt/" style="background:none">
    <img src="https://img.shields.io/pypi/dm/dlt?labelColor=191937&color=6F6FF7">
  </a>
</div>

## Installation

dlt supports Python 3.9+. Python 3.13 is supported but considered experimental at this time as not all of dlts extras have python 3.13. support. We additionally maintain a [forked version of pendulum](https://github.com/dlt-hub/pendulum) for 3.13 until there is a release for 3.13.

```sh
pip install dlt
```

More options: [Install via Conda or Pixi](https://dlthub.com/docs/reference/installation#install-dlt-via-pixi-and-conda)


## Quick Start

Load chess game data from chess.com API and save it in DuckDB:

```python
import dlt
from dlt.sources.helpers import requests

# Create a dlt pipeline that will load
# chess player data to the DuckDB destination
pipeline = dlt.pipeline(
    pipeline_name='chess_pipeline',
    destination='duckdb',
    dataset_name='player_data'
)

# Grab some player data from Chess.com API
data = []
for player in ['magnuscarlsen', 'rpragchess']:
    response = requests.get(f'https://api.chess.com/pub/player/{player}')
    response.raise_for_status()
    data.append(response.json())

# Extract, normalize, and load the data
pipeline.run(data, table_name='player')
```


Try it out in our **[Colab Demo](https://colab.research.google.com/drive/1NfSB1DpwbbHX9_t5vlalBTf13utwpMGx?usp=sharing)**

## Features

- **Automatic Schema:** Data structure inspection and schema creation for the destination.
- **Data Normalization:** Consistent and verified data before loading.
- **Seamless Integration:** Colab, AWS Lambda, Airflow, and local environments.
- **Scalable:** Adapts to growing data needs in production.
- **Easy Maintenance:** Clear data pipeline structure for updates.
- **Rapid Exploration:** Quickly explore and gain insights from new data sources.
- **Versatile Usage:** Suitable for ad-hoc exploration to advanced loading infrastructures.
- **Start in Seconds with CLI:** Powerful CLI for managing, deploying and inspecting local pipelines.
- **Incremental Loading:** Load only new or changed data and avoid loading old records again.
- **Open Source:** Free and Apache 2.0 Licensed.

## Ready to use Sources and Destinations

Explore ready to use sources (e.g. Google Sheets) in the [Verified Sources docs](https://dlthub.com/docs/dlt-ecosystem/verified-sources) and supported destinations (e.g. DuckDB) in the [Destinations docs](https://dlthub.com/docs/dlt-ecosystem/destinations).

## Documentation

For detailed usage and configuration, please refer to the [official documentation](https://dlthub.com/docs).

## Examples

You can find examples for various use cases in the [examples](docs/examples) folder.

## Adding as dependency

`dlt` follows the semantic versioning with the [`MAJOR.MINOR.PATCH`](https://peps.python.org/pep-0440/#semantic-versioning) pattern.

* `major` means breaking changes and removed deprecations
* `minor` new features, sometimes automatic migrations
* `patch` bug fixes

We suggest that you allow only `patch` level updates automatically:
* Using the [Compatible Release Specifier](https://packaging.python.org/en/latest/specifications/version-specifiers/#compatible-release). For example **dlt~=1.0** allows only versions **>=1.0** and less than **<1.1**
* Poetry [caret requirements](https://python-poetry.org/docs/dependency-specification/). For example **^1.0** allows only versions **>=1.0** to **<1.0**

## Get Involved

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

- **Connect with the Community**: Join other dlt users and contributors on our [Slack](https://dlthub.com/community)
- **Report issues and suggest features**: Please use the [GitHub Issues](https://github.com/dlt-hub/dlt/issues) to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
- **Track progress of our work and our plans**: Please check out our [public Github project](https://github.com/orgs/dlt-hub/projects/9)
- **Contribute Verified Sources**: Contribute your custom sources to the [dlt-hub/verified-sources](https://github.com/dlt-hub/verified-sources) to help other folks in handling their data tasks.
- **Contribute code**: Check out our [contributing guidelines](CONTRIBUTING.md) for information on how to make a pull request.
- **Improve documentation**: Help us enhance the dlt documentation.

## License

`dlt` is released under the [Apache 2.0 License](LICENSE.txt).

