Metadata-Version: 2.1
Name: pyspark-iomete
Version: 0.0.2
Summary: IOMETE's PySpark library that contains useful utilities for working with PySpark
Home-page: https://github.com/iomete/pyspark-iomete
Author: IOMETE
Author-email: vusal@iomete.com
License: Apache License 2.0
Keywords: iomete,pyspark,utils
Classifier: Development Status :: 5 - Production/Stable
Classifier: Topic :: Software Development :: Build Tools
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: dev

# Pyspark IOMETE Library

This library is providing a set of utility functions to speed up the development of pyspark applications.


## Installation

```bash
pip install pyspark-iomete
```


## Utility functions

### get_spark_logger

This function is returning a spark logger instance.

As you may know, spark is using log4j as a logging framework. This function is returning a spark logger instance that is using the log4j logger. Standard python logging is not working with pyspark. The following function get the spark logger instance and returns it.

Usage:
```python
from pyspark_iomete.utils import get_spark_logger
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
logger = get_spark_logger(spark=spark)

# or with a custom name
logger = get_spark_logger(spark=spark, name="my_custom_logger")
```

## Test utility functions

### table_name_with_random_suffix

This function is returning a table name with a random suffix. This is useful for testing purposes.

Usage:
```python
from pyspark_iomete.test_utils import table_name_with_random_suffix

table_name = table_name_with_random_suffix("my_table")
```
 
