Metadata-Version: 2.1
Name: k8s-spark-helper-liangdao-data
Version: 0.0.3
Summary: Run spark task in k8s
Home-page: 
Author: yuchao.li
Author-email: yuchao.li@liangdao.ai
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE

###spark-helper

####在k8s集群中调用spark (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator)

Author: yuchao.li@liangdao.ai
Created: 2023.3.9

```
Usage:
    s3_config = {'endpoint': 'http://minio.minio-tenant',
                'access': 'xxx',
                'secret': 'xxx'}

    set_config(image='pyspark:driver-3.3.1',
               namespace='spark-operator',
               service_account='sparkoperator',
               s3_config = s3_config )

    start_spark('hello-world-spark',
                's3a://airflow-dags/spark/pyspark_helloworld.py',
                False)
```

####配置优先级：
    从高到低
    set_config 设置的配置
    auto_load_config 自动加载的环境变量配置

