Metadata-Version: 2.4
Name: pyspark-explorer
Version: 0.2.0
Summary: Explore data files with pyspark
Project-URL: Homepage, https://github.com/krzys9876/pyspark_explorer
Project-URL: Repository, https://github.com/krzys9876/pyspark_explorer
Author-email: Krzysztof Ruta <krzys9876@gmail.com>
Maintainer-email: Krzysztof Ruta <krzys9876@gmail.com>
License: MIT License
        
        Copyright (c) 2024 Krzysztof Ruta
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: data,explorer,pyspark,spark
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python
Requires-Python: >=3.11
Requires-Dist: pyspark>=3.5.1
Requires-Dist: textual>=1.0.0
Description-Content-Type: text/markdown

# Spark File Explorer
When developing spark applications I came across the growing number of data files that I create. 

## CSVs are fine but what about JSON and complex PARQUET files?

To open and explore a file I used Excel to view CSV files, text editors with plugins to view JSON files, 
but there was nothing handy to view PARQUETs. Event formatted JSONs were not always readable. What about viewing schemas? 

Each time I had to use spark and write simple apps which was not a problem itself but was tedious and boring.

## Why not a database?

Well, for tabular data there problems is already solved - just use your preferred database.
Quite often we can load text files or even parquets directly to the database. 

So what's the big deal?

## Hierarchical data sets

Unfortunately the files I often deal with have hierarchical structure. They cannot be simply visualized as tables
or rather some fields contain tables of other structures. Each of these structures is a table itself but how to load 
and explore such embedded tables in a database?

## For Spark files use... Spark! 

Hold on - since I generate files using Apache Spark, why can't I use it to explore them?
I can easily handle complex structures and file types using built-in features. So all I need is to build a use interface 
to display directories, files and their contents.

## Why console?

I use Kubernetes in production environment, I develop Spark applications locally or in VM. 
In all environments I would like to have _one tool to rule them all_.  

I like console tools a lot, they require some sort of simplicity. They can run locally or over SSH connection on 
the remote cluster. Sounds perfect. All I needed was a console UI library, so I wouldn't have to reinvent the wheel.

## Textual

What a great project [_textual_](https://textual.textualize.io/) is! 

Years ago I used [_curses_](https://docs.python.org/3/library/curses.html) but 
[_textual_](https://textual.textualize.io/) is so superior to what I used back then. It has so many features packed in
a friendly form of simple to use components. Highly recommended.

# Usage

Install package with pip:
    
    pip install pyspark-explorer

Run:

    pyspark-explorer

You may wish to provide a base path upfront. It can be changed at any time (press _o_ for _Options_).

For local files that could be for example:

    # Linux
    pyspark-explorer file:///home/myuser/datafiles/base_path
    # Windows
    pyspark-explorer file:/c:/datafiles/base_path

For remote location:

    # Remote hdfs cluster
    pyspark-explorer hdfs://somecluster/datafiles/base_path

Default path is set to /, which represents local root filesystem and works fine even in Windows thanks to Spark logics.

Configuration files are saved to your home directory (_.pyspark-explorer_ subdirectory). 
These are json files so you are free to edit them.