Metadata-Version: 2.1
Name: splatnet3-scraper
Version: 0.5.3
Summary: Scraper for SplatNet 3 for Splatoon 3
License: GPL-3.0-or-later
Author: Cesar E Garza
Author-email: cesar@cegarza.com
Requires-Python: >=3.10,<4.0
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Provides-Extra: all
Provides-Extra: examples
Provides-Extra: parquet
Requires-Dist: pandas[examples] (>=1.5.3,<2.0.0) ; extra == "examples"
Requires-Dist: psycopg2[examples] (>=2.9.5,<3.0.0) ; extra == "examples"
Requires-Dist: pyarrow[parquet] (>=10.0.1,<11.0.0) ; extra == "parquet"
Requires-Dist: requests (>=2.28.1,<3.0.0)
Requires-Dist: sqlalchemy[examples] (>=2.0.1,<3.0.0) ; extra == "examples"
Description-Content-Type: text/markdown

# SplatNet 3 Scraper

[![Tests Status](./reports/junit/tests-badge.svg?dummy=8484744)](https://htmlpreview.github.io/?https://github.com/cesaregarza/SplatNet3_Scraper/blob/main/reports/junit/report.html) ![Coverage Status](./reports/coverage/coverage-badge.svg?dummy=8484744) ![Flake8 Status](./reports/flake8/flake8-badge.svg?dummy=8484744) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

**SplatNet 3 Scraper** is a Python library for scraping data from the Splatoon 3 SplatNet 3 API. It is designed to be as lightweight as possible, with minimal dependencies to make it easy to integrate into other projects.

**SplatNet 3 Scraper** started as a fork of **[s3s](https://github.com/frozenpandaman/s3s)**, but has since been rewritten from scratch while incorporating much of the login flow logic of s3s. As a result, I am deeply indebted to the authors of s3s for their work. This project would not have been possible without their efforts.

## Features

* Lightweight and minimal dependencies. Only requires the `requests` library. Requires Python 3.10 or later.
* The `scraper` module provides a user-level API that enables a quick and easy way to get data from the SplatNet 3 API, only requiring the user to provide their session token.
* The `query` module provides a high-level API that provides a simple way to make queries to the SplatNet 3 API. It automatically handles authentication and query handling, and provides a simple interface for accessing the response data.
* The `auth` module provides a low level API that allows for more fine-grained control over the scraping process. It greatly simplifies the process of authentication.
* Configuration file support is compatible with the configuration file format used by `s3s`.
* Responses from the SplatNet 3 API can be saved and loaded from disk, currently supporting the following formats:
  * JSON
  * gzip-compressed JSON
  * csv
  * parquet (by installing `splatnet3_scraper[parquet]` or the `pyarrow` library)

## Installation

**SplatNet 3 Scraper** is currently under active development but is currently available on PyPI. It can be installed using pip:

```bash
pip install splatnet3_scraper
```

Note that the current versions of **SplatNet 3 Scraper** are currently `v0.x.y`, which means that the API is not guaranteed to be stable and may change at any moment. As such, it is highly recommended that you pin the version of **SplatNet 3 Scraper** that you are using until the API is stabilized with the release of `v1.0.0`.

## Usage

There are three ways to use **SplatNet 3 Scraper**. The first is to use the `scraper` module, which provides a top-level API that greatly simplifies the process of retrieving commonly requested data from SplatNet 3. This module is greatly recommended for most users. The second is to use the `query` module, which provides a high-level API that provides a simple interface to make queries to the SplatNet 3 API. This module is recommended for developers who can't find what they need with the `scraper` module. The third is to use the `auth` module, which provides a low-level API that gives the user the most control over the scraping process. This module is recommended for advanced developers who need to have full control over the authentication process. Whichever module you choose to use, all of them require providing a session token.

### Using the `scraper` module

The `scraper` module is by far the easiest way to get data from the SplatNet 3 API and the module that is recommended for most users, especially those who are not highly experienced with Python. The `scraper` module provides multiple functions that can be used to retrieve commonly requested data from the SplatNet 3 API. The `scraper` module is designed to be used by users who are not highly experienced with Python or users who do not need to have full control over the scraping process.

This module is currently under active development and is not yet complete. Please check back later for more functions.

### Using the `query` module

The `query` module is an easy-to-use module that enables fast and painless querying to the SplatNet 3 API. It handles authentication and query handling automagically, and provides a simple interface for accessing the response data. The `query` module is designed to be used by advanced users who need more control over the queries they make to the SplatNet 3 API. If you are looking for a simple way to get data from the SplatNet 3 API, you should use the `scraper` module instead.

The `query` module provides the `QueryHandler` class, which is used to make queries to the SplatNet 3 API. The `QueryHandler` class can be instantiated in one of a few ways: by providing a session token, by providing the path to a configuration file, or by loading environment variables.

#### Instantiating the `QueryHandler` class by providing a session token

```python
from splatnet3_scraper.query import QueryHandler
handler = QueryHandler.from_session_token("session_token")
handler.query("StageScheduleQuery")
```

#### Instantiating the `QueryHandler` class by providing the path to a configuration file

```python
from splatnet3_scraper.query import QueryHandler
handler = QueryHandler.from_config_file(".splatnet3_scraper")
handler.query("StageScheduleQuery")
```

#### Instantiating the `QueryHandler` class by loading environment variables

The following environment variables are supported:

* `SN3S_SESSION_TOKEN`
* `SN3S_GTOKEN`
* `SN3S_BULLET_TOKEN`

```python
from splatnet3_scrape.query import QueryHandler
handler = QueryHandler.from_env()
handler.query("StageScheduleQuery")
```

#### Querying the SplatNet 3 API

The `QueryHandler` class provides a `query` method that can be used to make queries to the SplatNet 3 API. The `query` method takes a single argument, which is the name of the query to make. The `query` method returns a `QueryResponse` object, which contains the response data from the SplatNet 3 API. The `QueryResponse` object provides a `data` property that can be used to access the response data. The `QueryResponse` module also supports numpy-style indexing, which can be used to quickly and clearly access specific parts of the response data. For example, the following code will print the game mode name of the the current stage rotation schedule:

```python
from splatnet3_scraper.query import QueryHandler
handler = QueryHandler.from_env()
response = handler.query("StageScheduleQuery")
print(response["xSchedules", "nodes", 0, "vsRule", "name"])
```

#### Saving and loading responses

The `QueryResponse` class provides a `parsed_json` method that can be used to generate a `JSONParser` object from the response data. The `JSONParser` class provides multiple ways of interacting with the given data, including the ability to save the data to disk in a variety of formats. There are currently four different formats that are supported and can be used by passing the desired format to a `to_*` method such as `to_json`. The following formats are supported:

* JSON
* gzip-compressed JSON
* csv
* parquet (by installing `splatnet3_scraper[parquet]` or the `pyarrow` library)

Note: csv and parquet formats work by converting the response data from a nested dictionary to a columnar format. This is not recommended for single queries, but can be useful for interacting with large amounts of data as it deduplicates the JSON structure and allows for more efficient storage and querying.

The following code will save the response data to a file named `response.json` in the current directory:

```python
from splatnet3_scraper.query import QueryHandler
handler = QueryHandler.from_env()
response = handler.query("StageScheduleQuery")
response.parsed_json().to_json("response.json")
```

Additionally, the `JSONParser` class provides a `from_*` method that can be used to load data from a file. The following code will load the response data from the file `response.json` in the current directory:

```python
from splatnet3_scraper.query import JSONParser
parser = JSONParser.from_json("response.json")
```

## Symbols

| Symbol | Meaning |
| ------ | ------- |
| :white_check_mark: | Implemented |
| :construction: | In progress |
| :world_map: | Planned |
| :x: | Not planned |

## Roadmap

| Feature | Status |
| ------- | ------ |
| Support for the SplatNet 3 API | :white_check_mark: |
| Full support for the SplatNet 3 API | :world_map: |
| Support for the SplatNet 2 API | :x: |
| Obtaining session tokens | :white_check_mark: |
| Full documentation | :construction: |
| Full unit test coverage | :white_check_mark: |
| Columnar data format support | :construction: |
| CLI interface | :x: |
| Integration with stat.ink | :x: |
| PyPI package | :white_check_mark: |
| Docker image | :world_map: |
| Executable binary | :x: |

## Docker Note

This project currently uses the standard library heavily, and as such it is not compatible with the `python:alpine` Docker image. I have no plans to change this. Use the `python:slim` image instead.

QueryHandler is licensed under the GPLv3. See the LICENSE file for more details.

