Metadata-Version: 2.1
Name: paramsurvey-tooling
Version: 0.3.3
Summary: A toolkit to make paramsurvey easy to use on clusters and cloud
Home-page: https://github.com/wumpus/paramsurvey_tooling
Author: Greg Lindahl and others
Author-email: lindahl@pbm.com
License: Apache 2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Operating System :: POSIX :: Linux
Classifier: Environment :: MacOS X
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.6.*
Description-Content-Type: text/markdown
Provides-Extra: test
Provides-Extra: package

# paramsurvey-tooling

## Install

```
pip install paramsurvey-tooling
```

## paramsurvey usage on clusters with batch queues

Make 3 batch scripts: head, driver, and child. They all have a similar header:

```
#FOO batch header lines

. ~/setup-my-software-environment.sh
```

The script to set up your environment should activate your Conda environment, or
load modules or whatever.

Then each of these scripts gets one of the following commands:

* `pstool start head`
* `pstool start driver ./my-script.py foo bar baz`
* `pstool start child`

Finally, submit the head job, then the driver, and finally a bunch of children.

Instead of using a driver batch script, you can also run the driver on a head
node:

```
pstool submit ./my-script.py foo bar baz  # NOT YET WORKING
```

## Containers and paramsurvey

Many compute clusters do not support Docker containers, for security reasons.
These clusters often do support Singularity containers, and it's not hard to
turn an arbitrary Docker container into a Singularity container. Build
the Docker container on a machine that does have docker installed, and
then export it to a file:

```
docker save IMAGE_ID | gzip > my_docker_image.tar.gz
```

Transfer that file to a host with `paramsurvey-tooling` and `singularity`
installed, and do:

```
pstool build my_docker_image.tar.gz
```

## If you find a bug

This tool attempts to support many systems that the author can not
directly test. If you find a bug in your environment, run this:

```
pstool debug > DEBUG
```

and attach this file `DEBUG` to your bug report.

## Documentation

Installing code in an environment:
* Conda
* OS packages
* modules

Containerized environments:
* Docker
* Singularity

Execution environment:
* [Laptop or a single server](Execution-Laptop.md)
* [Cluster with batch queue:](Execution-Cluster.md)
  * Philosophy
  * Details
    * slurm -- Harvard cluster
    * PBSPro -- ASIAA cluster
    * Grid Engine -- Smithsonian Hydra cluster
  * Tearing down
* Cloud-native:
  * Philosophy
  * Open Science Grid
    * https://bhpire.arizona.edu/2022/09/12/new-webinar-performing-large-scale-parameter-surveys-with-osg-services/
  * Ray Cluster on AWS or GCE
  * Ray Cluster over Kubernetes






