Metadata-Version: 2.1
Name: jupyter-openbis-server
Version: 0.4.2rc0
Summary: Server Extension for Jupyter notebooks to connect to openBIS and download/upload datasets, inluding the notebook itself
Home-page: https://sissource.ethz.ch/sispub/jupyter-openbis-server
Author: Swen Vermeul |  ID SIS | ETH Zürich
Author-email: swen@ethz.ch
License: Apache Software License Version 2.0
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: JavaScript
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.3
Description-Content-Type: text/markdown
License-File: LICENSE

# Jupyter openBIS Server

This server is an extension to the Jupyter notebook server and is part of the `jupyter-openbis-extension` and `jupyterlab-openbis` notebook extensions. It uses the `pyBIS` module internally to communicate with openBIS and ommunicates with the notebook extensions via the built-in tornado webserver.

This extension has been successfully tested with Safari 12.0.3, Chrome 72.0 and Firefox 66.0. There is a known incompatibility before Firefox 61.0b13 with Tornado > 6.x (the webserver used by Jupyter). If you encounter such incompatibilities, try to downgrade to Tornado 5.1.1. However, Tornado 5.1.1 will not work with Jupyter Lab 3.

## Install the server extension

The server extension will be automatically installed when you install the Jupyter Notebook Extension (the «classic» Jupyter Notebook):

```
$ pip install --upgrade jupyter-openbis-extension
```

If you need to install or upgrade the server extension alone, you can do so by:

```
$ pip install --upgrade jupyter-openbis-server
```

Make sure your extension is recognised by Jupyter and enabled (your output may vary, but ensure you see the line: `jupyter-openbis-server.server OK`)

```
$ jupyter serverextension list
config dir: /Users/your_username/.pyenv/versions/3.6.9/etc/jupyter
    jupyter-openbis-server.server  enabled
    - Validating...
      jupyter-openbis-server.server OK
```

## Create a connection configuration file

When the module is installed, you can create a connection configuration file.

```bash
$ jupyter-openbis-conn --help
Usage: jupyter-openbis [OPTIONS]

  Generate an openBIS connection file for use in Jupyter notebooks.

Options:
  -n, --name TEXT
  -h, --hostname TEXT
  --verfiy / --no-verify
  --https / --no-https
  -u, --username TEXT
  -p, --password TEXT
  -d, --destination [/Users/your_username/.jupyter|/Users/your_username/.pyenv/versions/3.6.9/etc/jupyter|/usr/local/etc/jupyter|/etc/your_username]
                                  [default: /Users/your_username/.jupyter]
  --help                          Show this message and exit.
```

You can start the utility as-is to get prompted for every parameter. Username and password are optional.

```
$ jupyter-openbis-conn
```

## install Jupyter extension manually

In most cases, a simple `pip install --upgrade jupyter-openbis-server` will install the server extension. However, in some cases (e.g. when installing via `pip install -e .`) you need to issue the following command to register the extension:

**In the library path, e.g. `/etc/jupyter/`**

```
$ jupyter serverextension enable --py jupyter-openbis-server --sys-prefix
```

This will create a file `~/.jupyter/jupyter_notebook_config.json` with the following content:

```
{
  "NotebookApp": {
    "nbserver_extensions": {
      "jupyter-openbis-server.server": true
    }
  }
}
```

## Uninstall Jupyter openBIS Server

Unfortunately, `pip` doesn't automatically clean up the Jupyter configuration when uninstalling. You have to do it yourself:

```
$ jupyter serverextension disable --py jupyter-openbis-server
$ pip uninstall jupyter-openbis-server
```

## Server extension API documentation

### XSRF Token in `POST`, `PUT` and `DELETE` requests

XSRF (or CSRF) stands for Cross-Site-Request-Forgery.

For all **POST**, **PUT** and **DELETE** requests, the following **http headers** must be submitted as http headers:

```
"X-XSRFToken": xsrf_token,
"credentials": "same-origin"
```

The value of the `xsrf_token` is the value of the `_xsrf` cookie which is stored in the users' browser. Without this http header information, the request will fail. All **GET** requests can be established without a special header.

The underlying Tornado-Webserver which handles all requests to the Jupyter serverextension will throw an error if the X-XSRF Token is not present.

### Errors

Errors caused by a `POST`, `PUT` and `DELETE` request will result in a HTTP Status > 300 and an error message:

```
{
	"reason": "Incorrect username or password for openBIS instance"
}
```

### get openBIS connections

**GET `/openbis/conns`**

Returns an array of JSON objects:

```
{
  "status": 200,
  "connections": [
    {
      "name": "openBIS instance",
      "url": "https://openbis.instance.ch",
      "status": "connected",
      "username": "user_name",
      "password": "******",
      "isMounted": false,
      "mountpoint": ""
    }
  ],
  "notebook_dir": "/home/user_name/project_dir"
}
```

- the **`name`** is the name of the connection being used when downloading or uploading dataSets (see below)
- the **`url`** of the openBIS instance
- the values of `status` can be either **connected** or **not connected**
- the **`username`** being used in openBIS
- the **`password`** really only consists of a number of asteriks **\***. If they are passed as such to re-connect to openBIS, the server tries to use the internally saved password instead. The password only lives in memory of the singleuser notebook-server and is not saved persistently.
- **`isMounted`** is either **true** or **false**, depending whether there is a current FUSE/SSHFS mountpoint available which connects to the openBIS dataStore
- `mountpoint` is the path to the mounted openBIS dataStore. It defaults to `$HOME/<openbis hostname>`

### login to an openBIS connection

An openBIS connection that has to be established or has timed out: a new login has to take place.

**PUT `/openbis/conn`**

Body:

```
{
    "username": username,
    "password": password,
    "action": "login",
}
```

The `action` attribute defaults to `login`. Returns:

```
{
    "status": 200,
    "connection": {
        "name": "openBIS instance",
        "url": "https://openbis.instance.ch",
        "status": "connected",
        "username": "some_username",
        "password": "******",
        "isMounted": false,
        "mountpoint": ""
    }
}
```

### logout

Logs out from an openBIS instance, i.e. the token is invalidated. The mount might still persist, as it is a separate connection. The status changes from **connected** to **not connected**

**PUT `/openbis/conn`**

Body:

```
{
    "action": "logout",
}
```

Returns:

```
{
    "status": 200,
    "connection": {
        "name": "openBIS instance",
        "url": "https://openbis.instance.ch",
        "status": "not connected",
        "username": "some_username",
        "password": "******",
        "isMounted": true,
        "mountpoint": "/Users/some_username/openbis.instance.ch"
    }
}
```

### Mount to an openBIS dataStore

#### Prerequisites

On the Jupyter Server, FUSE/SSHFS must be installed beforehand (requires root privileges). For the actual mount to the openBIS dataStore, no special privileges are required.

For **Mac OS X**, follow the installation instructions on [https://osxfuse.github.io](https://osxfuse.github.io)

For **Unix Cent OS 7**, do the following:

```
$ sudo yum install epel-release
$ sudo yum --enablerepo=epel -y install fuse-sshfs
$ user="$(whoami)"
$ usermod -a -G fuse "$user"
```

**Windows** is currently not supported, sorry!

By default, the mountpoint is the same as the hostname of the instance and it is located inside the home of the user. FUSE/SSHFS needs an empty directory to do this, so it will automatically be created.

**PUT `/openbis/conn`**

Body:

```
{
    "username": username,
    "password": password,
    "action"  : "mount"
}
```

Returns:

```
{
    "status": 200,
    "connection": {
        "name": "openBIS instance",
        "url": "https://openbis.instance.ch",
        "status": "connected",
        "username": "some_username",
        "password": "******",
        "isMounted": true,
        "mountpoint": "/Users/some_username/openbis.instance.ch"
    }
}
```

### Unmount from openBIS dataStore

**PUT `/openbis/conn`**

Body:

```
{
    "action"  : "mount"
}
```

Returns:

```
{
    "status": 200,
    "connection": {
        "name": "openBIS instance",
        "url": "https://openbis.instance.ch",
        "status": "connected",
        "username": "some_username",
        "password": "******",
        "isMounted": false,
        "mountpoint": ""
    }
}
```

### Register a new openBIS connection

For the lifetime (runtime) of the Jupyter server, this will create a connection to openBIS.

**POST `/openbis/conns`**

Body:

```
{
    "name": connection_name,
    "url": connection_url,
    "username": username,
    "password": password
}
```

### Unregister/delete a new openBIS connection

For the lifetime (runtime) of the Jupyter server, this will drop an existing openBIS connection:

**DELETE `/openbis/conn/<connection name>`**

### Upload a dataSet

**POST `/openbis/dataset/<connection_name>/<permId>/<downloadPath>`**

### Download a dataSet

**GET `/openbis/dataset/<connection_name>/<permId>/<downloadPath>`**

- the `connection_name` is the name of the connection given in the connections dialog.
- the `permId` is the identifer of the dataSet that needs to be downloaded.
- the `downloadPath` is the absolute path on the host system where the dataSet files should be downloaded to. The `downloadPath` must be URL-encoded to not to be confused with the URL itself.

In case of a **successful download**, the API returns a JSON like this

```
{
    'url'       : conn.url,
    'permId'    : dataset.permId,
    'path'      : path,
    'dataStore' : dataset.dataStore,
    'location'  : dataset.physicalData.location,
    'size'      : dataset.physicalData.size,
    'files'     : dataset.file_list,
    'statusText': 'Data for DataSet {} was successfully downloaded to: {}'.format(dataset.permId, path)
}
```

In case of an **error**, the API returns one of these errors (HTTP Status > 200):

**general connection error**

```
HTTP-Status: 500
{
	"reason": 'connection to {} could not be established: {}'.format(conn.name, exc)
}
```

**dataSet not found error**

```
HTTP-Status: 404
{
	"reason": 'No such dataSet found: {}'.format(permId)
}
```

**dataSet download error**

```
HTTP-Status: 500
{
	"reason": 'Data for DataSet {} could not be downloaded: {}'.format(permId, exc)
}
```

### Save `requirements.txt` and `runtime.txt` file

Note: The requirements list and the runtime must be evaluated by executing actual Python or R code from wtihin a notebook cell. The Python used by the Jupyter server might differ from the Python used by the kernel. The usual `pip freeze` doesn't work, as we cannot access the pip CLI from within Python.

For the Python `requirements.txt` we use this script:

```
import pkg_resources
print(
	"\n".join(
		["{}=={}".format(i.key, i.version) for i in pkg_resources.working_set]
	)
)
```

For the Python `runtime.txt`:

```
import sys
print('python-' + str(sys.version_info[0]) + '.' + str(sys.version_info[1]))
```

Once submitted to the server, the server will join the relative `notebook_path` (from the UI) with the server-side `notebook_dir`. These files will be stored in the same location on the filesystem as the notebook itself.

**POST `/openbis/requirements`**

Body:

```
{
    "notebook_path": notebook_path,
    "requirements_list": state.requirements_list,
    "requirements_filename": state.requirements_filename,
    "runtime": state.runtime,
    "runtime_filename": state.runtime_filename
}
```


