Metadata-Version: 2.1
Name: llama-index-llms-mistral-rs
Version: 0.2.1
Summary: llama-index llms mistral-rs integration
License: MIT
Author: EricLBuehler
Maintainer: jerryjliu
Requires-Python: >=3.8.1,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: llama-index-core (>=0.11.0,<0.12.0)
Description-Content-Type: text/markdown

# LlamaIndex Llms Integration: `mistral.rs`

To use this integration, please install the Python `mistralrs` package:

## Installation of `mistralrs` from PyPi

0. Install Rust: https://rustup.rs/

   ```bash
   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   source $HOME/.cargo/env
   ```

1. `mistralrs` depends on the `openssl` library.

To install it on Ubuntu:

```
sudo apt install libssl-dev
sudo apt install pkg-config
```

2. Install it!

- CUDA

  `pip install mistralrs-cuda`

- Metal

  `pip install mistralrs-metal`

- Apple Accelerate

  `pip install mistralrs-accelerate`

- Intel MKL

  `pip install mistralrs-mkl`

- Without accelerators

  `pip install mistralrs`

All installations will install the `mistralrs` package. The suffix on the package installed by `pip` only controls the feature activation.

## Installation from source

Please follow the instructions [here](https://github.com/EricLBuehler/mistral.rs/blob/master/mistralrs-pyo3/README.md).

## Usage

```python
from llama_index.llms.mistral_rs import MistralRS
from mistralrs import Which

llm = MistralRS(
    which=Which.GGUF(
        tok_model_id="mistralai/Mistral-7B-Instruct-v0.1",
        quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
        quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
        tokenizer_json=None,
        repeat_last_n=64,
    ),
    max_new_tokens=4096,
    context_window=1024 * 5,
)
```

