Metadata-Version: 2.1
Name: llama-rs-python
Version: 0.0.2
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
License-File: LICENSE
Summary: Unofficial python bindings for llama-rs. 🐍❤️🦀
Author: Lukas Kreussel
Requires-Python: >=3.7
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Bug Tracker, https://github.com/LLukas22/llama-rs-python/issues
Project-URL: Homepage, https://github.com/LLukas22/llama-rs-python

# llama-rs-python
Unofficial python bindings for [llama-rs](https://github.com/rustformers/llama-rs) created with [PyO3](https://github.com/PyO3/pyo3). 🐍❤️🦀

This package gives access to the basic functionality of the llama-rs project.

GGML converted models can be loaded and executed.

## Installation

Simply install it via pip: `pip install llama-rs-python`

## Usage

The package is typehinted for easy usage.

A usage example could look like this:

```python 
from llama_rs_python import Model

#load the model
model = Model("path/to/model.bin")

#generate
print(model.generate("The meaning of life is"))
```

The package also supports callbacks to get each token as it is generated.
The callback-function also supports canceling the generation by returning a `True` value from the pytohn side.

```python 
from llama_rs_python import Model
import sys
from typing import Optional

#load the model
model = Model("path/to/model.bin")

#define the callback
def callback(token:str)->Optional[bool]:
    print(token,end="")
    sys.stdout.flush()
    # (return True here to cancel the generation)

#start generation
model.generate("The meaning of life is",callback=callback)
```

The configuration of the generation is handled by the `GenerationConfig` class.

```python 
from llama_rs_python import Model, GenerationConfig

#load the model
model = Model("path/to/model.bin")

#create a config
config = GenerationConfig(top_p=0.9,seed=1441,max_new_tokens=1024)

#generate
print(model.generate("The meaning of life is",generation_config=config))
```

To configure model specific settings the `SessionConfig` class can be used.

```python
from llama_rs_python import Model, SessionConfig

#define the session
session_config = SessionConfig(threads=8,context_length=512)

#load the model
model = Model("path/to/model.bin",session_config=session_config)
```
