# sqlite-rembed Integration into ProxySQL

## Overview

This document describes the integration of the `sqlite-rembed` Rust SQLite extension into ProxySQL, enabling text embedding generation from remote AI APIs (OpenAI, Nomic, Ollama, Cohere, etc.) directly within ProxySQL's SQLite3 Server.

## What is sqlite-rembed?

`sqlite-rembed` is a Rust-based SQLite extension that provides:
- `rembed()` function for generating text embeddings via HTTP requests
- `temp.rembed_clients` virtual table for managing embedding API clients
- Support for multiple embedding providers: OpenAI, Nomic, Cohere, Ollama, Llamafile
- Automatic handling of API authentication, request formatting, and response parsing

## Integration Architecture

The integration follows the same pattern as `sqlite-vec` (vector search extension):

### Static Linking Approach
1. **Source packaging**: `sqlite-rembed-0.0.1-alpha.9.tar.gz` included in git repository
2. **Rust static library**: `libsqlite_rembed.a` built from extracted source
3. **Build system integration**: Makefile targets for tar.gz extraction and Rust compilation
4. **Auto-registration**: `sqlite3_auto_extension()` in ProxySQL initialization
5. **Single binary deployment**: No external dependencies at runtime

### Technical Implementation

```
ProxySQL Binary
├── C++ Core (libproxysql.a)
├── SQLite3 (sqlite3.o)
├── sqlite-vec (vec.o)
└── sqlite-rembed (libsqlite_rembed.a) ← Rust static library
```

## Build Requirements

### Rust Toolchain
```bash
# Required for building sqlite-rembed
rustc --version
cargo --version

# Development dependencies
clang
libclang-dev
```

### Build Process
1. Rust toolchain detection in `deps/Makefile`
2. Extract `sqlite-rembed-0.0.1-alpha.9.tar.gz` from GitHub release
3. Static library build with `cargo build --release --features=sqlite-loadable/static --lib`
4. Linking into `libproxysql.a` via `lib/Makefile`
5. Final binary linking via `src/Makefile`

### Packaging
Following ProxySQL's dependency packaging pattern, sqlite-rembed is distributed as a compressed tar.gz file:
- `deps/sqlite3/sqlite-rembed-0.0.1-alpha.9.tar.gz` - Official GitHub release tarball
- Extracted during build via `tar -zxf sqlite-rembed-0.0.1-alpha.9.tar.gz`
- Clean targets remove extracted source directories

## Code Changes Summary

### 1. `deps/Makefile`
- Added Rust toolchain detection (`rustc`, `cargo`)
- SQLite environment variables for sqlite-rembed build
- New target: `sqlite3/libsqlite_rembed.a` that extracts from tar.gz and builds
- Added dependency to `sqlite3` target
- Clean targets remove `sqlite-rembed-*/` and `sqlite-rembed-source/` directories

### 2. `lib/Makefile`
- Added `SQLITE_REMBED_LIB` variable pointing to static library
- Library included in `libproxysql.a` dependencies (via src/Makefile)

### 3. `src/Makefile`
- Added `SQLITE_REMBED_LIB` variable
- Added `$(SQLITE_REMBED_LIB)` to `LIBPROXYSQLAR` dependencies

### 4. `lib/Admin_Bootstrap.cpp`
- Added `extern "C" int sqlite3_rembed_init(...)` declaration
- Added `sqlite3_auto_extension((void(*)(void))sqlite3_rembed_init)` registration
- Registered after `sqlite-vec` initialization

## Usage Examples

### Basic Embedding Generation
```sql
-- Register an OpenAI client
INSERT INTO temp.rembed_clients(name, format, model, key)
VALUES ('openai_client', 'openai', 'text-embedding-3-small', 'your-api-key');

-- Generate embedding
SELECT rembed('openai_client', 'Hello world') as embedding;

-- Use with vector search
CREATE VECTOR TABLE docs (embedding float[1536]);
INSERT INTO docs(rowid, embedding)
VALUES (1, rembed('openai_client', 'Document text here'));

-- Search similar documents
SELECT rowid, distance FROM docs
WHERE embedding MATCH rembed('openai_client', 'Query text');
```

### Multiple API Providers
```sql
-- OpenAI
INSERT INTO temp.rembed_clients(name, format, model, key, url)
VALUES ('gpt', 'openai', 'text-embedding-3-small', 'sk-...');

-- Ollama (local)
INSERT INTO temp.rembed_clients(name, format, model, url)
VALUES ('ollama', 'ollama', 'nomic-embed-text', 'http://localhost:11434');

-- Cohere
INSERT INTO temp.rembed_clients(name, format, model, key)
VALUES ('cohere', 'cohere', 'embed-english-v3.0', 'co-...');

-- Nomic
INSERT INTO temp.rembed_clients(name, format, model, key)
VALUES ('nomic', 'nomic', 'nomic-embed-text-v1.5', 'nm-...');
```

## Configuration

### Environment Variables (for building)
```bash
export SQLITE3_INCLUDE_DIR=/path/to/sqlite-amalgamation
export SQLITE3_LIB_DIR=/path/to/sqlite-amalgamation
export SQLITE3_STATIC=1
```

### Runtime Configuration
- API keys: Set via `temp.rembed_clients` table
- Timeouts: Handled by underlying HTTP client (ureq)
- Model selection: Per-client configuration

## Error Handling

The extension provides SQLite error messages for:
- Missing client registration
- API authentication failures
- Network connectivity issues
- Invalid input parameters
- Provider-specific errors

## Performance Considerations

### HTTP Latency
- Embedding generation involves HTTP requests to remote APIs
- Consider local embedding models (Ollama, Llamafile) for lower latency
- Batch processing not currently supported (single text inputs only)

### Caching
- No built-in caching layer
- Applications should cache embeddings when appropriate
- Consider database-level caching with materialized views

## Limitations

### Current Implementation
1. **Blocking HTTP requests**: Synchronous HTTP calls may block SQLite threads
2. **Single text input**: `rembed()` accepts single text string, not batches
3. **No async support**: HTTP requests are synchronous
4. **Rust dependency**: Requires Rust toolchain for building ProxySQL

### Security Considerations
- API keys stored in `temp.rembed_clients` table (in-memory, per-connection)
- Network access required for remote APIs
- No encryption of API keys in transit (use HTTPS endpoints)

## Testing

### Build Verification
```bash
# Clean and rebuild with tar.gz extraction
cd deps && make cleanpart && make sqlite3

# Verify tar.gz extraction and Rust library build
ls deps/sqlite3/sqlite-rembed-source/
ls deps/sqlite3/libsqlite_rembed.a

# Verify symbol exists
nm deps/sqlite3/libsqlite_rembed.a | grep sqlite3_rembed_init
```

### Functional Testing
```sql
-- Test extension registration
SELECT rembed_version();
SELECT rembed_debug();

-- Test client registration
INSERT INTO temp.rembed_clients(name, format, model)
VALUES ('test', 'ollama', 'nomic-embed-text');

-- Test embedding generation (requires running Ollama)
-- SELECT rembed('test', 'test text');
```

## Future Enhancements

### Planned Improvements
1. **Async HTTP**: Non-blocking requests using async Rust
2. **Batch processing**: Support for multiple texts in single call
3. **Embedding caching**: LRU cache for frequently generated embeddings
4. **More providers**: Additional embedding API support
5. **Configuration persistence**: Save clients across connections

### Integration with sqlite-vec
- Complete AI pipeline: `rembed()` → vector storage → `vec_search()`
- Example: Document embedding and similarity search
- Potential for RAG (Retrieval-Augmented Generation) applications

## Troubleshooting

### Build Issues
1. **Missing clang**: Install `clang` and `libclang-dev`
2. **Rust not found**: Install Rust toolchain via `rustup`
3. **SQLite headers**: Ensure `sqlite-amalgamation` is extracted

### Runtime Issues
1. **Client not found**: Verify `temp.rembed_clients` entry exists
2. **API errors**: Check API keys, network connectivity, model availability
3. **Memory issues**: Large embeddings may exceed SQLite blob limits

## References

- [sqlite-rembed GitHub](https://github.com/asg017/sqlite-rembed)
- [sqlite-vec Documentation](../doc/SQLite3-Server.md)
- [SQLite Loadable Extensions](https://www.sqlite.org/loadext.html)
- [Rust C FFI](https://doc.rust-lang.org/nomicon/ffi.html)

### Source Distribution
- `deps/sqlite3/sqlite-rembed-0.0.1-alpha.9.tar.gz` - Official GitHub release tarball
- Extracted to `deps/sqlite3/sqlite-rembed-source/` during build

## Maintainers

- Integration: [Your Name/Team]
- Original sqlite-rembed: [Alex Garcia (@asg017)](https://github.com/asg017)
- ProxySQL Team: [ProxySQL Maintainers](https://github.com/sysown/proxysql)

## License

- sqlite-rembed: Apache 2.0 / MIT (see `deps/sqlite3/sqlite-rembed-source/LICENSE-*`)
- ProxySQL: GPL v3
- Integration code: Same as ProxySQL