24 Commits (db5f1ea498f736d4ff733bbd41ec649d073cf932)

Author	SHA1	Message	Date
Wazir Ahmed	b5807fe141	RAG: Improve system prompt, replace hardcoded values with env vars ... Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	2 months ago
Wazir Ahmed	09bd69eed2	RAG: Improve system prompt, include SQLite server interface ... Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	2 months ago
Rahim Kanji	02f2ff5e2c	Fix logging system: remove stderr bypass, thread-safe timestamps, and std::tolower UB ... - Remove std::cerr calls that bypassed --log-level (no more duplicate output) - Use localtime_r for thread-safe timestamp formatting - Fix std::tolower undefined behavior with unsigned char cast - Minor: add markdown language tag, standardize CLI option docs	2 months ago
Rahim Kanji	79ee743a42	Add detailed logging and per-source commits to rag_ingest ... - Implement Logger class with 5 levels (error/warn/info/debug/trace) - Add --log-level option to control verbosity - Add comprehensive logging throughout ingestion pipeline - Change transaction handling: commit per-source instead of all-in-one - Update INGEST_USAGE_GUIDE.md with logging documentation	2 months ago
Rahim Kanji	e9cabc0d3b	Merge remote-tracking branch 'v4.0' into v4.0_rag_ingest_sqlite_server	2 months ago
Rahim Kanji	8a1732d15e	Add SQLite Server verification on connect ... Add verify_sqlite_server() method to MySQLDB class that queries sqlite_master table to confirm the connected server is ProxySQL SQLite3 Server. If verification fails, exit with a helpful error message explaining that rag_ingest writes RAG index data to SQLite Server (default port 6030), not a regular MySQL server. This prevents accidental connection to wrong backend which could cause confusing errors later during RAG index operations.	2 months ago
Rahim Kanji	bc70988935	Added Chunking and Embedding guide	2 months ago
Wazir Ahmed	591f4bc9b8	RAG: Add system prompt for agentic workflow ... Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	2 months ago
Rahim Kanji	1b322ed2bb	add comprehensive test suite for SQLite Server via MySQL protocol ... Add test_rag_ingest_sqlite_server.sh with 5 test phases: - Phase 1: Basic ingestion and watermark sync - Phase 2: WHERE filter support - Phase 3: Chunking support - Phase 4: Stub embeddings - Phase 5: OpenAI-compatible embeddings with vector search Tests rag_ingest tool connecting to ProxySQL SQLite3 Server (port 6030) using MySQL protocol, with backend MySQL as data source. Added documentation	2 months ago
Rahim Kanji	d66248eb4c	add verbose logging for bulk embedding operations ... Add logs to track embedding generation progress: - Show batch size when collecting chunks for embedding - Display OpenAI API call details (endpoint, model, chunk count) - Confirm successful storage after each batch	2 months ago
Rahim Kanji	211179bdcb	rag_ingest: Rewrite to use MySQL protocol (Sqlite_Server) instead of SQLite3 API ... Convert rag_ingest from direct SQLite database access to MySQL protocol communication with SQLite Server. Major changes: - Replace SQLite3 direct API with MySQL/mariadb client library - Update connection flow: rag_ingest -> MySQL protocol -> SQLite Server -> RAG DB - Remove sqlite3 amalgamation dependency from rag_ingest target - Keep all ingestion functionality intact (chunking, FTS5, vec0 embeddings) Schema initialization: - Add init command to create RAG schema via MySQL protocol - Add ingest command to run data ingestion pipeline - Add query command for vector similarity search with embeddings	2 months ago
Rene Cannao	35b316d4f3	Fix absolute paths in documentation ... Replace absolute paths (/home/rene/pr5312/proxysql/) with relative paths from repository root for portability.	2 months ago
Rene Cannao	a5edd4ad97	Add comprehensive RAG ingestion usage guide ... Document step-by-step usage of rag_ingest tool including: - Quick start instructions - Step-by-step setup guide - Common configurations (basic, chunking, embeddings) - Configuration reference (chunking_json, embedding_json, doc_map_json) - Common workflows (initial setup, re-run, add embeddings) - Troubleshooting tips This guide complements EMBEDDING_TEST_PLAN.md which focuses specifically on testing embeddings.	2 months ago
Rene Cannao	5fd779464d	Add embedding testing plan documentation ... Document how to test RAG ingestion with embeddings using external OpenAI-compatible services. Users only need to set 4 environment variables (OPENAI_API_BASE, OPENAI_API_KEY, OPENAI_MODEL, OPENAI_EMBEDDING_DIM) and run the test script. Related to issue #5320 (Batch embedding generation)	2 months ago
Rene Cannao	ff50e834f0	Implement batch embedding generation in rag_ingest ... Previously, embeddings were generated one chunk at a time, with each chunk requiring a separate HTTP API call. This was inefficient due to HTTP call overhead (latency). Changes: - Add PendingEmbedding struct to hold chunk metadata for batching - Add flush_embedding_batch() function to process multiple embeddings in a single API call - Modify ingestion loop to collect chunks and flush batches when full - Use the existing batch_size configuration from embedding_json (default: 16 chunks per batch) Performance improvement: - Before: 1000 chunks = 1000 HTTP calls (e.g., 100ms each = 100 seconds) - After: 1000 chunks = ~63 HTTP calls (1000 / 16 = ~63 batches) This significantly reduces ingestion time for sources with embeddings enabled.	2 months ago
Rene Cannao	950f163bfb	Fix compilation issues: use static libcurl and improve includes ... - Add curl include path from deps/curl/curl/include - Link against static libcurl.a from deps - Add zlib dependency (-lz) required by libcurl - Use "" style includes for local headers (sqlite3.h, mysql.h, crypt.h, curl/curl.h) - Rename sqlite_bind_text helper to bind_text to avoid collision with sqlite3_bind_text API - Add comprehensive Doxygen documentation to rag_ingest.cpp	2 months ago
Rene Cannao	38e5e8e56b	Fix critical issues from coderabbitai review ... - Fix NULL pointer dereference in rag_ingest.cpp: use str_or_empty() helper for all sqlite3_column_text results assigned to std::string - Fix NULL tags/links crash in MySQL_Catalog.cpp: add null guards before assigning sqlite3_column_text results to std::string - Fix missing curl_global_cleanup on error path in rag_ingest.cpp - Fix std::out_of_range exception in rag_ingest.cpp: wrap std::stoll calls in try-catch blocks, fall back to string comparison on overflow	2 months ago
Rahim Kanji	bc33c33295	Add vec embedding handling with stub and OpenAI providers ... Added rag_sync_state implementation	2 months ago
Rahim Kanji	d47e196fc6	Added rag_chunks and rag_fts_chunks test	3 months ago
Rahim Kanji	2f6b058f7b	Added test_rag_ingest.sh	3 months ago
Rahim Kanji	fb3673dd9d	added sample sql files	3 months ago
Rahim Kanji	57d8c3f3b8	Make rag_ingest compile	3 months ago
Rahim Kanji	93517540a5	Added Makefile for rag_ingest	3 months ago
Rene Cannao	803115f504	Add RAG capability blueprint documents ... These documents serve as blueprints for implementing RAG (Retrieval-Augmented Generation) capabilities in ProxySQL: - schema.sql: Database schema for RAG implementation - rag_ingest.cpp: PoC ingester blueprint to be integrated into ProxySQL - architecture-data-model.md: Data model architecture for RAG - architecture-runtime-retrieval.md: Runtime retrieval architecture - mcp-tools.md: MCP tools integration design - sql-examples.md: SQL usage examples for RAG - embeddings-design.md: Embeddings design for vector search These files will guide the upcoming RAG implementation in ProxySQL.	3 months ago