proxysql

Commit Graph

Author	SHA1	Message	Date
Rahim Kanji	02f2ff5e2c	Fix logging system: remove stderr bypass, thread-safe timestamps, and std::tolower UB - Remove std::cerr calls that bypassed --log-level (no more duplicate output) - Use localtime_r for thread-safe timestamp formatting - Fix std::tolower undefined behavior with unsigned char cast - Minor: add markdown language tag, standardize CLI option docs	3 months ago
Rahim Kanji	79ee743a42	Add detailed logging and per-source commits to rag_ingest - Implement Logger class with 5 levels (error/warn/info/debug/trace) - Add --log-level option to control verbosity - Add comprehensive logging throughout ingestion pipeline - Change transaction handling: commit per-source instead of all-in-one - Update INGEST_USAGE_GUIDE.md with logging documentation	3 months ago
Rahim Kanji	e9cabc0d3b	Merge remote-tracking branch 'v4.0' into v4.0_rag_ingest_sqlite_server	3 months ago
Rahim Kanji	8a1732d15e	Add SQLite Server verification on connect Add verify_sqlite_server() method to MySQLDB class that queries sqlite_master table to confirm the connected server is ProxySQL SQLite3 Server. If verification fails, exit with a helpful error message explaining that rag_ingest writes RAG index data to SQLite Server (default port 6030), not a regular MySQL server. This prevents accidental connection to wrong backend which could cause confusing errors later during RAG index operations.	3 months ago
Rahim Kanji	bc70988935	Added Chunking and Embedding guide	3 months ago
Wazir Ahmed	888eb24b14	Merge pull request #5328 from sysown/v4.0_rag_ingest_2	3 months ago
Wazir Ahmed	591f4bc9b8	RAG: Add system prompt for agentic workflow Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Rahim Kanji	1b322ed2bb	add comprehensive test suite for SQLite Server via MySQL protocol Add test_rag_ingest_sqlite_server.sh with 5 test phases: - Phase 1: Basic ingestion and watermark sync - Phase 2: WHERE filter support - Phase 3: Chunking support - Phase 4: Stub embeddings - Phase 5: OpenAI-compatible embeddings with vector search Tests rag_ingest tool connecting to ProxySQL SQLite3 Server (port 6030) using MySQL protocol, with backend MySQL as data source. Added documentation	3 months ago
Rahim Kanji	d66248eb4c	add verbose logging for bulk embedding operations Add logs to track embedding generation progress: - Show batch size when collecting chunks for embedding - Display OpenAI API call details (endpoint, model, chunk count) - Confirm successful storage after each batch	3 months ago
Rahim Kanji	211179bdcb	rag_ingest: Rewrite to use MySQL protocol (Sqlite_Server) instead of SQLite3 API Convert rag_ingest from direct SQLite database access to MySQL protocol communication with SQLite Server. Major changes: - Replace SQLite3 direct API with MySQL/mariadb client library - Update connection flow: rag_ingest -> MySQL protocol -> SQLite Server -> RAG DB - Remove sqlite3 amalgamation dependency from rag_ingest target - Keep all ingestion functionality intact (chunking, FTS5, vec0 embeddings) Schema initialization: - Add init command to create RAG schema via MySQL protocol - Add ingest command to run data ingestion pipeline - Add query command for vector similarity search with embeddings	3 months ago
René Cannaò	8ef8503d13	Merge pull request #5323 from wazir-ahmed/v4.0_rag_ingest_2 RAG MCP Fixes	3 months ago
Javier Jaramago Fernández	1a7668b73d	fix: SQLite3 server support for MySQL client '8.1.0' For more details check commit #ecf4f69.	3 months ago
René Cannaò	c5c726ea98	Merge pull request #5321 from sysown/v4.0-mcp_rules_test WIP-Fixes: Add/Improve TAP tests for MCP query_rules/digests	3 months ago
Rene Cannao	87d8c25074	Reapply "fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins'" This reverts commit `48260cbf09`.	3 months ago
René Cannaò	342272367d	Merge pull request #5324 from sysown/v4.0_rag_ingest_2 V4.0 rag ingest 2	3 months ago
Rene Cannao	35b316d4f3	Fix absolute paths in documentation Replace absolute paths (/home/rene/pr5312/proxysql/) with relative paths from repository root for portability.	3 months ago
Rene Cannao	a5edd4ad97	Add comprehensive RAG ingestion usage guide Document step-by-step usage of rag_ingest tool including: - Quick start instructions - Step-by-step setup guide - Common configurations (basic, chunking, embeddings) - Configuration reference (chunking_json, embedding_json, doc_map_json) - Common workflows (initial setup, re-run, add embeddings) - Troubleshooting tips This guide complements EMBEDDING_TEST_PLAN.md which focuses specifically on testing embeddings.	3 months ago
Wazir Ahmed	e4f4dc95ce	RAG: bm25 and MATCH do not work with table alias Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Wazir Ahmed	c52c621b27	MCP: Add mcp-rag_endpoint_auth config Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Wazir Ahmed	eb495f42ee	AI: Fix vector_db table creation Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Wazir Ahmed	22c4e94d53	AI: Fix sqlite-vec extension loading Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Wazir Ahmed	7167f95247	AI: Enable extensions for vector_db Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Wazir Ahmed	4031f85394	AI: Fix vector_db initialization Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Wazir Ahmed	cecb975f66	fix: LOAD GENAI TO RUNTIME does not initialize the module Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Wazir Ahmed	50de536534	MCP: Fix crash during server restarts Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Rene Cannao	48260cbf09	Revert "fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins'" This reverts commit `df221617c0`.	3 months ago
Javier Jaramago Fernández	51e91378c4	test: Add/Improve TAP tests for MCP query_rules/digests	3 months ago
Rene Cannao	5fd779464d	Add embedding testing plan documentation Document how to test RAG ingestion with embeddings using external OpenAI-compatible services. Users only need to set 4 environment variables (OPENAI_API_BASE, OPENAI_API_KEY, OPENAI_MODEL, OPENAI_EMBEDDING_DIM) and run the test script. Related to issue #5320 (Batch embedding generation)	3 months ago
Javier Jaramago Fernández	08e7c8ce97	chore: Remove data files mistakenly added to root folder	3 months ago
Javier Jaramago Fernández	df221617c0	fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins' Since 'proxy_sqlite3_open_v2' is now being set by default due to the new dedicated TU ('proxy_sqlite3_symbols.cpp'). 'LoadPlugins' might not be called, not properly setting 'SQLITE_CONFIG_URI' and invalidating many configuration that are currently specified in the conn URI.	3 months ago
Rene Cannao	ff50e834f0	Implement batch embedding generation in rag_ingest Previously, embeddings were generated one chunk at a time, with each chunk requiring a separate HTTP API call. This was inefficient due to HTTP call overhead (latency). Changes: - Add PendingEmbedding struct to hold chunk metadata for batching - Add flush_embedding_batch() function to process multiple embeddings in a single API call - Modify ingestion loop to collect chunks and flush batches when full - Use the existing batch_size configuration from embedding_json (default: 16 chunks per batch) Performance improvement: - Before: 1000 chunks = 1000 HTTP calls (e.g., 100ms each = 100 seconds) - After: 1000 chunks = ~63 HTTP calls (1000 / 16 = ~63 batches) This significantly reduces ingestion time for sources with embeddings enabled.	3 months ago
René Cannaò	6166adaf05	Merge pull request #5318 from rahim-kanji/v4.0_rag_ingest Add RAG ingestion with vector embeddings	3 months ago
Rene Cannao	07154ca661	Empty commit to differentiate from v4.0_rag_ingest_2	3 months ago
Rene Cannao	42a67ebaf2	Merge rahim/v4.0_rag_ingest into v4.0_rag_ingest_2 Merge changes from PR #5318 (RAG ingestion feature) into the new development branch v4.0_rag_ingest_2. Changes include: - RAG ingestion tool (rag_ingest) with chunking and embeddings - MySQL_Catalog fixes for NULL pointer handling - MCP server updates for RAG tools - Comprehensive documentation and test scripts	3 months ago
Rene Cannao	950f163bfb	Fix compilation issues: use static libcurl and improve includes - Add curl include path from deps/curl/curl/include - Link against static libcurl.a from deps - Add zlib dependency (-lz) required by libcurl - Use "" style includes for local headers (sqlite3.h, mysql.h, crypt.h, curl/curl.h) - Rename sqlite_bind_text helper to bind_text to avoid collision with sqlite3_bind_text API - Add comprehensive Doxygen documentation to rag_ingest.cpp	3 months ago
Rene Cannao	38e5e8e56b	Fix critical issues from coderabbitai review - Fix NULL pointer dereference in rag_ingest.cpp: use str_or_empty() helper for all sqlite3_column_text results assigned to std::string - Fix NULL tags/links crash in MySQL_Catalog.cpp: add null guards before assigning sqlite3_column_text results to std::string - Fix missing curl_global_cleanup on error path in rag_ingest.cpp - Fix std::out_of_range exception in rag_ingest.cpp: wrap std::stoll calls in try-catch blocks, fall back to string comparison on overflow	3 months ago
Rene Cannao	9ba3df0ce7	Address AI code review feedback from PR #5318 - Fix Makefile: Use $(CXXFLAGS) directly for consistency with build philosophy - Fix MySQL_Catalog: Return proper error JSON instead of empty array on missing query	3 months ago
René Cannaò	1f38d9210c	Merge pull request #5313 from sysown/pr-5312-fixes Fix security issues identified in PR #5312 code review	3 months ago
Rahim Kanji	d28444a02e	Merge remote-tracking branch 'v4.0' into v4.0_rag_ingest	3 months ago
Rahim Kanji	bc33c33295	Add vec embedding handling with stub and OpenAI providers Added rag_sync_state implementation	3 months ago
Rene Cannao	3ccfa2bcc1	Address AI code review feedback for PR #5313 This commit addresses concerns raised by AI code reviewers (gemini-code-assist, Copilot, coderabbitai) on the initial security fixes. Critical fixes: - Fix lock.release() → lock.unlock() in GenAI_Thread.cpp worker_loop (lock.release() detaches without unlocking, causing deadlock) - Add missing early return after schema validation failure in Query_Tool_Handler.cpp Code quality improvements: - Improve escape_string() memory management in MySQL_Tool_Handler.cpp: - Use std::string instead of new[]/delete[] for buffer management - Check return value of mysql_real_escape_string() for errors - Remove redundant validation checks in validate_sql_identifier functions (character class loop already rejects unsafe characters) - Add backslash escaping to escape_string_literal() for defense-in-depth - Improve column list validation in MySQL_Tool_Handler sample_rows(): - Replace blacklist approach with proper column identifier parsing - Allow qualified identifiers (table.column) - Allow AS aliases (column AS alias) - No longer rejects legitimate column names containing "JOIN" These changes improve robustness while maintaining the security posture of the original SQL injection fixes.	3 months ago
Rene Cannao	c914feb230	Fix security issues identified in PR #5312 code review This commit addresses critical and important security vulnerabilities found during comprehensive code review of the Gen AI features merge. Critical fixes: - SQL injection vulnerabilities in MySQL_Tool_Handler.cpp: - Added validate_sql_identifier() for schema/table validation - Added escape_string() for MySQL string escaping using mysql_real_escape_string - Fixed list_tables(), describe_table(), sample_rows(), sample_distinct() - SQL injection vulnerabilities in Query_Tool_Handler.cpp: - Added validate_sql_identifier_sqlite() for identifier validation - Added escape_string_literal() for SQLite string escaping - Fixed list_tables tool and catalog.get_relationships function - Use-after-free race condition in GenAI_Thread: - Changed shutdown_ from int to std::atomic<int> for proper memory ordering - Added additional shutdown check in worker_loop after popping request Important fixes: - Buffer overflow risks from sprintf usage: - Converted all sprintf() calls to snprintf() in GenAI_Thread.cpp - Converted sprintf() to snprintf() in MySQL_Session.cpp - Worker loop shutdown race condition: - Added shutdown check after popping request from queue - Properly clean up client_fd when shutdown is detected These fixes ensure: 1. All user input is properly validated before use in SQL queries 2. String values are properly escaped using database-specific escaping 3. Thread-safe shutdown with proper memory ordering guarantees 4. Bounds-safe string formatting to prevent buffer overflows	3 months ago
René Cannaò	4075e7cf62	Merge pull request #5312 from ProxySQL/v3.1-vec Port of Gen AI commits to proxysql (2nd attempt)	3 months ago
René Cannaò	19d29a7346	Merge pull request #28 from ProxySQL/v3.1-vec-loadplugin-fix fix: Re-enable SQLite3DB::LoadPlugin() with allow_load_plugin flag	3 months ago
Rene Cannao	03e58146ea	fix: Re-enable SQLite3DB::LoadPlugin() with allow_load_plugin flag This addresses an issue from PR #22 where LoadPlugin() was completely disabled. The function performs necessary initialization even when no plugin is loaded (initializes built-in sqlite3 function pointers). Changes: - Added `const bool allow_load_plugin = false` flag in LoadPlugin() - Modified `if (plugin_name)` to `if (plugin_name && allow_load_plugin == true)` - Re-enabled the LoadPlugin() call in LoadPlugins() The plugin loading code remains disabled (allow_load_plugin=false) while the function pointer initialization from built-in SQLite3 now works correctly. TODO: Revisit plugin loading safety mechanism to allow actual plugin loading.	3 months ago
René Cannaò	3f1397c014	Merge pull request #27 from ProxySQL/v3.1-MCP2_QR fix: Multiple issues with MCP query_(rules/digests)	3 months ago
Rene Cannao	a3afde3472	fix: Address copilot review concerns for Discovery_Schema.cpp - Fix comment mismatch: Changed _2 suffix to match actual function name (mysql_query_digest_and_first_comment, not _2) - Make get_def_mysql_opts() static to avoid symbol pollution - Fix NULL first_comment parameter to prevent potential segfault - Pass valid char* pointer instead of NULL - Free first_comment if allocated by the function	3 months ago
Rene Cannao	7b6966b9c2	fix: Complete JSON escaping in fingerprint_mcp_args Address coderabbitai review - implement full JSON escaping for SQL digest: - Handle backslash (\) and double quote (") - Handle control characters: newline (\n), carriage return (\r), tab (\t) - Handle other control characters (U+0000 through U+001F) with \uXXXX escapes This ensures digest_text in stats_mcp_query_digest is always valid JSON, preventing parsing errors for consumers of this data.	3 months ago
Rene Cannao	f2536f01d2	Merge v3.1-vec into v3.1-MCP2_QR Resolved merge conflict in lib/ProxySQL_Admin_Stats.cpp: - Kept v3.1-MCP2_QR version with proper NULL handling - Uses stmt_unique_ptr (RAII smart pointer) - Checks for NULL before calling atoll() on numeric fields - Discarded v3.1-vec version which lacked NULL checks The merged code correctly handles NULL values for: - run_id - count_star - first_seen - last_seen - sum_time - min_time - max_time	3 months ago
Rene Cannao	5d4318b547	fix: Address coderabbitai review concerns for PR #27 - Fix Discovery_Schema.cpp fingerprint output JSON consistency - Quote placeholders (e.g., "?" instead of ?) for valid JSON - Fix test_mcp_query_rules_block.sh exec_admin_silent function - Add -B and -N flags for batch mode with no headers - Remove unused YELLOW variable from test_phase1_crud.sh - Fix test_phase2_load_save.sh runtime DELETE to be non-fatal - Remove unused INITIAL_COUNT from test_phase3_runtime.sh - Fix test_phase3_runtime.sh to verify exact ID ordering - Fix test_phase4_stats.sh read-only test assertion - Make INSERT/DELETE non-fatal and fix assertion logic - Remove unused DIGEST_TEXT_CHECK from test_phase5_digest.sh - Fix test_phase8_eval_timeout.sh non-timeout response handling - Return failure for non-timeout responses - Remove stray exit 1 from test_phase8_eval_timeout.sh	3 months ago

1 2 3 4 5 ...

8329 Commits (02f2ff5e2c72f9572cd7d8d313f11485f8b6ff48) All Branches Search

8329 Commits (02f2ff5e2c72f9572cd7d8d313f11485f8b6ff48)

All Branches