proxysql

Commit Graph

Author	SHA1	Message	Date
Rahim Kanji	bc70988935	Added Chunking and Embedding guide	3 months ago
Rahim Kanji	1b322ed2bb	add comprehensive test suite for SQLite Server via MySQL protocol Add test_rag_ingest_sqlite_server.sh with 5 test phases: - Phase 1: Basic ingestion and watermark sync - Phase 2: WHERE filter support - Phase 3: Chunking support - Phase 4: Stub embeddings - Phase 5: OpenAI-compatible embeddings with vector search Tests rag_ingest tool connecting to ProxySQL SQLite3 Server (port 6030) using MySQL protocol, with backend MySQL as data source. Added documentation	3 months ago
Rahim Kanji	d66248eb4c	add verbose logging for bulk embedding operations Add logs to track embedding generation progress: - Show batch size when collecting chunks for embedding - Display OpenAI API call details (endpoint, model, chunk count) - Confirm successful storage after each batch	3 months ago
Rahim Kanji	211179bdcb	rag_ingest: Rewrite to use MySQL protocol (Sqlite_Server) instead of SQLite3 API Convert rag_ingest from direct SQLite database access to MySQL protocol communication with SQLite Server. Major changes: - Replace SQLite3 direct API with MySQL/mariadb client library - Update connection flow: rag_ingest -> MySQL protocol -> SQLite Server -> RAG DB - Remove sqlite3 amalgamation dependency from rag_ingest target - Keep all ingestion functionality intact (chunking, FTS5, vec0 embeddings) Schema initialization: - Add init command to create RAG schema via MySQL protocol - Add ingest command to run data ingestion pipeline - Add query command for vector similarity search with embeddings	3 months ago
René Cannaò	c5c726ea98	Merge pull request #5321 from sysown/v4.0-mcp_rules_test WIP-Fixes: Add/Improve TAP tests for MCP query_rules/digests	3 months ago
Rene Cannao	87d8c25074	Reapply "fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins'" This reverts commit `48260cbf09`.	3 months ago
René Cannaò	342272367d	Merge pull request #5324 from sysown/v4.0_rag_ingest_2 V4.0 rag ingest 2	3 months ago
Rene Cannao	35b316d4f3	Fix absolute paths in documentation Replace absolute paths (/home/rene/pr5312/proxysql/) with relative paths from repository root for portability.	3 months ago
Rene Cannao	a5edd4ad97	Add comprehensive RAG ingestion usage guide Document step-by-step usage of rag_ingest tool including: - Quick start instructions - Step-by-step setup guide - Common configurations (basic, chunking, embeddings) - Configuration reference (chunking_json, embedding_json, doc_map_json) - Common workflows (initial setup, re-run, add embeddings) - Troubleshooting tips This guide complements EMBEDDING_TEST_PLAN.md which focuses specifically on testing embeddings.	3 months ago
Rene Cannao	48260cbf09	Revert "fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins'" This reverts commit `df221617c0`.	3 months ago
Javier Jaramago Fernández	51e91378c4	test: Add/Improve TAP tests for MCP query_rules/digests	3 months ago
Rene Cannao	5fd779464d	Add embedding testing plan documentation Document how to test RAG ingestion with embeddings using external OpenAI-compatible services. Users only need to set 4 environment variables (OPENAI_API_BASE, OPENAI_API_KEY, OPENAI_MODEL, OPENAI_EMBEDDING_DIM) and run the test script. Related to issue #5320 (Batch embedding generation)	3 months ago
Javier Jaramago Fernández	08e7c8ce97	chore: Remove data files mistakenly added to root folder	3 months ago
Javier Jaramago Fernández	df221617c0	fix: 'SQLITE_CONFIG_URI' not being set on 'LoadPlugins' Since 'proxy_sqlite3_open_v2' is now being set by default due to the new dedicated TU ('proxy_sqlite3_symbols.cpp'). 'LoadPlugins' might not be called, not properly setting 'SQLITE_CONFIG_URI' and invalidating many configuration that are currently specified in the conn URI.	3 months ago
Rene Cannao	ff50e834f0	Implement batch embedding generation in rag_ingest Previously, embeddings were generated one chunk at a time, with each chunk requiring a separate HTTP API call. This was inefficient due to HTTP call overhead (latency). Changes: - Add PendingEmbedding struct to hold chunk metadata for batching - Add flush_embedding_batch() function to process multiple embeddings in a single API call - Modify ingestion loop to collect chunks and flush batches when full - Use the existing batch_size configuration from embedding_json (default: 16 chunks per batch) Performance improvement: - Before: 1000 chunks = 1000 HTTP calls (e.g., 100ms each = 100 seconds) - After: 1000 chunks = ~63 HTTP calls (1000 / 16 = ~63 batches) This significantly reduces ingestion time for sources with embeddings enabled.	3 months ago
René Cannaò	6166adaf05	Merge pull request #5318 from rahim-kanji/v4.0_rag_ingest Add RAG ingestion with vector embeddings	3 months ago
Rene Cannao	07154ca661	Empty commit to differentiate from v4.0_rag_ingest_2	3 months ago
Rene Cannao	42a67ebaf2	Merge rahim/v4.0_rag_ingest into v4.0_rag_ingest_2 Merge changes from PR #5318 (RAG ingestion feature) into the new development branch v4.0_rag_ingest_2. Changes include: - RAG ingestion tool (rag_ingest) with chunking and embeddings - MySQL_Catalog fixes for NULL pointer handling - MCP server updates for RAG tools - Comprehensive documentation and test scripts	3 months ago
Rene Cannao	950f163bfb	Fix compilation issues: use static libcurl and improve includes - Add curl include path from deps/curl/curl/include - Link against static libcurl.a from deps - Add zlib dependency (-lz) required by libcurl - Use "" style includes for local headers (sqlite3.h, mysql.h, crypt.h, curl/curl.h) - Rename sqlite_bind_text helper to bind_text to avoid collision with sqlite3_bind_text API - Add comprehensive Doxygen documentation to rag_ingest.cpp	3 months ago
Rene Cannao	38e5e8e56b	Fix critical issues from coderabbitai review - Fix NULL pointer dereference in rag_ingest.cpp: use str_or_empty() helper for all sqlite3_column_text results assigned to std::string - Fix NULL tags/links crash in MySQL_Catalog.cpp: add null guards before assigning sqlite3_column_text results to std::string - Fix missing curl_global_cleanup on error path in rag_ingest.cpp - Fix std::out_of_range exception in rag_ingest.cpp: wrap std::stoll calls in try-catch blocks, fall back to string comparison on overflow	3 months ago
Rene Cannao	9ba3df0ce7	Address AI code review feedback from PR #5318 - Fix Makefile: Use $(CXXFLAGS) directly for consistency with build philosophy - Fix MySQL_Catalog: Return proper error JSON instead of empty array on missing query	3 months ago
René Cannaò	1f38d9210c	Merge pull request #5313 from sysown/pr-5312-fixes Fix security issues identified in PR #5312 code review	3 months ago
Rahim Kanji	d28444a02e	Merge remote-tracking branch 'v4.0' into v4.0_rag_ingest	3 months ago
Rahim Kanji	bc33c33295	Add vec embedding handling with stub and OpenAI providers Added rag_sync_state implementation	3 months ago
Rene Cannao	3ccfa2bcc1	Address AI code review feedback for PR #5313 This commit addresses concerns raised by AI code reviewers (gemini-code-assist, Copilot, coderabbitai) on the initial security fixes. Critical fixes: - Fix lock.release() → lock.unlock() in GenAI_Thread.cpp worker_loop (lock.release() detaches without unlocking, causing deadlock) - Add missing early return after schema validation failure in Query_Tool_Handler.cpp Code quality improvements: - Improve escape_string() memory management in MySQL_Tool_Handler.cpp: - Use std::string instead of new[]/delete[] for buffer management - Check return value of mysql_real_escape_string() for errors - Remove redundant validation checks in validate_sql_identifier functions (character class loop already rejects unsafe characters) - Add backslash escaping to escape_string_literal() for defense-in-depth - Improve column list validation in MySQL_Tool_Handler sample_rows(): - Replace blacklist approach with proper column identifier parsing - Allow qualified identifiers (table.column) - Allow AS aliases (column AS alias) - No longer rejects legitimate column names containing "JOIN" These changes improve robustness while maintaining the security posture of the original SQL injection fixes.	3 months ago
Rene Cannao	c914feb230	Fix security issues identified in PR #5312 code review This commit addresses critical and important security vulnerabilities found during comprehensive code review of the Gen AI features merge. Critical fixes: - SQL injection vulnerabilities in MySQL_Tool_Handler.cpp: - Added validate_sql_identifier() for schema/table validation - Added escape_string() for MySQL string escaping using mysql_real_escape_string - Fixed list_tables(), describe_table(), sample_rows(), sample_distinct() - SQL injection vulnerabilities in Query_Tool_Handler.cpp: - Added validate_sql_identifier_sqlite() for identifier validation - Added escape_string_literal() for SQLite string escaping - Fixed list_tables tool and catalog.get_relationships function - Use-after-free race condition in GenAI_Thread: - Changed shutdown_ from int to std::atomic<int> for proper memory ordering - Added additional shutdown check in worker_loop after popping request Important fixes: - Buffer overflow risks from sprintf usage: - Converted all sprintf() calls to snprintf() in GenAI_Thread.cpp - Converted sprintf() to snprintf() in MySQL_Session.cpp - Worker loop shutdown race condition: - Added shutdown check after popping request from queue - Properly clean up client_fd when shutdown is detected These fixes ensure: 1. All user input is properly validated before use in SQL queries 2. String values are properly escaped using database-specific escaping 3. Thread-safe shutdown with proper memory ordering guarantees 4. Bounds-safe string formatting to prevent buffer overflows	3 months ago
René Cannaò	4075e7cf62	Merge pull request #5312 from ProxySQL/v3.1-vec Port of Gen AI commits to proxysql (2nd attempt)	3 months ago
René Cannaò	19d29a7346	Merge pull request #28 from ProxySQL/v3.1-vec-loadplugin-fix fix: Re-enable SQLite3DB::LoadPlugin() with allow_load_plugin flag	3 months ago
Rene Cannao	03e58146ea	fix: Re-enable SQLite3DB::LoadPlugin() with allow_load_plugin flag This addresses an issue from PR #22 where LoadPlugin() was completely disabled. The function performs necessary initialization even when no plugin is loaded (initializes built-in sqlite3 function pointers). Changes: - Added `const bool allow_load_plugin = false` flag in LoadPlugin() - Modified `if (plugin_name)` to `if (plugin_name && allow_load_plugin == true)` - Re-enabled the LoadPlugin() call in LoadPlugins() The plugin loading code remains disabled (allow_load_plugin=false) while the function pointer initialization from built-in SQLite3 now works correctly. TODO: Revisit plugin loading safety mechanism to allow actual plugin loading.	3 months ago
René Cannaò	3f1397c014	Merge pull request #27 from ProxySQL/v3.1-MCP2_QR fix: Multiple issues with MCP query_(rules/digests)	3 months ago
Rene Cannao	a3afde3472	fix: Address copilot review concerns for Discovery_Schema.cpp - Fix comment mismatch: Changed _2 suffix to match actual function name (mysql_query_digest_and_first_comment, not _2) - Make get_def_mysql_opts() static to avoid symbol pollution - Fix NULL first_comment parameter to prevent potential segfault - Pass valid char* pointer instead of NULL - Free first_comment if allocated by the function	3 months ago
Rene Cannao	7b6966b9c2	fix: Complete JSON escaping in fingerprint_mcp_args Address coderabbitai review - implement full JSON escaping for SQL digest: - Handle backslash (\) and double quote (") - Handle control characters: newline (\n), carriage return (\r), tab (\t) - Handle other control characters (U+0000 through U+001F) with \uXXXX escapes This ensures digest_text in stats_mcp_query_digest is always valid JSON, preventing parsing errors for consumers of this data.	3 months ago
Rene Cannao	f2536f01d2	Merge v3.1-vec into v3.1-MCP2_QR Resolved merge conflict in lib/ProxySQL_Admin_Stats.cpp: - Kept v3.1-MCP2_QR version with proper NULL handling - Uses stmt_unique_ptr (RAII smart pointer) - Checks for NULL before calling atoll() on numeric fields - Discarded v3.1-vec version which lacked NULL checks The merged code correctly handles NULL values for: - run_id - count_star - first_seen - last_seen - sum_time - min_time - max_time	3 months ago
Rene Cannao	5d4318b547	fix: Address coderabbitai review concerns for PR #27 - Fix Discovery_Schema.cpp fingerprint output JSON consistency - Quote placeholders (e.g., "?" instead of ?) for valid JSON - Fix test_mcp_query_rules_block.sh exec_admin_silent function - Add -B and -N flags for batch mode with no headers - Remove unused YELLOW variable from test_phase1_crud.sh - Fix test_phase2_load_save.sh runtime DELETE to be non-fatal - Remove unused INITIAL_COUNT from test_phase3_runtime.sh - Fix test_phase3_runtime.sh to verify exact ID ordering - Fix test_phase4_stats.sh read-only test assertion - Make INSERT/DELETE non-fatal and fix assertion logic - Remove unused DIGEST_TEXT_CHECK from test_phase5_digest.sh - Fix test_phase8_eval_timeout.sh non-timeout response handling - Return failure for non-timeout responses - Remove stray exit 1 from test_phase8_eval_timeout.sh	3 months ago
Rahim Kanji	d47e196fc6	Added rag_chunks and rag_fts_chunks test	3 months ago
Rahim Kanji	2f6b058f7b	Added test_rag_ingest.sh	3 months ago
Rahim Kanji	fb3673dd9d	added sample sql files	3 months ago
Rahim Kanji	57d8c3f3b8	Make rag_ingest compile	3 months ago
René Cannaò	b16d16fb20	Merge pull request #26 from ProxySQL/v3.1-vec-resolved Merge v3.1-MCP2 into v3.1-vec (conflicts resolved)	3 months ago
Rene Cannao	ffe5690360	fix: Address coderabbitai review - use-after-free, missing responses, SQL injection Fix issues identified by coderabbitai review: 1. Admin_Handler.cpp: Fix typo in strncasecmp for LOAD MCP QUERY RULES - Line 2365 had "LOAD MCP RULES FROM DISK" instead of "LOAD MCP QUERY RULES FROM DISK" 2. Admin_Handler.cpp: Fix use-after-free and missing client response - Removed l_free(ql,q) which freed q before caller used it - Added send_error_msg_to_client calls on all error paths - Added send_ok_msg_to_client call on success path - Changed return value from true to false (to match handler pattern) - Applied to both LOAD MCP QUERY RULES and SAVE MCP QUERY RULES handlers 3. ProxySQL_Admin_Stats.cpp: Fix sprintf SQL injection in stats___mcp_query_rules - Replaced sprintf with prepared statement using positional parameters - Changed from char query with malloc/free to sqlite3_stmt with prepare_v2 - Both columns now bound as parameters (?1, ?2)	3 months ago
Rene Cannao	5ece563514	fix: Correct SQL prepared statement API usage and template variable access Fix compilation errors in the SQL injection fixes: 1. ProxySQL_Admin_Stats.cpp: Use public statsdb->prepare_v2() API - Changed from direct proxy_sqlite3_prepare_v2() calls with statsdb->db - statsdb->db is private, must use public prepare_v2(query, &stmt) method 2. Admin_Handler.cpp: Add SPA cast for template function access - Added ProxySQL_Admin SPA=(ProxySQL_Admin )pa; declaration - Changed all admindb->execute to SPA->admindb->execute - Removed unused 'error' and 'success' variables The build now completes successfully.	3 months ago
Rene Cannao	9f07e9631e	fix: Use prepared statements in ProxySQL_Admin_Stats to prevent SQL injection Fix two SQL injection vulnerabilities identified by coderabbitai in ProxySQL_Admin_Stats.cpp by converting sprintf/snprintf interpolation to SQLite prepared statements. 1. stats___mcp_query_digest (lines 2581-2636): - Changed from sprintf with format string to prepared statement - digest_text field now properly bound as parameter (was unsafe) - All 10 columns now use positional parameters (?1-?10) 2. stats___mcp_query_tools_counters (lines 1587-1635): - Changed from snprintf with unescaped fields to prepared statement - Fixed incorrect table name logic (was appending _reset incorrectly) - All 8 columns now use positional parameters (?1-?8) These changes prevent SQL injection when resultset fields contain quotes, backslashes, or other SQL metacharacters.	3 months ago
Rene Cannao	3bcee22700	fix: Execute MCP query rules DELETE+INSERT as explicit transaction Fix multi-statement execution in Admin_Handler.cpp for MCP query rules. The previous code built a single SQL string with "DELETE ...; INSERT ..." but SQLite only executed the first statement. Changed to execute statements as an explicit transaction: 1. BEGIN 2. DELETE FROM target_table 3. INSERT OR REPLACE INTO target_table SELECT * FROM source_table 4. COMMIT (or ROLLBACK on error) Applied to both: - LOAD MCP QUERY RULES FROM DISK/TO MEMORY - SAVE MCP QUERY RULES TO DISK Addresses coderabbitai review comment.	3 months ago
Rene Cannao	188aef90fe	fix: Use delete instead of free for SQLite3_result in load_mcp_query_rules_to_runtime Change free(resultset) to delete resultset in ProxySQL_Admin::load_mcp_query_rules_to_runtime. SQLite3_result is a C++ class allocated with new, so it must be deallocated with delete, not free(). Using free() causes undefined behavior and memory leaks. Addresses coderabbitai review comment.	3 months ago
Rene Cannao	bbc04974f1	fix: Fix mysql_query failure path and affected_rows race condition Fix two issues in Query_Tool_Handler's execute_query functions: 1. mysql_query() failure path now returns immediately after return_connection() instead of continuing to process on bad state. 2. Capture affected_rows BEFORE return_connection() to avoid race condition. Previously, mysql_affected_rows() was called after return_connection(), potentially accessing a stale connection. Apply fixes to both execute_query() and execute_query_with_schema(). Addresses coderabbitai review comments.	3 months ago
Rene Cannao	b3edc6524b	fix: Escape SQL strings in harvest_view_definitions Add escape_sql_string() helper function that doubles single quotes to prevent SQL injection when strings are used in SQL string concatenation. Update harvest_view_definitions to use this function for view_def, schema_name, and view_name. This prevents SQL injection in the UPDATE statement that stores view definitions in the catalog. Addresses coderabbitai review comment.	3 months ago
Rene Cannao	6835713f11	fix: Correct column indexes in build_quick_profiles Fix off-by-one errors in row->fields indices. The SELECT clause returns: object_id(0), schema_name(1), object_name(2), object_type(3), engine(4), table_rows_est(5), data_length(6), index_length(7), has_primary_key(8), has_foreign_keys(9), has_time_column(10) But the code was reading from fields[3..9] instead of [4..10]. Added comment documenting the correct column order. Addresses coderabbitai review comment.	3 months ago
Rene Cannao	e9abee625b	fix: Execute prepared statement in execute_parameterized_query The execute_parameterized_query function in RAG_Tool_Handler was creating a prepared statement and binding parameters, but then executing the raw query string instead of the prepared statement. This completely bypassed the parameter binding, making the function useless for preventing SQL injection. Changed to use vector_db->execute_prepared(stmt, ...) to execute the bound statement instead of the raw query, so the bound parameters are actually used. Addresses coderabbitai review comment.	3 months ago
Rene Cannao	6305537ba8	fix: Use delete instead of free for SQLite3_result deallocation Change free(resultset) to delete resultset in Query_Tool_Handler (extract_schema_name function). SQLite3_result is a C++ class allocated with new, so it must be deallocated with delete, not free(). Using free() causes mixed allocator UB. Addresses coderabbitai review comment.	3 months ago
Rene Cannao	5e121399aa	fix: Add AFTER UPDATE trigger to keep catalog_fts index in sync for upserts Add catalog_au AFTER UPDATE trigger in MySQL_Catalog that mirrors the delete+insert pattern used in catalog_ad/catalog_ai. This keeps the FTS index current when upserts occur (INSERT OR REPLACE ... ON CONFLICT ... DO UPDATE), since the UPDATE doesn't trigger INSERT/DELETE triggers. The trigger first deletes the old entry from catalog_fts then inserts the new entry, ensuring the full-text search index stays synchronized with the catalog table. Addresses coderabbitai review comment.	3 months ago

1 2 3 4 5 ...

8313 Commits (bc70988935a69850da237d2fde8ce415c7be67db) All Branches Search

8313 Commits (bc70988935a69850da237d2fde8ce415c7be67db)

All Branches