18 Commits (e53d2c76f93567eec5d342ccddf7ff9c2bd8fb25)

Author	SHA1	Message	Date
Rene Cannao	1af768f932	Fix critical issues in GenAI code (PR #5339) ... This commit addresses critical issues identified in the PR review: 1. Build system redundancy (src/Makefile): - Remove redundant PROXYSQLGENAI conditionals in linking commands - The flag doesn't affect linking, so simplify to PROXYSQLCLICKHOUSE only 2. Missing PROXYSQLGENAI guard (lib/Admin_Handler.cpp): - Add #ifdef PROXYSQLGENAI around MCP VARIABLES DISK commands - Ensures MCP commands are only available when GenAI is enabled 3. Broken retry logic (lib/LLM_Clients.cpp): - Remove misleading is_retryable_error() checks that used stale values - last_curl_code and last_http_code were never updated - Simplify to retry on empty responses only (documented limitation) - Fix thread-safety: use thread_local std::mt19937 instead of rand() 4. Resource leak (lib/MySQL_Tool_Handler.cpp): - Clean up previously created connections on init failure - Both mysql_init and mysql_real_connect error paths now clean up 5. Race condition (lib/Query_Tool_Handler.cpp): - Add missing pool_lock in find_connection() - Prevents race condition when accessing connection_pool Fixes identified by automated PR review agents.	2 months ago
Rene Cannao	b965fc6df4	Fix PROXYSQLGENAI build - resolve circular includes and missing headers ... This commit fixes compilation errors when building with PROXYSQLGENAI=1: Header file fixes: - Add missing #endif /* PROXYSQLGENAI */ before header guards in multiple headers - Remove #include cpp.h from GenAI headers to avoid circular dependencies Source file fixes: - Add #include proxysql.h to GenAI .cpp files that were missing it - Add #include Static_Harvester.h to Query_Tool_Handler.cpp for forward decl Build system fixes: - Remove vec.o from libproxysql.a (it's linked separately in src/Makefile) - Prevents duplicate symbol errors during linking Test files: - Rename MCP test files with .sh suffix to prevent make clean deletion Both build modes now work: - make build_lib_debug (without GenAI) - PROXYSQLGENAI=1 make build_lib_debug (with GenAI)	2 months ago
Rene Cannao	48bc7dd7bf	Merge v4.0 GenAI features into v3.0 with conditional compilation ... This commit merges the experimental v4.0 GenAI/MCP features into the stable v3.0 branch using conditional compilation. All v4.0 features are disabled by default and only enabled when PROXYSQLGENAI=1 is set at compile time. Changes: Build System: - Modified main Makefile to pass PROXYSQLGENAI flag to sub-makefiles - Modified deps/Makefile to conditionally build sqlite-vec and sqlite-rembed - Modified lib/Makefile to add PSQLGA flag and include GenAI object files - Modified src/Makefile to add PSQLGA flag and conditional linking Headers (wrapped with #ifdef PROXYSQLGENAI): - All 20 new GenAI header files in include/ - Modified cpp.h, proxysql_glovars.hpp, proxysql_admin.h - Modified ProxySQL_Admin_Tables_Definitions.h for GenAI/MCP tables Source Files: - All 22 new GenAI source files in lib/ wrapped with #ifdef PROXYSQLGENAI - Modified src/main.cpp for conditional global variables and init/shutdown - Modified Admin_Handler.cpp for conditional command handlers - Modified Admin_Bootstrap.cpp for conditional table registration - Modified Admin_FlushVariables.cpp for conditional variable flushing - Modified ProxySQL_Admin.cpp for conditional admin methods - Modified ProxySQL_Admin_Stats.cpp for conditional MCP stats functions - Modified proxy_sqlite3_symbols.cpp to always compile (needed by core) - Modified MySQL_Session.cpp for conditional GenAI function calls Test Files: - Renamed test_mcp_query_rules-t to test_mcp_query_rules-t.sh - Renamed test_mcp_rag_metrics-t to test_mcp_rag_metrics-t.sh - Modified anomaly_detection-t.cpp for conditional test execution Usage: # Build without GenAI (v3.0 mode - default) make clean && make build_deps -j$(nproc) && make build_lib -j$(nproc) && make build_src -j$(nproc) # Build with GenAI (v4.0 mode) make clean && PROXYSQLGENAI=1 make build_deps -j$(nproc) && PROXYSQLGENAI=1 make build_lib -j$(nproc) && PROXYSQLGENAI=1 make build_src -j$(nproc)	2 months ago
Javier Jaramago Fernández	e351a0df74	feat: Add leading comments (--) detection for 'run_sql_readonly' ... Read only queries are no longer flagged as non-readonly when starting with double dash comments.	2 months ago
Rene Cannao	3ccfa2bcc1	Address AI code review feedback for PR #5313 ... This commit addresses concerns raised by AI code reviewers (gemini-code-assist, Copilot, coderabbitai) on the initial security fixes. Critical fixes: - Fix lock.release() → lock.unlock() in GenAI_Thread.cpp worker_loop (lock.release() detaches without unlocking, causing deadlock) - Add missing early return after schema validation failure in Query_Tool_Handler.cpp Code quality improvements: - Improve escape_string() memory management in MySQL_Tool_Handler.cpp: - Use std::string instead of new[]/delete[] for buffer management - Check return value of mysql_real_escape_string() for errors - Remove redundant validation checks in validate_sql_identifier functions (character class loop already rejects unsafe characters) - Add backslash escaping to escape_string_literal() for defense-in-depth - Improve column list validation in MySQL_Tool_Handler sample_rows(): - Replace blacklist approach with proper column identifier parsing - Allow qualified identifiers (table.column) - Allow AS aliases (column AS alias) - No longer rejects legitimate column names containing "JOIN" These changes improve robustness while maintaining the security posture of the original SQL injection fixes.	3 months ago
Rene Cannao	c914feb230	Fix security issues identified in PR #5312 code review ... This commit addresses critical and important security vulnerabilities found during comprehensive code review of the Gen AI features merge. Critical fixes: - SQL injection vulnerabilities in MySQL_Tool_Handler.cpp: - Added validate_sql_identifier() for schema/table validation - Added escape_string() for MySQL string escaping using mysql_real_escape_string - Fixed list_tables(), describe_table(), sample_rows(), sample_distinct() - SQL injection vulnerabilities in Query_Tool_Handler.cpp: - Added validate_sql_identifier_sqlite() for identifier validation - Added escape_string_literal() for SQLite string escaping - Fixed list_tables tool and catalog.get_relationships function - Use-after-free race condition in GenAI_Thread: - Changed shutdown_ from int to std::atomic<int> for proper memory ordering - Added additional shutdown check in worker_loop after popping request Important fixes: - Buffer overflow risks from sprintf usage: - Converted all sprintf() calls to snprintf() in GenAI_Thread.cpp - Converted sprintf() to snprintf() in MySQL_Session.cpp - Worker loop shutdown race condition: - Added shutdown check after popping request from queue - Properly clean up client_fd when shutdown is detected These fixes ensure: 1. All user input is properly validated before use in SQL queries 2. String values are properly escaped using database-specific escaping 3. Thread-safe shutdown with proper memory ordering guarantees 4. Bounds-safe string formatting to prevent buffer overflows	3 months ago
Rene Cannao	b4f521c634	Merge v3.1-MCP2 into v3.1-vec ... Conflict resolution summary: **Features preserved from v3.1-vec:** - mcp_use_ssl variable (HTTP/HTTPS mode support) - RAG tool handler support - FTS (Full Text Search) functionality in MySQL_Tool_Handler **Changes accepted from v3.1-MCP2:** - Schema isolation for catalog (removed mcp_catalog_path/fts_path variables) - Schema parameter added to catalog functions - Comprehensive query rewriting improvements in Query_Tool_Handler - Format string fixes in vector_db_performance-t.cpp **Key decisions:** - catalog_path/fts_path: Removed per v3.1-MCP2 commit 35b0b224 (intentional) - MySQL_Catalog.cpp: Accepted v3.1-MCP2 version (schema isolation improvements) - Query_Tool_Handler.cpp: Accepted v3.1-MCP2 version (significant improvements) - Makefile: Kept v3.1-vec's EXCLUDE_TRACKING_VARIABLES (correct spelling) **Build notes:** - MySQL_Tool_Handler now requires schema parameter for catalog operations - Catalog path is hardcoded to datadir/mcp_catalog.db - FTS functionality preserved but may need updates for schema isolation	3 months ago
Rene Cannao	02918d18b8	Fix PR #25 Review: All AI code reviewer feedback addressed ... This commit addresses all recommendations from CodeRabbit, Gemini Code Assist, and Copilot for PR #25 (FTS security and code quality improvements). Critical Security Fixes: - MCP_Thread.cpp: Rollback fts_path on reset failure to keep config consistent - MySQL_FTS.cpp: Add escape_mysql_identifier() for MySQL query identifier escaping - MySQL_FTS.cpp: Add unique hash-based fallback to sanitize_name() for empty strings - MySQL_FTS.cpp: Add where_clause validation to block dangerous SQL patterns Memory Safety Fixes: - MySQL_FTS.cpp: Fix indexes_result memory leak on early return in search() - MySQL_FTS.h: Delete copy/move operations to prevent accidental resource duplication Thread Safety Documentation: - MySQL_Tool_Handler.cpp: Add comment explaining FTS lock design rationale Test Script Improvements: - test_mcp_fts.sh: Add curl timeouts (5s connect, 30s max) - test_mcp_fts.sh: Remove unused delete_response variable - test_mcp_fts_detailed.sh: Make cleanup tolerant of non-existent indexes Build Fixes: - Makefile: Fix EXCLUDE_TRACKING_VARAIABLES typo to EXCLUDE_TRACKING_VARIABLES - vector_db_performance-t.cpp: Fix printf format specifiers to %lld with cast Schema Fixes: - Query_Tool_Handler.cpp: Change fts_index_table columns schema from string to array Code Cleanup: - MySQL_Tool_Handler.cpp: Remove all remaining debug fprintf statements (34 lines) - Documentation: Change "Full Text" to "Full-Text" (hyphenated) Total: ~50 fixes across 10 files	3 months ago
Rene Cannao	a10c09bcc9	Fix PR #21 review: Security, memory safety, thread safety, and code cleanup ... Security fixes: - Add escape_identifier() helper for proper SQLite identifier escaping - Replace sanitize_name() with allowlist validation (ASCII letters, digits, underscore only) - Fix MATCH clause FTS5 operator injection by wrapping query in double quotes - Apply escape_identifier() to all DDL statements (CREATE, DROP, triggers) Memory safety fixes: - Replace VLA with std::vector in MySQL_FTS::init(), add delete on error path - Fix memory leak: free error string before return in list_indexes() - Fix reindex_json["error"] potential exception using .value() with default Thread safety fixes: - reinit_fts(): Add mutex lock around pointer swap - reset_fts_path(): Move blocking init() outside lock, only swap pointer under lock Code cleanup: - Remove 7 debug fprintf statements from Query_Tool_Handler.cpp - Remove unused #include <memory> from MySQL_FTS.h Test script security fixes: - Use MYSQL_PWD environment variable instead of -p"..." for password - Add escape_sql() function and apply to INSERT statement - Fix CURL_OPTS quoting: ${CURL_OPTS:+"${CURL_OPTS}"} - Remove unused FTS_INDEX_NAME and SEARCH_QUERIES variables Documentation fixes: - Fix bare URL to markdown link format - Add code block language identifiers (text, bash)	3 months ago
Rahim Kanji	0d56918744	Add full-text search (FTS) tools to MCP query server ... Implement BM25-ranked full-text search capability for MySQL/MariaDB tables using SQLite-based external FTS index. Changes: - Add MySQL_FTS class for managing SQLite FTS indexes - Add FTS tools: fts_index_table, fts_search, fts_reindex, fts_delete_index, fts_list_indexes, fts_rebuild_all - Add thread-safe FTS lifecycle management with fts_lock mutex - Add reset_fts_path() for runtime FTS database path configuration - Add comprehensive FTS test scripts (test_mcp_fts.sh, test_mcp_fts_detailed.sh)	3 months ago
Rene Cannao	7de3f0c510	feat: Add schema separation to MCP catalog and discovery scope constraint ... This commit addresses two issues: 1. MCP Catalog Schema Separation: - Add 'schema' column to catalog table for proper isolation - Update all catalog methods (upsert, get, search, list, remove) to accept schema parameter - Update MCP tool handlers and JSON-RPC parameter parsing - Unique constraint changed from (kind, key) to (schema, kind, key) - FTS table updated to include schema column 2. Discovery Prompt Scope Constraint: - Add explicit SCOPE CONSTRAINT section to multi_agent_discovery_prompt.md - Agents now respect Target Schema and skip list_schemas when specified - Prevents analyzing all schemas when only one is targeted Files modified: - include/MySQL_Catalog.h: Add schema parameter to all catalog methods - include/MySQL_Tool_Handler.h: Update wrapper method signatures - lib/MySQL_Catalog.cpp: Implement schema filtering in all operations - lib/MySQL_Tool_Handler.cpp: Update wrapper implementations - lib/Query_Tool_Handler.cpp: Extract schema from JSON-RPC params, update tool descriptions - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/multi_agent_discovery_prompt.md: Add scope constraint	3 months ago
Rene Cannao	f852900365	Fix: Correct MCP catalog JSON parsing to handle special characters ... The catalog_search() and catalog_list() methods in MySQL_Catalog.cpp were manually building JSON strings by concatenating raw TEXT from SQLite without proper escaping. This caused parse errors when stored JSON contained quotes, backslashes, or newlines. Changes: - MySQL_Catalog.cpp: Use nlohmann::json to build proper nested JSON in search() and list() methods instead of manual concatenation - MySQL_Tool_Handler.cpp: Add try-catch for JSON parsing in catalog_get() - test_catalog.sh: Fix MCP URL path, add jq extraction for MCP protocol responses, add 3 special character tests (CAT013-CAT015) Test Results: All 15 catalog tests pass, including new tests that verify special characters (quotes, backslashes) are preserved.	3 months ago
Rene Cannao	ef5b99edbf	Fix MCP tool bugs: NULL value handling and query validation ... - Fixed NULL value handling in execute_query: use empty string instead of nullptr to avoid "basic_string: construction from null" errors - Fixed validate_readonly_query: corrected substring length check from substr(0,6)!="SELECT " to substr(0,6)!="SELECT" - Fixed test script: added proper variable_name parameter for get_config/set_config tools Query endpoint tools now pass all tests.	3 months ago
Rene Cannao	acb4c57db3	Fix case sensitivity issues in MySQL_Tool_Handler::execute_query ... MySQL returns column names in uppercase for information_schema tables, but the code was expecting lowercase column names. This caused crashes when accessing JSON keys that didn't exist. Changes: 1. Convert all column names to lowercase in execute_query() 2. Store lowercase column names in a vector for efficient access 3. Use lowercase column names as keys in JSON row objects This ensures consistent column name casing across all queries, preventing JSON access errors for information_schema columns. Also includes the previous use-after-free fix.	3 months ago
Rene Cannao	904283330a	Fix critical use-after-free bug in MySQL_Tool_Handler::execute_query ... The code was creating a dangling pointer by calling c_str() on a temporary std::string object, causing undefined behavior and crashes when processing query results. Before: const char* col_name = columns[i].get<std::string>().c_str(); // ^ temporary string destroyed here, col_name is dangling After: std::string col_name = columns[i].get<std::string>(); // ^ col_name is valid until end of scope This bug was causing ProxySQL to crash when running MCP tool tests.	3 months ago
Rene Cannao	06aa6d6ef7	Add comprehensive Doxygen documentation for connection pool ... Added missing documentation for MySQL connection pool implementation: Header (MySQL_Tool_Handler.h): - Added MySQLConnection struct documentation with member descriptions - Added member variable documentation using ///< Doxygen style Implementation (MySQL_Tool_Handler.cpp): - Added Doxygen blocks for close() method - Added Doxygen blocks for init_connection_pool() with detailed behavior - Added Doxygen blocks for get_connection() with thread-safety notes - Added Doxygen blocks for return_connection() with reuse behavior - Added Doxygen blocks for execute_query() with JSON format documentation All new connection pool methods now have complete @brief, @param, and @return documentation following Doxygen conventions.	3 months ago
Rene Cannao	4eab519848	Implement MySQL connection pool for MySQL_Tool_Handler ... Added built-in connection pool to MySQL_Tool_Handler for direct MySQL connections to backend servers. Changes: - Added MySQLConnection struct with MYSQL* pointer, host, port, in_use flag - Added connection_pool vector, pool_lock mutex, pool_size counter - Implemented init_connection_pool() to create MYSQL connections using mysql_init/mysql_real_connect - Implemented get_connection() and return_connection() with thread-safe locking - Implemented execute_query() helper method for executing SQL and returning JSON results - Updated tool methods to use actual MySQL connections: - list_schemas: Query information_schema.schemata - list_tables: Query information_schema.tables with metadata - describe_table: Query columns, primary keys, indexes - sample_rows: Execute SELECT with LIMIT - sample_distinct: Execute SELECT DISTINCT with GROUP BY - run_sql_readonly: Execute validated SELECT queries - explain_sql: Execute EXPLAIN queries - Fixed MYSQL forward declaration (use typedef struct st_mysql MYSQL) The connection pool creates one connection per configured host:port pair with 5-second timeouts for connect/read/write operations.	3 months ago
Rene Cannao	221ff23991	Add MySQL exploration MCP tools with SQLite catalog ... Implemented MCP (Model Context Protocol) server providing tools for LLM-based MySQL database exploration: - MySQL_Catalog: SQLite-based catalog for LLM external memory with upsert, get, search, list, merge, delete operations and FTS support - MySQL_Tool_Handler: 17+ database exploration tools with guardrails: * Inventory: list_schemas, list_tables * Structure: describe_table, get_constraints, describe_view * Profiling: table_profile, column_profile * Sampling: sample_rows (max 20), sample_distinct (max 50) * Query: run_sql_readonly (max 200 rows, 2s timeout, SELECT-only) * Relationship: suggest_joins, find_reference_candidates * Catalog: catalog_upsert, catalog_get, catalog_search, catalog_list, catalog_merge, catalog_delete - MCP Module Integration: * Added 6 new configuration variables for MySQL tool handler (mysql_hosts, mysql_ports, mysql_user, mysql_password, mysql_schema, catalog_path) * Added MySQL_Tool_Handler pointer to MCP_Threads_Handler * Implemented tool routing in MCP endpoint for tools/list, tools/describe, and tools/call methods - TAP Tests: Updated to expect 14 MCP variables (was 8) Files: - include/MySQL_Catalog.h, lib/MySQL_Catalog.cpp - include/MySQL_Tool_Handler.h, lib/MySQL_Tool_Handler.cpp - include/MCP_Thread.h, lib/MCP_Thread.cpp - include/MCP_Endpoint.h, lib/MCP_Endpoint.cpp - lib/Makefile, test/tap/tests/mcp_module-t.cpp	3 months ago