proxysql

Commit Graph

Author	SHA1	Message	Date
René Cannaò	ce9fdc9d7a	Merge branch 'v3.0' into v3.0.6-implement_FLUSH_STATS Signed-off-by: René Cannaò <rene@proxysql.com>	1 month ago
Rene Cannao	79756e78d7	Fix missing #endif for PROXYSQLGENAI guard in Admin_Handler.cpp	1 month ago
Rene Cannao	1af768f932	Fix critical issues in GenAI code (PR #5339 ) This commit addresses critical issues identified in the PR review: 1. Build system redundancy (src/Makefile): - Remove redundant PROXYSQLGENAI conditionals in linking commands - The flag doesn't affect linking, so simplify to PROXYSQLCLICKHOUSE only 2. Missing PROXYSQLGENAI guard (lib/Admin_Handler.cpp): - Add #ifdef PROXYSQLGENAI around MCP VARIABLES DISK commands - Ensures MCP commands are only available when GenAI is enabled 3. Broken retry logic (lib/LLM_Clients.cpp): - Remove misleading is_retryable_error() checks that used stale values - last_curl_code and last_http_code were never updated - Simplify to retry on empty responses only (documented limitation) - Fix thread-safety: use thread_local std::mt19937 instead of rand() 4. Resource leak (lib/MySQL_Tool_Handler.cpp): - Clean up previously created connections on init failure - Both mysql_init and mysql_real_connect error paths now clean up 5. Race condition (lib/Query_Tool_Handler.cpp): - Add missing pool_lock in find_connection() - Prevents race condition when accessing connection_pool Fixes identified by automated PR review agents.	1 month ago
Rene Cannao	48bc7dd7bf	Merge v4.0 GenAI features into v3.0 with conditional compilation This commit merges the experimental v4.0 GenAI/MCP features into the stable v3.0 branch using conditional compilation. All v4.0 features are disabled by default and only enabled when PROXYSQLGENAI=1 is set at compile time. Changes: Build System: - Modified main Makefile to pass PROXYSQLGENAI flag to sub-makefiles - Modified deps/Makefile to conditionally build sqlite-vec and sqlite-rembed - Modified lib/Makefile to add PSQLGA flag and include GenAI object files - Modified src/Makefile to add PSQLGA flag and conditional linking Headers (wrapped with #ifdef PROXYSQLGENAI): - All 20 new GenAI header files in include/ - Modified cpp.h, proxysql_glovars.hpp, proxysql_admin.h - Modified ProxySQL_Admin_Tables_Definitions.h for GenAI/MCP tables Source Files: - All 22 new GenAI source files in lib/ wrapped with #ifdef PROXYSQLGENAI - Modified src/main.cpp for conditional global variables and init/shutdown - Modified Admin_Handler.cpp for conditional command handlers - Modified Admin_Bootstrap.cpp for conditional table registration - Modified Admin_FlushVariables.cpp for conditional variable flushing - Modified ProxySQL_Admin.cpp for conditional admin methods - Modified ProxySQL_Admin_Stats.cpp for conditional MCP stats functions - Modified proxy_sqlite3_symbols.cpp to always compile (needed by core) - Modified MySQL_Session.cpp for conditional GenAI function calls Test Files: - Renamed test_mcp_query_rules-t to test_mcp_query_rules-t.sh - Renamed test_mcp_rag_metrics-t to test_mcp_rag_metrics-t.sh - Modified anomaly_detection-t.cpp for conditional test execution Usage: # Build without GenAI (v3.0 mode - default) make clean && make build_deps -j$(nproc) && make build_lib -j$(nproc) && make build_src -j$(nproc) # Build with GenAI (v4.0 mode) make clean && PROXYSQLGENAI=1 make build_deps -j$(nproc) && PROXYSQLGENAI=1 make build_lib -j$(nproc) && PROXYSQLGENAI=1 make build_src -j$(nproc)	1 month ago
Miro Stauder	cc1d93d08b	implement admin command 'PROXYSQL FLUSH STATS' for DEBUG builds	2 months ago
Rene Cannao	ffe5690360	fix: Address coderabbitai review - use-after-free, missing responses, SQL injection Fix issues identified by coderabbitai review: 1. Admin_Handler.cpp: Fix typo in strncasecmp for LOAD MCP QUERY RULES - Line 2365 had "LOAD MCP RULES FROM DISK" instead of "LOAD MCP QUERY RULES FROM DISK" 2. Admin_Handler.cpp: Fix use-after-free and missing client response - Removed l_free(ql,q) which freed q before caller used it - Added send_error_msg_to_client calls on all error paths - Added send_ok_msg_to_client call on success path - Changed return value from true to false (to match handler pattern) - Applied to both LOAD MCP QUERY RULES and SAVE MCP QUERY RULES handlers 3. ProxySQL_Admin_Stats.cpp: Fix sprintf SQL injection in stats___mcp_query_rules - Replaced sprintf with prepared statement using positional parameters - Changed from char query with malloc/free to sqlite3_stmt with prepare_v2 - Both columns now bound as parameters (?1, ?2)	2 months ago
Rene Cannao	5ece563514	fix: Correct SQL prepared statement API usage and template variable access Fix compilation errors in the SQL injection fixes: 1. ProxySQL_Admin_Stats.cpp: Use public statsdb->prepare_v2() API - Changed from direct proxy_sqlite3_prepare_v2() calls with statsdb->db - statsdb->db is private, must use public prepare_v2(query, &stmt) method 2. Admin_Handler.cpp: Add SPA cast for template function access - Added ProxySQL_Admin SPA=(ProxySQL_Admin )pa; declaration - Changed all admindb->execute to SPA->admindb->execute - Removed unused 'error' and 'success' variables The build now completes successfully.	2 months ago
Rene Cannao	3bcee22700	fix: Execute MCP query rules DELETE+INSERT as explicit transaction Fix multi-statement execution in Admin_Handler.cpp for MCP query rules. The previous code built a single SQL string with "DELETE ...; INSERT ..." but SQLite only executed the first statement. Changed to execute statements as an explicit transaction: 1. BEGIN 2. DELETE FROM target_table 3. INSERT OR REPLACE INTO target_table SELECT * FROM source_table 4. COMMIT (or ROLLBACK on error) Applied to both: - LOAD MCP QUERY RULES FROM DISK/TO MEMORY - SAVE MCP QUERY RULES TO DISK Addresses coderabbitai review comment.	2 months ago
Rene Cannao	709649232b	fix: Address AI code review concerns from PR #19 This commit addresses valid concerns raised by coding agents (Gemini, Copilot, CoderabbitAI): 1. Fix stats_mcp_query_digest naming conflict (ProxySQL_Admin.cpp): - Made reset and non-reset paths mutually exclusive using else block - Prevents both flags from being true, matching MySQL pattern - Ensures reset takes precedence over non-reset 2. Fix INSERT OR REPLACE sync issue (Admin_Handler.cpp): - Added DELETE before INSERT OR REPLACE in LOAD/SAVE MCP QUERY RULES - Prevents stale rules from persisting when syncing disk <-> memory - Ensures deleted source rows are also removed from target 3. Fix integer division truncation for timeout (Query_Tool_Handler.cpp): - Changed timeout_ms/1000 to (timeout_ms+999)/1000 for ceiling division - Ensures sub-second timeouts (e.g., 500ms) become at least 1 second - Prevents zero-second timeouts from causing unexpected behavior 4. Remove confusing comment (Discovery_Schema.cpp): - Simplified column count comment to be clear and accurate Note: The re_modifiers parsing code already correctly handles VARCHAR "CASELESS" to int conversion (lines 2414-2425), so that review comment was already addressed.	2 months ago
Rene Cannao	a1d9d2f1ba	docs: Add comprehensive documentation to MCP features Add detailed function-level documentation to all MCP query rules, query digest, static harvester, and catalog components. Static_Harvester.cpp: - Document all 18+ harvest functions (schemas, objects, columns, indexes, FKs, views) - Document lifecycle methods (init, close, connect, disconnect) - Document helper methods (is_time_type, is_id_like_name) - Document run management (start_run, finish_run, run_full_harvest) - Document statistics methods (get_harvest_stats) Query_Tool_Handler.cpp: - Document JSON helper functions (json_string, json_int, json_double) - Document digest tracking section with flow explanation MySQL_Catalog.cpp: - Document schema isolation architecture - Document CRUD operations (upsert, get, search, list, remove, merge) Discovery_Schema.cpp: - Document MCP query rules evaluation (evaluate_mcp_query_rules) - Document digest functions (compute_mcp_digest, fingerprint_mcp_args) - Document update/get functions for rules and digests ProxySQL_Admin_Stats.cpp: - Document stats collection functions ProxySQL_Admin.cpp: - Document load/save functions for query rules Admin_Handler.cpp: - Document MCP query rules command handlers include/ProxySQL_Admin_Tables_Definitions.h: - Add comments explaining table purposes	2 months ago
Rene Cannao	8c9aecce9b	feat: Add LOAD MCP QUERY RULES FROM DISK / TO MEMORY commands - Add LOAD MCP QUERY RULES FROM DISK command - Add LOAD MCP QUERY RULES TO MEMORY command - Both commands copy rules from disk.mcp_query_rules to main.mcp_query_rules This completes the full set of MCP query rules LOAD/SAVE commands, matching the MySQL query rules pattern.	2 months ago
Rene Cannao	7e6f9f0ab3	fix: Add MCP query rules LOAD/SAVE command handlers - Add separate MCP QUERY RULES command block in Admin_Handler - Fix string length comparison (21 chars for "SAVE/LOAD MCP QUERY RULES ") - Add handlers for: - LOAD MCP QUERY RULES TO RUNTIME - SAVE MCP QUERY RULES TO DISK - SAVE MCP QUERY RULES TO MEMORY / FROM RUNTIME - Register mcp_query_rules in disk database (tables_defs_config) Previously MCP commands were incorrectly nested inside MYSQL/PGSQL block and could not be reached. Now they have their own conditional block.	2 months ago
Rene Cannao	f01fc79584	feat: Add runtime_mcp_query_rules table and fix stats_mcp_query_rules schema - Add ADMIN_SQLITE_TABLE_RUNTIME_MCP_QUERY_RULES schema (17 columns, same as mcp_query_rules) - Fix STATS_SQLITE_TABLE_MCP_QUERY_RULES to only have rule_id and hits columns - Add runtime_mcp_query_rules detection and refresh in ProxySQL_Admin - Implement save_mcp_query_rules_from_runtime(bool _runtime) for both config and runtime tables - Update get_mcp_query_rules() to return 17 columns (no hits) - get_stats_mcp_query_rules() returns 2 columns (rule_id, hits) Mirrors the MySQL query rules pattern: - mcp_query_rules: config table (17 cols) - runtime_mcp_query_rules: runtime state (17 cols) - stats_mcp_query_rules: hit counters (2 cols)	2 months ago
Rene Cannao	3fe8a48f70	Fix genai variable handling and add API key masking - Add has_variable() method to GenAI_Threads_Handler for variable validation - Add genai- prefix check in is_valid_global_variable() - Auto-initialize NL2SQL converter when genai-nl2sql_enabled is set to true at runtime - Make init_nl2sql() public to allow runtime initialization - Mask API keys in logs (show only first 2 chars, rest as 'x')	2 months ago
René Cannaò	313f637cf0	Merge branch 'v3.1-vec' into v3.1-MCP1 Signed-off-by: René Cannaò <rene.cannao@gmail.com>	2 months ago
Rene Cannao	5a85ef04f6	Fix MCP variables persistence and add DISK command support This commit fixes several issues with MCP (Model Context Protocol) variables not being properly persisted across storage layers and adds support for DISK commands. Changes: 1. lib/Admin_FlushVariables.cpp: - Fixed flush_mcp_variables___runtime_to_database() to properly insert variables into runtime_global_variables using db->execute() with formatted strings (matching admin pattern) - Fixed SQL format string to avoid double-prefix bug (qualified_name already contains "mcp-" prefix) - Fixed lock ordering by releasing outer wrlock before calling runtime_to_database with use_lock=true, then re-acquiring - Removed explicit BEGIN/COMMIT transactions to match admin pattern 2. lib/Admin_Handler.cpp: - Added MCP DISK command handlers that rewrite commands to SQL queries: * LOAD MCP VARIABLES FROM DISK -> INSERT OR REPLACE INTO main.global_variables * SAVE MCP VARIABLES TO DISK -> INSERT OR REPLACE INTO disk.global_variables * SAVE MCP VARIABLES FROM MEMORY/MEM -> INSERT OR REPLACE INTO disk.global_variables - Separated DISK command handlers from MEMORY/RUNTIME handlers 3. lib/ProxySQL_Admin.cpp: - Added flush_mcp_variables___runtime_to_database() call to stats section to ensure MCP variables are repopulated when runtime_global_variables is cleared and refreshed 4. tests/mcp_module-t.cpp: - Added verbose diagnostic output throughout tests - Added section headers and test numbers for clarity - Added variable value logging and error logging All 52 MCP module tests now pass.	2 months ago
Rene Cannao	ef07831780	Add MCP module to admin bootstrap and SHOW MCP VARIABLES command The MCP module was not being loaded because: 1. The admin bootstrap process was not calling flush_mcp_variables___database_to_runtime - Added the call after flush_sqliteserver_variables___database_to_runtime 2. There was no SHOW MCP VARIABLES command handler - Added the handler in Admin_Handler.cpp, following the same pattern as SHOW MYSQL VARIABLES and SHOW PGSQL VARIABLES Now after this change: - MCP variables (mcp-enabled, mcp-port, mcp-mysql_hosts, etc.) will be automatically inserted into global_variables table during ProxySQL startup - Users can run "SHOW MCP VARIABLES" to list all MCP configuration variables - The configure_mcp.sh script will work correctly Note: Requires rebuilding ProxySQL for changes to take effect.	2 months ago
Rene Cannao	b032c3f690	Fix boolean literal handling in SET command for MCP variables When SET commands use boolean literals (true/false), SQLite was interpreting them as boolean keywords and storing 1/0 instead of the string values "true"/"false". Fixed by detecting boolean literals in admin_handler_command_set() and quoting them as strings in the UPDATE statement. All 52 MCP module TAP tests now pass.	2 months ago
Rene Cannao	81c53896bc	Fix MCP module TAP test failures - Add MCP variables to load_save_disk_commands map for LOAD/SAVE commands - Add MCP variable validation in is_valid_global_variable() for SET commands - Implement has_variable() method in MCP_Threads_Handler - Add CHECKSUM command handlers for MCP VARIABLES (DISK/MEMORY/MEM) Test results improved from 28 passed / 16 failed to 49 passed / 3 failed. Remaining 3 failures are test expectation issues (boolean representation).	2 months ago
Rene Cannao	87fff9e046	Add MCP (Model Context Protocol) module skeleton Add new MCP module supporting multiple MCP server endpoints over HTTPS with JSON-RPC 2.0 protocol skeleton. Each endpoint (/mcp/config, /mcp/observe, /mcp/query, /mcp/admin, /mcp/cache) is a distinct MCP server with its own authentication configuration. Features: - HTTPS server using existing ProxySQL TLS certificates - JSON-RPC 2.0 skeleton implementation (actual protocol TBD) - 5 MCP endpoints with per-endpoint auth configuration - LOAD/SAVE MCP VARIABLES admin commands - Configuration file support (mcp_variables section) Implementation follows GenAI module pattern: - MCP_Threads_Handler: Main module handler with variable management - ProxySQL_MCP_Server: HTTPS server wrapper using libhttpserver - MCP_JSONRPC_Resource: Base endpoint class with JSON-RPC skeleton	2 months ago
Rene Cannao	59f0b8b1fa	Fix GenAI module admin commands - correct character check The bug was checking query_no_space[5] == 'A' for GENAI commands, but position 5 in "SAVE GENAI VARIABLES" is 'G', not 'A'. Fixed two locations: 1. LOAD/SAVE VARIABLES command handler (line 1659) 2. LOAD FROM CONFIG command handler (line 1734) All GenAI admin commands now work correctly: - SAVE GENAI VARIABLES TO DISK - LOAD GENAI VARIABLES FROM DISK - SAVE GENAI VARIABLES FROM RUNTIME - LOAD GENAI VARIABLES TO RUNTIME - SAVE GENAI VARIABLES TO MEMORY - LOAD GENAI VARIABLES FROM MEMORY - LOAD GENAI VARIABLES FROM CONFIG	2 months ago
Rene Cannao	c476f56f97	Add initial GenAI module placeholder Implement a new GenAI module for ProxySQL with basic infrastructure: - GenAI_Threads_Handler class for managing GenAI module configuration - Support for genai- prefixed variables in global_variables table - Dummy variables: genai-var1 (string) and genai-var2 (integer) - Config file support via genai_variables section - Flush functions for runtime_to_database and database_to_runtime - Module lifecycle: initialization at startup, graceful shutdown - LOAD/SAVE GENAI VARIABLES admin command infrastructure Core functionality verified: - Config file loading works - Variables persist in global_variables table - Disk save/load via SQL works - Module initializes and shuts down properly Related files: - include/GenAI_Thread.h: New GenAI thread handler class - lib/GenAI_Thread.cpp: Implementation with dummy variables - lib/Admin_Handler.cpp: Added GENAI command vectors and handlers - lib/Admin_FlushVariables.cpp: Added genai flush functions - lib/ProxySQL_Admin.cpp: Added init_genai_variables() and load_save_disk_commands entry - include/proxysql_admin.h: Added function declarations - lib/Makefile: Added GenAI_Thread.oo to build - src/main.cpp: Added module initialization and cleanup - src/proxysql.cfg: Added genai_variables configuration section	2 months ago
Rahim Kanji	38befb845f	Fixed PROXYSQL STOP	5 months ago
Rahim Kanji	67d383de4e	Resolved issues with admin PROXYSQL PAUSE/RESUME commands	5 months ago
René Cannaò	360646b7d5	Merge pull request #5165 from sysown/crash-sqlite-max-len processlist: Prevent crash due to SQLITE_TOOBIG	5 months ago
Wazir Ahmed	5219d4be93	processlist: Prevent crash due to SQLITE_TOOBIG - Add new mysql/pgsql variable `processlist_max_query_length`. - Min: 1K - Max: 32M - Default: 2M - Truncate current query based on the configuration before inserting into `stats_*_processlist` tables. - Refactor/fix code related to other processlist configurations. 1. `session_idle_show_processlist` value was not updated in `ProxySQL_Admin.variables`. 2. Pass processlist config as an argument to `MySQL_Threads_Handler::SQL3_Processlist` instead of using thread-local variables. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	5 months ago
René Cannaò	562e98d551	Merge pull request #5130 from wazir-ahmed/stats-schema Remove `stats` schema name from the query in stats user session	5 months ago
René Cannaò	ccf72b606c	Merge pull request #5134 from wazir-ahmed/mysqldump Improve `mysqldump` compatibility	5 months ago
René Cannaò	3b4369bba9	Merge pull request #5113 from wazir-ahmed/load-from-config-validation Add validation for `LOAD * FROM CONFIG`	5 months ago
René Cannaò	abafaccbc0	Merge pull request #5156 from sysown/v3.0_pgsql_stat_activity_issue_4682 Add backend_pid and backend_state to stats_pgsql_processlist and create stats_pgsql_stat_activity view	5 months ago
Rahim Kanji	fc4d7f76a0	Added 'SHOW FULL PGSQL ACTIVITY' and 'SHOW PGSQL ACTIVITY' handling	5 months ago
Rahim Kanji	b3e96b05b5	Added KILL PGSQL CONNECTION support	6 months ago
Wazir Ahmed	b6d796c9f2	Improve `mysqldump` compatibility Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago
Wazir Ahmed	80744b4488	Remove `stats` schema name from the query in stats user session Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago
Wazir Ahmed	7194437d77	Add `PROXYSQL SET CONFIG FILE` command - New admin command to dynamically change configuration file path Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago
Wazir Ahmed	0a336de048	Add validation for `LOAD * FROM CONFIG` - Add validation methods for `mysql_users`, `pgsql_users`, `mysql_servers`, `pgsql_servers` and `proxysql_servers` - Check for duplicates and mandatory fields - Return descriptive error messages to clients when validation fails Signed-off-by: Wazir Ahmed <wazir@proxysql.com> Co-authored-by: takaidohigasi <takaidohigasi@gmail.com>	6 months ago
René Cannaò	2551d350e9	Port to 3.0 of in-memory query logging See #4713 for further details	1 year ago
Rahim Kanji	6d614a4b99	Merge branch 'v3.0'	1 year ago
Rahim Kanji	110ce8d511	Fixed crash caused by sending single semicolon (;) on admin interface	1 year ago
Rahim Kanji	fc744096ea	Improved PROXYSQL FLUSH PGSQL QUERY CACHE	1 year ago
Rahim Kanji	720441af20	Refactored and Optimized Query Cache with PgSQL Support * Added Query Cache Support for PgSQL: * Introduced new variables for PgSQL Query Cache management: * pgsql-query_cache_size_MB * pgsql-query_cache_soft_ttl_pct * pgsql-query_cache_stores_empty_result * pgsql-query_cache_handle_warnings (not yet utilized) * Overwriting Transaction Status: * Sending current transaction state to the client when a result is returned from the query cache. * This currently does not differentiate between normal and error transaction states—both are treated as active transaction states. * Cache Lifetime Management: Replaced manual reference counting with std::shared_ptr to efficiently manage the lifecycle of cache entries, ensuring proper memory deallocation and reducing complexity. * Unified Purging Logic: Refactored the query cache purging logic to use a single thread for both MySQL and PgSQL cache entries. * Code Cleanup: Removed unnecessary methods and data members to streamline the codebase and improve maintainability. * Performance Optimization: Improved memory management, reducing memory footprint and enhancing performance.	1 year ago
Rahim Kanji	0229daeb64	Added stats_pgsql_query_digest and stats_pgsql_query_digest_reset tables	1 year ago
Rahim Kanji	e723082a1b	Refactored Query Processor Architecture - Introduced separate query processors for MySQL and PostgreSQL. - Removed unnecessary GTID logic from `PgSQL_Session`. - Added a command counter for PostgreSQL queries. - Refactored `Query_Processor` to serve as a base class containing core functionalities applicable to both MySQL and PostgreSQL. Changes: - Removed GTID-related code from `PgSQL_Logger` and `PgSQL_Session`: - bool with_gtid; - char gtid_buf[128]; - int gtid_hid; - Added new PostgreSQL-specific tables: - `pgsql_query_rules`, `pgsql_firewall_whitelist_rules`, `pgsql_query_rules_fast_routing` - `runtime_pgsql_query_rules`, `runtime_pgsql_firewall_whitelist_rules`, `runtime_pgsql_query_rules_fast_routing` - Added PostgreSQL-related variables: pgsql_thread___commands_stats pgsql_thread___query_digests pgsql_thread___query_digests_lowercase pgsql_thread___query_digests_no_digits pgsql_thread___query_digests_replace_null pgsql_thread___query_digests_normalize_digest_text pgsql_thread___query_digests_track_hostname pgsql_thread___query_digests_keep_comment pgsql_thread___query_digests_max_digest_length pgsql_thread___query_digests_max_query_length pgsql_thread___query_digests_grouping_limit pgsql_thread___query_digests_groups_grouping_limit pgsql_thread___enable_load_data_local_infile pgsql_thread___auditlog_filename pgsql_thread___auditlog_filesize pgsql_thread___eventslog_filename pgsql_thread___eventslog_filesize pgsql_thread___eventslog_default_log pgsql_thread___eventslog_format pgsql_thread___firewall_whitelist_errormsg pgsql_thread___firewall_whitelist_enabled pgsql_thread___query_processor_iterations pgsql_thread___query_processor_regex	2 years ago
Rahim Kanji	745449fcbc	Removed Client_Session.h and Client_Session.cpp	2 years ago
Rahim Kanji	5bf827d8e4	Removed type selector	2 years ago
Rahim Kanji	43d3ec82c8	Added stats_pgsql_processlist table * Implemented 'SHOW FULL PGSQL PROCESSLIST' command. * Implemented 'SHOW PGSQL PROCESSLIST' command.	2 years ago
Rahim Kanji	c35a6a24cf	Added stats_pgsql_global table	2 years ago
Rahim Kanji	6faa1b11c0	Added SHOW PGSQL VARIABLES support	2 years ago
Rene Cannao	a1fd4af1a6	Splitting ProxySQL_Admin	2 years ago

49 Commits (2c8c27bf6425ef103589dd4019b698c5fef8d2aa)