1948 Commits (ef07831780e2ee153ddad1ce59042ab90cd18ecf)

Author	SHA1	Message	Date
Rene Cannao	06aa6d6ef7	Add comprehensive Doxygen documentation for connection pool ... Added missing documentation for MySQL connection pool implementation: Header (MySQL_Tool_Handler.h): - Added MySQLConnection struct documentation with member descriptions - Added member variable documentation using ///< Doxygen style Implementation (MySQL_Tool_Handler.cpp): - Added Doxygen blocks for close() method - Added Doxygen blocks for init_connection_pool() with detailed behavior - Added Doxygen blocks for get_connection() with thread-safety notes - Added Doxygen blocks for return_connection() with reuse behavior - Added Doxygen blocks for execute_query() with JSON format documentation All new connection pool methods now have complete @brief, @param, and @return documentation following Doxygen conventions.	2 months ago
Rene Cannao	4eab519848	Implement MySQL connection pool for MySQL_Tool_Handler ... Added built-in connection pool to MySQL_Tool_Handler for direct MySQL connections to backend servers. Changes: - Added MySQLConnection struct with MYSQL* pointer, host, port, in_use flag - Added connection_pool vector, pool_lock mutex, pool_size counter - Implemented init_connection_pool() to create MYSQL connections using mysql_init/mysql_real_connect - Implemented get_connection() and return_connection() with thread-safe locking - Implemented execute_query() helper method for executing SQL and returning JSON results - Updated tool methods to use actual MySQL connections: - list_schemas: Query information_schema.schemata - list_tables: Query information_schema.tables with metadata - describe_table: Query columns, primary keys, indexes - sample_rows: Execute SELECT with LIMIT - sample_distinct: Execute SELECT DISTINCT with GROUP BY - run_sql_readonly: Execute validated SELECT queries - explain_sql: Execute EXPLAIN queries - Fixed MYSQL forward declaration (use typedef struct st_mysql MYSQL) The connection pool creates one connection per configured host:port pair with 5-second timeouts for connect/read/write operations.	2 months ago
Rene Cannao	221ff23991	Add MySQL exploration MCP tools with SQLite catalog ... Implemented MCP (Model Context Protocol) server providing tools for LLM-based MySQL database exploration: - MySQL_Catalog: SQLite-based catalog for LLM external memory with upsert, get, search, list, merge, delete operations and FTS support - MySQL_Tool_Handler: 17+ database exploration tools with guardrails: * Inventory: list_schemas, list_tables * Structure: describe_table, get_constraints, describe_view * Profiling: table_profile, column_profile * Sampling: sample_rows (max 20), sample_distinct (max 50) * Query: run_sql_readonly (max 200 rows, 2s timeout, SELECT-only) * Relationship: suggest_joins, find_reference_candidates * Catalog: catalog_upsert, catalog_get, catalog_search, catalog_list, catalog_merge, catalog_delete - MCP Module Integration: * Added 6 new configuration variables for MySQL tool handler (mysql_hosts, mysql_ports, mysql_user, mysql_password, mysql_schema, catalog_path) * Added MySQL_Tool_Handler pointer to MCP_Threads_Handler * Implemented tool routing in MCP endpoint for tools/list, tools/describe, and tools/call methods - TAP Tests: Updated to expect 14 MCP variables (was 8) Files: - include/MySQL_Catalog.h, lib/MySQL_Catalog.cpp - include/MySQL_Tool_Handler.h, lib/MySQL_Tool_Handler.cpp - include/MCP_Thread.h, lib/MCP_Thread.cpp - include/MCP_Endpoint.h, lib/MCP_Endpoint.cpp - lib/Makefile, test/tap/tests/mcp_module-t.cpp	2 months ago
Rene Cannao	81c53896bc	Fix MCP module TAP test failures ... - Add MCP variables to load_save_disk_commands map for LOAD/SAVE commands - Add MCP variable validation in is_valid_global_variable() for SET commands - Implement has_variable() method in MCP_Threads_Handler - Add CHECKSUM command handlers for MCP VARIABLES (DISK/MEMORY/MEM) Test results improved from 28 passed / 16 failed to 49 passed / 3 failed. Remaining 3 failures are test expectation issues (boolean representation).	2 months ago
Rene Cannao	245e61ee86	Make MCP_Threads_Handler a standalone independent class ... Remove unnecessary inheritance from MySQL_Threads_Handler. The MCP module should be independent and not depend on MySQL/PostgreSQL thread handlers. Changes: - MCP_Threads_Handler now manages its own pthread_rwlock_t for synchronization - Simplified init() signature (removed unused num/stack parameters) - Added ProxySQL_Main_init_MCP_module() call in main initialization phase - Include only standard C++ headers (pthread.h, cstring, cstdlib)	2 months ago
Rene Cannao	87fff9e046	Add MCP (Model Context Protocol) module skeleton ... Add new MCP module supporting multiple MCP server endpoints over HTTPS with JSON-RPC 2.0 protocol skeleton. Each endpoint (/mcp/config, /mcp/observe, /mcp/query, /mcp/admin, /mcp/cache) is a distinct MCP server with its own authentication configuration. Features: - HTTPS server using existing ProxySQL TLS certificates - JSON-RPC 2.0 skeleton implementation (actual protocol TBD) - 5 MCP endpoints with per-endpoint auth configuration - LOAD/SAVE MCP VARIABLES admin commands - Configuration file support (mcp_variables section) Implementation follows GenAI module pattern: - MCP_Threads_Handler: Main module handler with variable management - ProxySQL_MCP_Server: HTTPS server wrapper using libhttpserver - MCP_JSONRPC_Resource: Base endpoint class with JSON-RPC skeleton	2 months ago
Javier Jaramago Fernández	5c8a32a0b0	Merge branch 'v3.0' of github.com:sysown/proxysql into v3.0-handle_unexp_ping	3 months ago
Javier Jaramago Fernández	6fea828e86	Improve logging in unexpected COM_PING packet handling ... Logging messages now include 'client address', 'session status' and 'data stream status'. Client address is also logged when OK packets are dispatched, this should help tracking if a client has received the expected packets or not.	3 months ago
René Cannaò	88edaac61b	Merge pull request #5258 from sysown/misc251219 ... Documentation additions and bug fix for vacuum_stats()	3 months ago
René Cannaò	0f7ff1f374	Merge branch 'v3.0' into v3.0_pgsql-query-digest-gen-5253 ... Signed-off-by: René Cannaò <rene@proxysql.com>	3 months ago
René Cannaò	2667540fcc	Merge pull request #5237 from sysown/v3.0_pgsql-monitor-sslsupport-5205 ... Add SSL support for backend connections in PGSQL monitor	3 months ago
Javier Jaramago Fernández	d0e88599ee	Add special handling for unexpected COM_PING packets ... Implements a workaround for the handling of unexpected 'COM_PING' packets received during query processing, while a resultset is yet being streamed to the client. Received 'COM_PING' packets are queued in the form of a counter. This counter is later used to sent the corresponding number of 'OK' packets to the client after 'MySQL_Session' has finished processing the current query.	3 months ago
Rene Cannao	efe0d4fe61	Add extensive doxygen documentation for vacuum_stats and stats_pgsql_stat_activity ... This commit documents: 1. The vacuum_stats() function's purpose, behavior, and the reason why stats_pgsql_stat_activity is excluded from bulk deletion operations 2. The fact that stats_pgsql_stat_activity is a SQL VIEW (not a table) and attempting DELETE on it would cause SQLite error: "cannot modify stats_pgsql_stat_activity because it is a view" The documentation explains: - Why TRUNCATE stats_mysql_query_digest triggers vacuum_stats(true) - Why both MySQL and PostgreSQL tables are cleared regardless of protocol - How the view is automatically cleared via its underlying table stats_pgsql_processlist - The importance of keeping the view excluded from deletion lists	3 months ago
Rahim Kanji	42864e8867	Improved Tokenizer for PostgreSQL ... - Added `process_pg_typecast()` to handle PostgreSQL type cast syntax (::) - Recognizes type casts in various contexts: 'value'::type, column::type, etc. - Added `process_array_literal()` for PostgreSQL array processing - Handles both ARRAY[] constructor and {} literal syntax - Processes multi-dimensional arrays and nested array structures - Added `process_literal_prefix_type()` for PostgreSQL prefixed literals - Processes E'' escape string constants with backslash escapes - Handles U&'' Unicode string literals with optional UESCAPE clauses - Supports x'' hex string literals and b'' bit string literals - Manages B'' bit strings and bytea literals (\\xDEADBEEF format) - Added `process_replace_boolean()` for boolean literal replacement - Replaces TRUE and FALSE literals with parameter placeholders - Maintains case-insensitive matching (true, True, TRUE, etc.) - Preserves boolean context in expressions and WHERE clauses	3 months ago
Rene Cannao	2987242d4f	Fix cache_empty_result=0 not caching non-empty resultsets (issue #5248) ... The `cache_empty_result` field in query rules has three possible values: • -1: Use global setting (`query_cache_stores_empty_result`) • 0: Do NOT cache empty resultsets, but cache non-empty resultsets • 1: Always cache resultsets (both empty and non-empty) Previously, when `cache_empty_result` was set to 0, nothing was cached at all, even for non-empty resultsets. This prevented users from disabling caching for empty resultsets while still allowing caching of non-empty resultsets on a per-rule basis. Changes: 1. Modified caching logic in MySQL_Session.cpp and PgSQL_Session.cpp to add the condition `(qpo->cache_empty_result == 0 && MyRS->num_rows)` (MySQL) and `(qpo->cache_empty_result == 0 && num_rows)` (PgSQL) to allow caching when cache_empty_result=0 AND result has rows. 2. Added comprehensive Doxygen documentation in query_processor.h explaining the semantics of cache_empty_result values. 3. Updated Query_Processor.cpp with inline comments explaining the three possible values. Now when cache_empty_result is set to 0: - Empty resultsets (0 rows) are NOT cached - Non-empty resultsets (>0 rows) ARE cached - This matches the intended per-rule behavior described in issue #5248. Fixes: https://github.com/sysown/proxysql/issues/5248	3 months ago
Rahim Kanji	fae283cf7e	Add SSL and non-SSL connection OK metrics for PostgreSQL monitor connections ... Adds two new metrics, ssl_connections_OK and non_ssl_connections_OK, to improve visibility into PostgreSQL monitor connection status.	3 months ago
Rene Cannao	6c97d3d244	Add extensive Doxygen documentation for ProxySQL_Config and Read_Global_Variables_from_configfile ... This commit adds detailed Doxygen documentation for: 1. The ProxySQL_Config class - describes its role in configuration management 2. The Read_Global_Variables_from_configfile() method - documents its behavior, parameters, return value, and the automatic prefix stripping feature The documentation explains the automatic prefix stripping behavior that handles cases where users mistakenly include module prefix (e.g., "mysql-") in variable names within configuration files.	3 months ago
Rahim Kanji	895c814c77	Added utility functions to support pgsql query digest testing	3 months ago
Rahim Kanji	285fb1b4e1	Add PostgreSQL dialect support: dollar-quoted strings, identifier quoting, and dialect-specific comment rules ... This change introduces PostgreSQL-aware tokenization by adding support for dollar-quoted strings, PostgreSQL’s double-quoted identifiers, and its comment rules. The tokenizer now correctly parses $$…$$ and $tag$…$tag$, treats " as an identifier delimiter in PostgreSQL, disables MySQL-only # comments, and accepts -- as a comment starter without requiring a trailing space. All new behavior is fully isolated behind the dialect flag to avoid impacting MySQL parsing. Add PostgreSQL dollar-quoted strings * New parser state: st_dollar_quote_string. * Recognizes $$ … $$ and $tag$ … $tag$ sequences. * Tracks opening tag and searches for matching terminator. * Normalizes entire literal to ?. * Integrated into get_next_st() and stage_1_parsing().	3 months ago
René Cannaò	b73160ef5f	Merge pull request #4901 from sysown/v3.0_wait_timeout ... [WIP] Setting client side wait_timeout	3 months ago
Rene Cannao	5a7b22181f	Fix metrics collection for wait_timeout counters ... The get_status_variable() function was only scanning worker threads but ignoring auxiliary threads (idle threads) where timeout terminations are detected. This caused the timeout termination counter to show incorrect/zero values. - Added idle thread scanning to both overloaded versions of get_status_variable() function - Now properly collects metrics from both worker and idle threads - Fixes the issue where proxysql_mysql_timeout_terminated_connections_total showed zero despite actual timeout terminations Resolves the metrics reading issue identified in the previous commits.	3 months ago
René Cannaò	65dbe904f0	Merge pull request #5199 from sysown/v3.0_refactor_monitoring_ping ... Fix artificially high ping latency in MySQL backend monitoring	3 months ago
Rene Cannao	dc4694d656	Refactor idle session scanning and improve test precision ... Code improvements: - Extract SESS_TO_SCAN_idle_thread constant to header file for better maintainability - Replace magic number 128 with named constant in idle_thread_to_kill_idle_sessions() - Improve code readability and consistency in session scanning logic Test enhancements: - Add mysql-poll_timeout configuration for more precise timeout testing - Reduce test sleep times to 13 seconds for faster test execution - Add diagnostic messages to clearly show timeout configurations in test output - Ensure tests properly validate timeout enforcement with precise timing The changes improve code maintainability and make tests more reliable and faster while maintaining accurate timeout validation.	3 months ago
Rene Cannao	86cc7cd3da	session: Fix wait_timeout member variable declaration and usage ... - Add wait_timeout member variable declaration to Base_Session class - Fix constructor initialization to use this->wait_timeout - Fix assignment in handler to properly scope member variable - Resolves compilation error for wait_timeout functionality	3 months ago
René Cannaò	b641c0d627	Merge pull request #5232 from sysown/fix/issue-4855 ... Fix issue 4855: Incorrect affected rows reporting for DDL queries	3 months ago
René Cannaò	3c4e09fec0	Merge pull request #5225 from sysown/v3.0_refactor_prepared_statement_cache_design_5211 ... Refactored Prepared-Statement Cache Design (Lock-Free Hot Path) - Part 2	3 months ago
Rahim Kanji	1251e4d539	Add Xoshiro128++ pseudo-random number generator to replace rand()	4 months ago
René Cannaò	d188715a7d	Merge branch 'v3.0' into fix/issue-4855 ... Signed-off-by: René Cannaò <rene@proxysql.com>	4 months ago
Rene Cannao	a577491f42	Refactor issue 4855 fix: Use sqlite3_total_changes64 difference approach ... PROBLEM: The initial fix used a DDL detection approach which required maintaining a list of query types that should return 0 affected rows. This approach was brittle and could miss edge cases like commented queries or complex statements. SOLUTION: Instead of detecting DDL queries, use sqlite3_total_changes64() to measure the actual change count before and after each query execution. The difference between total_changes before and after represents the true affected rows count for the current query, regardless of query type. CHANGES: - Added proxy_sqlite3_total_changes64 function pointer and initialization - Rewrote execute_statement() and execute_statement_raw() to use total_changes difference approach - This automatically handles all query types (DDL, DML, comments, etc.) - Added comprehensive TAP test covering INSERT, CREATE, DROP, VACUUM, UPDATE, and BEGIN operations BENEFITS: - More robust and accurate than DDL detection approach - Handles edge cases like commented queries automatically - No maintenance overhead for new query types - Simpler and cleaner implementation - Still fixes both Admin interface and SQLite3 Server This approach is mathematically sound: affected_rows = total_changes_after - total_changes_before, which gives the exact number of rows changed by the current query execution. Fixes #4855	4 months ago
Rahim Kanji	1a48aadf5e	Code cleanup	4 months ago
Wazir Ahmed	50c60284e6	gtid: Refactor reconnect logic & prevent `events_count` reset ... - This patch was originally added by commit 0a70fd5 and reverted by 8d1b5b5, prior to the release of `v3.0.3`. - The following issues are addressed in this update, - Fix for `use-after-free` issue which occured during CI test. - Fix for deadlock issue between `GTID_syncer` and `MySQL_Worker`. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	4 months ago
Rahim Kanji	c0f99c0e15	Refactor: Improved Prepared-Statement Cache Design (Lock-Free Hot Path) #5211 ... Concurrency and Memory Management * Lock-Free Ref Counting: Replaced global mutex-protected integer reference counts with `std::atomic<uint32_t>` within `PgSQL_STMT_Global_info`, eliminating lock contention during statement referencing. * Modern Ownership: Adopted std::shared_ptr<const PgSQL_STMT_Global_info> for global and local storage, providing automatic, thread-safe memory and lifecycle management. * Memory Optimization: Removed redundant auxiliary maps `global_id_to_stmt_names` and `map_stmt_id_to_info` from local and global statement managers respectively, reducing overall memory overhead. * Optimized Purging: Statement removal logic was simplified for efficiently identifying and cleaning up unused statements. Hot Path Performance (`BIND`, `DESCRIBE`, `EXECUTE`) * Bypassed Global Lookups: Local session maps now store the `shared_ptr` directly, removing the need to acquire the global lock and search the global map during hot path operations. * Direct Refcount Manipulation: Refcount modification functions now operate directly on the passed statement object, eliminating the overhead of searching the global map to find the object pointer based on statement id. Safety and Protocol Logic (`PARSE`) * Efficient Statement Reuse: Implemented a **local fast path** check for the unnamed statement (`""`), allowing immediate reuse of an identical query (same hash) upon re-parse, which bypasses global processing and locks. Cleanup * Cleaned up and class rename `PgSQL_STMT_Manager_v14` -> `PgSQL_STMT_Manager`.	4 months ago
Rene Cannao	44aa606caa	Implement fast forward grace close feature to prevent data loss ... Problem: In fast forward mode, ProxySQL forwards packets directly from client to backend without buffering them. If the backend connection closes unexpectedly (e.g., due to server crash, network failure, or other issues), ProxySQL immediately closes the client session. This can result in data loss because the client may have sent additional data that hasn't been fully transmitted yet, as ProxySQL does not wait for the output buffers to drain. Solution: Implement a configurable grace period for session closure in fast forward mode. When the backend closes unexpectedly, instead of closing the session immediately, ProxySQL waits for a configurable timeout (fast_forward_grace_close_ms, default 5000ms) to allow any pending client output data to be sent. During this grace period: - If the client output buffers become empty, the session closes gracefully. - If the timeout expires, the session closes anyway to prevent indefinite hanging. Changes: - Added global variable mysql_thread___fast_forward_grace_close_ms (0-3600000ms) - Added session flags: backend_closed_in_fast_forward, fast_forward_grace_start_time - Added data stream flag: defer_close_due_to_fast_forward - Modified MySQL_Data_Stream::read_from_net() to detect backend EOF and initiate grace close if client buffers are not empty - Modified MySQL_Session::handler() FAST_FORWARD case to implement grace close logic with timeout and buffer checks - Added extensive inline documentation explaining the feature and its mechanics This prevents data loss in fast forward scenarios while maintaining bounded session lifetime.	4 months ago
Rahim Kanji	d84444724a	Replaced use of the generic write_generic() helper with direct packet construction for selected PostgreSQL protocol messages to reduce overhead and improve performance.	4 months ago
Rahim Kanji	e744c2bbb7	Optimize transaction command parsing to avoid unnecessary tokenization ... Previously, the parser always tokenized the full command, even when we only needed to check whether it was a transaction command. Now, it first extracts the first word to determine relevance and performs full tokenization only when necessary.	4 months ago
Rahim Kanji	1c3c42959c	Improve ping accuracy ... - Introduced batching for ping task dispatch (default: 30 servers per batch) - Moved mysql_server_ping_log writes to a thread pool	4 months ago
Rahim Kanji	7a3a5c71df	Optimize hot path: replace std::string with char[] to avoid heap	4 months ago
Javier Jaramago Fernández	64b16172c7	Change 'COM_STMT_EXECUTE' params length decoding ... According to MySQL protocol, variable length strings are encoded using length encoded integers. For reference, see: - https://dev.mysql.com/doc/dev/mysql-server/9.4.0/page_protocol_com_stmt_execute.html - https://dev.mysql.com/doc/dev/mysql-server/9.4.0/page_protocol_basic_dt_integers.html#a_protocol_type_int2 The protocol specifies that values greater than 2^24 (16777216) should be encoded using '0xFE + 8-byte integer'. Yet, in reality MySQL ignores the upper section of these 8-byte integers, treating them effectively like '4-bytes'. For the sake of compatibility this commit changes the decoding behavior for 'COM_STMT_EXECUTE' to match MySQL one. This different is subtle but important, since in practice MySQL itself doesn't use the '8 bytes' from the field. This means that connectors that are compatible with MySQL could find issues when sending these packets through ProxySQL (like NodeJS 'mysql2' connector which writes the 8-bytes as a 4-bytes duplication, motivating these changes), situation that could result in rejection due to malformed packet detection (or crashes/invalid handling in the worse case scenario). The previous decoding function is now renamed into 'mysql_decode_length_ll' to honor MySQL naming 'net_field_length_ll'. For now, this protocol change is limited to 'COM_STMT_EXECUTE'.	4 months ago
Javier Jaramago Fernández	e35973b226	Add boundary checks in 'MySQL_Protocol::get_binds_from_pkt' ... These checks should prevent crashes when packets are received holding invalid lengths in the processed params.	4 months ago
Lisandro Pin	6cb522283d	Implement a ignore_min_gtid_annotation global variable for ProxySQL ... When true, all `min_gtid` query annotations are ignored; see https://proxysql.com/documentation/query-annotations/ for details. This is useful on ProxySQL setups with multiple layers, where some layers mandate GTID-based routing while others don't.	5 months ago
Rahim Kanji	61ba182465	Introduce inline functions for efficient ASCII whitespace detection and uint32-to-string conversion	5 months ago
Wazir Ahmed	8d1b5b597c	Revert "gtid: Refactor reconnect logic & prevent events_count reset" ... This reverts commit 0a70fd515f.	5 months ago
René Cannaò	5018cb66d9	Merge pull request #5168 from sysown/v3.0-debug_log_pass ... Allow to dump passwords used during auth in debug builds	5 months ago
René Cannaò	c6cbce34b1	Merge pull request #5117 from wazir-ahmed/gtid-reconn ... gtid: Refactor reconnect logic & prevent `events_count` reset	5 months ago
René Cannaò	360646b7d5	Merge pull request #5165 from sysown/crash-sqlite-max-len ... processlist: Prevent crash due to SQLITE_TOOBIG	5 months ago
Wazir Ahmed	5219d4be93	processlist: Prevent crash due to SQLITE_TOOBIG ... - Add new mysql/pgsql variable `processlist_max_query_length`. - Min: 1K - Max: 32M - Default: 2M - Truncate current query based on the configuration before inserting into `stats_*_processlist` tables. - Refactor/fix code related to other processlist configurations. 1. `session_idle_show_processlist` value was not updated in `ProxySQL_Admin.variables`. 2. Pass processlist config as an argument to `MySQL_Threads_Handler::SQL3_Processlist` instead of using thread-local variables. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	5 months ago
René Cannaò	7022833213	Merge pull request #5163 from sysown/v3.0_assert_to_warning ... verify_server_variables() -> Change assert to warning + Various improvements	5 months ago
Javier Jaramago Fernández	4304b995a3	Improve debug message with context dump (passwords) during auth handshake ... This message is dump with each call to 'process_pkt_handshake_response' printing the updated context. When the verbosity value for module 'debug_mysql_protocol' is >= 5, the stored and client supplied passwords will be dumped in HEX format, for values < 5, the passwords will be masked.	5 months ago
René Cannaò	436bcccc62	Merge pull request #5158 from sysown/v3.0.3-makefiles_cleanup ... V3.0.3 makefiles cleanup	5 months ago
Miro Stauder	d020f577b2	reduce duplication in makefiles, use include for common stuff	5 months ago