proxysql

Commit Graph

Author	SHA1	Message	Date
Rahim Kanji	ca45405942	Merge remote-tracking branch 'Master/v3.0-moveInfra' into v3.0_pgsql-pipeline-set-reset-discard-fix-5432	1 month ago
Rahim Kanji	000a9794e4	Don't short-circuit in pipeline mode, fall through to backend execution	1 month ago
Rahim Kanji	abb66cc19f	Addressed AI review feeback	1 month ago
Rene Cannao	6b46b26931	chore: implement PR review feedback for release tier system - Makefile: normalized GIT_VERSION by stripping 'v' prefix before arithmetic. - Makefile: corrected .PHONY declaration for build_deps_debug_default. - Admin Handler: properly guarded TSDB commands to return error when disabled. - REST API: removed redundant nested guards inside tsdb_resource class. - Statistics: implemented rigorous input validation for set_variable. - Statistics: added return value checks and rollback logic for TSDB migrations. - Statistics: implemented bounded concurrency (batching) for backend probes. - FFTO: implemented cumulative buffer limit enforcement in MySQL and PgSQL. - FFTO: ensured initial packet capture in PgSQL_Session by early initialization. - FFTO: improved reassembly buffer compaction logic. - Headers: default-initialized ffto_bypassed to false. - Code cleanup: merged redundant PROXYSQLFFTO guards.	1 month ago
Rene Cannao	c2f678a9de	feat: implement release tiers and feature guards for v3.0, v3.1, and v4.0 This commit introduces a structured release tier system controlled by environment variables, allowing three distinct ProxySQL versions to be built from the same codebase while maintaining a clear upgrade path. Key Changes: - Tier 1 (v3.0.x Stable): Default build including bug fixes and core enhancements. - Tier 2 (v3.1.x Innovative): Enabled via PROXYSQL31=1. Adds FFTO and TSDB. - Tier 3 (v4.0.x AI/MCP): Enabled via PROXYSQLGENAI=1. Adds the AI stack. Implementation Details: - Makefile: Implemented dynamic versioning logic that increments the minor version for v3.1 and the major version for v4.0. Added tier documentation. - Feature Guards: Introduced PROXYSQLFFTO and PROXYSQLTSDB guards across ProxySQL_Admin, ProxySQL_Statistics, REST API, and HTTP server. - FFTO: Guarded Fast Forward Traffic Observer in MySQL and PgSQL sessions, including thread-local variables and configuration parameters. - TSDB: Guarded the entire Time Series Database subsystem, including SQLite table definitions, background loops, and REST API endpoints. - Build Fixes: * Fixed 'proxy_debug_func' linkage by adding extern "C" and moving the definition outside of DEBUG guards to ensure non-debug builds link correctly. * Corrected misplaced/nested DEBUG and PROXYSQLFFTO guards. * Verified build stability for all tiers (unset, PROXYSQL31=1, PROXYSQLGENAI=1).	1 month ago
Rahim Kanji	e84433159b	Merge remote-tracking branch 'v3.0' into v3.0_pgsql-pipeline-set-reset-discard-fix-5432	1 month ago
Rahim Kanji	96bc668735	Fix pipeline mode transaction state handling and SET/RESET/DISCARD support Addresses issues with PostgreSQL Extended Query Protocol (Pipeline mode) in ProxySQL: Transaction State Handler Migration: - Moved handle_transaction_state() from PgSQL_Session::RequestEnd() to PgSQL_Session::handler() - Now called only for client queries (PROCESSING_QUERY/PROCESSING_STMT_EXECUTE) - Prevents internal SET commands from triggering transaction logic incorrectly - Added transaction state reset at 7 locations for clean connection pool return SET/RESET/DISCARD Pipeline Mode Support (lib/PgSQL_Session.cpp): - SET statements now execute properly in pipeline mode via RunQuery() - RESET ALL and DISCARD ALL blocked with proper error messages in pipeline mode - Added startup parameter mismatch detection for RESET with locked hostgroup	1 month ago
Rahim Kanji	742ff05ecc	Fix PostgreSQL transaction state management for pipeline mode Move transaction state tracking from RequestEnd to connection handler to enable proper binary pipeline mode support: - Call handle_transaction_state() at ASYNC_QUERY_END/ASYNC_STMT_EXECUTE_END in PgSQL_Connection instead of RequestEnd - Add reset_state() calls in 7 locations to prevent stale state: * Session reset * Connection detach (finishQuery) * Error handlers (4 paths) * Base session housekeeping - Refactor destructor to use reset_state() Fixes transaction state tracking in pipeline mode where connections can be returned to pool mid-transaction.	1 month ago
Rahim Kanji	4d86e6949c	Add defensive null guard before reading PSarrayIN->len	1 month ago
Rahim Kanji	8c9af56f38	Address AI review feedback	1 month ago
Rahim Kanji	05b5821fbc	Fix integer underflow in PostgreSQL query packet handling and null terminator checking * Add validation to reject packets smaller than 6 bytes before processing * Add null terminator check	1 month ago
Rahim Kanji	866c3593bb	Set query retries for PostgreSQL extended query protocol operations Add query_retries_on_failure configuration for PARSE, DESCRIBE, and EXECUTE operations in PostgreSQL extended query protocol.	1 month ago
Rahim Kanji	27119169e4	Reset extended query frame if an unsupported message type is received	1 month ago
Rahim Kanji	176362c77d	Addressed feedback from AI review	1 month ago
Rahim Kanji	a82d653c0b	Handle late COPY protocol messages after session exits fast_forward mode When a COPY FROM STDIN operation encounters an error, the session switches back to normal mode. However, the client may have already pipelined CopyData('d'), CopyDone('c'), or CopyFail('f') messages that are still in the input queue. Previously, these messages fell through to the default case, generating a spurious "Feature not supported" error. This change adds explicit handling to discard these messages when session_fast_forward == SESSION_FORWARD_TYPE_NONE, preventing the race condition from causing errors. The client does not expect a response for these messages in this scenario.	2 months ago
Rene Cannao	e53d2c76f9	Address PR #5410 review findings across FFTO, sessions, TAP, and docs Apply actionable bot review feedback from PR #5410 without behavior drift:\n- fix MySQL/PgSQL FFTO digest normalization and max-length hashing\n- add early FFTO packet/message size guards for bypass safety\n- restore MySQL session GTID reset and harden GenAI async error ownership\n- add null-check for stripped KILL query normalization path\n- call FFTO on_close() before observer reset/bypass in session flows\n- fix processlist query-options default pagination limit semantics\n- fix TAP noise_failures locking race and noise-aware plan count\n- extend wait_get_enpoint_ready with optional basic-auth passthrough\n- update MCP architecture docs from observe->stats naming\n- clean corrupted character in pgsql query-cache test comment\n\nValidation run:\n- git diff --check\n- make -C test/tap/tap tap.o utils_mariadb.o utils_mysql57.o utils_mysql8.o -j4\n- make -C lib obj/MySQLFFTO.oo obj/PgSQLFFTO.oo obj/MySQL_Session.oo obj/PgSQL_Session.oo obj/Query_Processor.oo -j4	2 months ago
Rene Cannao	1258a1f2f2	Address PR reviews: improve performance, safety, and robustness of FFTO	2 months ago
Rahim Kanji	194e20b3e8	Revert "Reset current_hostgroup to default when no query rule matches" This reverts commit `1b3d20388e`.	2 months ago
Rene Cannao	3a3be5a559	Implement Fast Forward Traffic Observer (FFTO) for MySQL and PostgreSQL - Added TrafficObserver base interface - Implemented MySQLFFTO and PgSQLFFTO classes for protocol-aware observation - Integrated FFTO hooks into MySQL and PostgreSQL session handlers in FAST_FORWARD state - Added configuration variables: mysql-ffto_enabled, mysql-ffto_max_buffer_size, pgsql-ffto_enabled, pgsql-ffto_max_buffer_size - Implemented query digest extraction and reporting parity with internal stats - Added session-level bypass when packet size exceeds buffer threshold - Included architectural documentation in doc/ffto_design.md	2 months ago
Rahim Kanji	1b3d20388e	Reset current_hostgroup to default when no query rule matches In the extended query protocol flow (Parse/Bind/Execute/Sync messages), queries without matching query rules were incorrectly retaining the current_hostgroup value from the previous query instead of being reset to the default_hostgroup. This happened because: - Simple query protocol goes through STATE_SLEEP between queries, which resets current_hostgroup to default_hostgroup - Extended query protocol does NOT go through STATE_SLEEP between Parse/Bind/Execute messages, so current_hostgroup was never reset	2 months ago
Rene Cannao	af58865b06	Fix uninitialized memory in QueryParserArgs.buf Use memset to zero-initialize the entire QueryParserArgs struct instead of individually setting only digest_text and first_comment to NULL. This fixes Valgrind errors about use of uninitialised values in strcasestr() when reading from the uninitialized buf array. Fixes errors like: Use of uninitialised value of size 8 at tolower (ctype.c:46) by strcasestr by MySQL_Connection::ProcessQueryAndSetStatusFlags(char*)	2 months ago
René Cannaò	2637d28f36	Merge pull request #5299 from sysown/v3.0_pg-cancel-terminate-backend-param-support_5298 Add parameterized PID support for pg_cancel_backend/pg_terminate_backend in extended query protocol	3 months ago
Rahim Kanji	9ec045ca74	Fix PostgreSQL deadlock with Close Statement flood exceeding threshold_resultset_size Bug Description: ProxySQL would deadlock when processing extended query frames where: 1. Many Close Statement messages accumulate responses in PSarrayOUT 2. Total response size exceeds pgsql-threshold_resultset_size 3. A backend operation (Describe/Execute) follows in the same frame Root Cause: - Close Statement operations are handled locally by ProxySQL (no backend routing) - Their CloseComplete responses accumulate in PSarrayOUT - When threshold_resultset_size is exceeded, ProxySQL stops reading from backend - Subsequent backend operations (Describe/Execute) need backend responses to complete - This creates a deadlock: ProxySQL won't read, backend operation can't complete - Extended query frame never finishes, query times out The Fix: When PSarrayOUT exceeds threshold_resultset_size and a backend operation is pending, ProxySQL now flushes all accumulated data in PSarrayOUT to the client first, then continues processing backend operations. This breaks the deadlock by clearing the buffer before attempting to read more data from the backend.	3 months ago
Rahim Kanji	67cbe46450	Simplify PID extraction Using bind message to obtain parameter information, rather than determining whether the query is parameterized from the query itself. Multiple parameters are not possible in this case, as PostgreSQL itself rejects multi-parameter pg_cancel_backend() and pg_terminate_backend() and only accepts a single parameter for these functions.	3 months ago
Rahim Kanji	5066ddd181	Removed isdigit	3 months ago
Rahim Kanji	ce42c188f5	Improvements	3 months ago
Rahim Kanji	a1e10e3055	Add parameterized PID support for pg_cancel_backend/pg_terminate_backend This commit extends the existing pg_cancel_backend() and pg_terminate_backend() support to work with parameterized queries in the extended query protocol. While literal PID values were already supported in both simple and extended query protocols, this enhancement adds support for parameterized queries like SELECT pg_cancel_backend($1).	3 months ago
Rahim Kanji	556b1023c4	Removed change_user_auth_switch flag	3 months ago
René Cannaò	0f7ff1f374	Merge branch 'v3.0' into v3.0_pgsql-query-digest-gen-5253 Signed-off-by: René Cannaò <rene@proxysql.com>	4 months ago
Rahim Kanji	42864e8867	Improved Tokenizer for PostgreSQL - Added `process_pg_typecast()` to handle PostgreSQL type cast syntax (::) - Recognizes type casts in various contexts: 'value'::type, column::type, etc. - Added `process_array_literal()` for PostgreSQL array processing - Handles both ARRAY[] constructor and {} literal syntax - Processes multi-dimensional arrays and nested array structures - Added `process_literal_prefix_type()` for PostgreSQL prefixed literals - Processes E'' escape string constants with backslash escapes - Handles U&'' Unicode string literals with optional UESCAPE clauses - Supports x'' hex string literals and b'' bit string literals - Manages B'' bit strings and bytea literals (\\xDEADBEEF format) - Added `process_replace_boolean()` for boolean literal replacement - Replaces TRUE and FALSE literals with parameter placeholders - Maintains case-insensitive matching (true, True, TRUE, etc.) - Preserves boolean context in expressions and WHERE clauses	4 months ago
Rene Cannao	2987242d4f	Fix cache_empty_result=0 not caching non-empty resultsets (issue #5248 ) The `cache_empty_result` field in query rules has three possible values: • -1: Use global setting (`query_cache_stores_empty_result`) • 0: Do NOT cache empty resultsets, but cache non-empty resultsets • 1: Always cache resultsets (both empty and non-empty) Previously, when `cache_empty_result` was set to 0, nothing was cached at all, even for non-empty resultsets. This prevented users from disabling caching for empty resultsets while still allowing caching of non-empty resultsets on a per-rule basis. Changes: 1. Modified caching logic in MySQL_Session.cpp and PgSQL_Session.cpp to add the condition `(qpo->cache_empty_result == 0 && MyRS->num_rows)` (MySQL) and `(qpo->cache_empty_result == 0 && num_rows)` (PgSQL) to allow caching when cache_empty_result=0 AND result has rows. 2. Added comprehensive Doxygen documentation in query_processor.h explaining the semantics of cache_empty_result values. 3. Updated Query_Processor.cpp with inline comments explaining the three possible values. Now when cache_empty_result is set to 0: - Empty resultsets (0 rows) are NOT cached - Non-empty resultsets (>0 rows) ARE cached - This matches the intended per-rule behavior described in issue #5248. Fixes: https://github.com/sysown/proxysql/issues/5248	4 months ago
Rahim Kanji	1a48aadf5e	Code cleanup	4 months ago
Rahim Kanji	c0f99c0e15	Refactor: Improved Prepared-Statement Cache Design (Lock-Free Hot Path) #5211 Concurrency and Memory Management * Lock-Free Ref Counting: Replaced global mutex-protected integer reference counts with `std::atomic<uint32_t>` within `PgSQL_STMT_Global_info`, eliminating lock contention during statement referencing. * Modern Ownership: Adopted std::shared_ptr<const PgSQL_STMT_Global_info> for global and local storage, providing automatic, thread-safe memory and lifecycle management. * Memory Optimization: Removed redundant auxiliary maps `global_id_to_stmt_names` and `map_stmt_id_to_info` from local and global statement managers respectively, reducing overall memory overhead. * Optimized Purging: Statement removal logic was simplified for efficiently identifying and cleaning up unused statements. Hot Path Performance (`BIND`, `DESCRIBE`, `EXECUTE`) * Bypassed Global Lookups: Local session maps now store the `shared_ptr` directly, removing the need to acquire the global lock and search the global map during hot path operations. * Direct Refcount Manipulation: Refcount modification functions now operate directly on the passed statement object, eliminating the overhead of searching the global map to find the object pointer based on statement id. Safety and Protocol Logic (`PARSE`) * Efficient Statement Reuse: Implemented a local fast path check for the unnamed statement (`""`), allowing immediate reuse of an identical query (same hash) upon re-parse, which bypasses global processing and locks. Cleanup * Cleaned up and class rename `PgSQL_STMT_Manager_v14` -> `PgSQL_STMT_Manager`.	5 months ago
Rahim Kanji	d84444724a	Replaced use of the generic write_generic() helper with direct packet construction for selected PostgreSQL protocol messages to reduce overhead and improve performance.	5 months ago
Rahim Kanji	e744c2bbb7	Optimize transaction command parsing to avoid unnecessary tokenization Previously, the parser always tokenized the full command, even when we only needed to check whether it was a transaction command. Now, it first extracts the first word to determine relevance and performs full tokenization only when necessary.	5 months ago
Rahim Kanji	7a3a5c71df	Optimize hot path: replace std::string with char[] to avoid heap	5 months ago
Rahim Kanji	9eb934e7f0	Buffer response until Extended Query frame completes; send early only if resultset threshold is reached.	5 months ago
René Cannaò	a44dee127d	Merge pull request #5174 from sysown/v3.0_optimise_extended_query_response Removed unnecessary writeout()	6 months ago
Javier Jaramago Fernández	2668de77c2	Fix compilation warning for 'strncpy' due string truncation As per the GCC manual, calls to 'strn*' function family shall be replaced by 'memcpy' when truncation (no null termination) is expected.	6 months ago
Javier Jaramago Fernández	c0978f9371	Fix potential 'use-after-free' by 'SHOW PROCESSLIST' (PgSQL) Accesses by 'stats___pgsql_processlist' to 'myconn->query.ptr' could lead to invalid memory accesses, as the pointed query could already have been free by the session after being issued.	6 months ago
Rahim Kanji	8291361bc1	Removed unnecessary writeout()	6 months ago
René Cannaò	360646b7d5	Merge pull request #5165 from sysown/crash-sqlite-max-len processlist: Prevent crash due to SQLITE_TOOBIG	6 months ago
Wazir Ahmed	5219d4be93	processlist: Prevent crash due to SQLITE_TOOBIG - Add new mysql/pgsql variable `processlist_max_query_length`. - Min: 1K - Max: 32M - Default: 2M - Truncate current query based on the configuration before inserting into `stats_*_processlist` tables. - Refactor/fix code related to other processlist configurations. 1. `session_idle_show_processlist` value was not updated in `ProxySQL_Admin.variables`. 2. Pass processlist config as an argument to `MySQL_Threads_Handler::SQL3_Processlist` instead of using thread-local variables. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago
Rahim Kanji	ed20d09b8a	Replaced std::get_if with std::visit	6 months ago
René Cannaò	ecad6f9249	Merge pull request #5160 from sysown/v3.0_disable_pgsql_prometheus_5159 Separate Query Cache Metrics for MySQL and PostgreSQL & Disable Prometheus Metrics for PostgreSQL	6 months ago
René Cannaò	73c15ba3ef	Merge pull request #5153 from sysown/v3.0_transaction_alias_support_5152 Add support for PostgreSQL transaction command aliases and ROLLBACK AND CHAIN	6 months ago
René Cannaò	01862a247a	Merge pull request #5149 from sysown/v3.0_track_search_path_param_5148 Add search_path parameter to tracked ProxySQL variables list - PgSQL	6 months ago
René Cannaò	42f6d9d74f	Merge pull request #5144 from sysown/v3.0_datestyle-parser-tweak Align DateStyle parsing with PostgreSQL	6 months ago
Rahim Kanji	cb2062c1d1	Few fixes	6 months ago
Rahim Kanji	45271572ec	Add support for PostgreSQL transaction command aliases and ROLLBACK AND CHAIN	6 months ago

1 2 3 4 5

220 Commits (6d6015a2034dbee2ea4eb50d2e84381df008fa46)