240 Commits (7ddb3dc015d1440e8a0fb0bbafa2df9aaaa2669d)

Author	SHA1	Message	Date
René Cannaò	3e7b77456e	Merge branch 'v3.0' into v3.0_partition-gate	1 month ago
Rene Cannao	b04caafef1	fix(coverage): unbreak lcov capture for {MySQL,PgSQL}_Session.cpp ... Both lib/PgSQL_Session.cpp and lib/MySQL_Session.cpp contain a doxygen @note that *literally spells out* the strings `LCOV_EXCL_START` and `LCOV_EXCL_STOP` while documenting why those directives wrap the `assert(0)` block below the comment. lcov's geninfo scans every source line (comments included) for those tokens and treats them as real directives -- so the doc occurrence opens an exclude block, the real opener on the next assert opens it again, and geninfo aborts with: ERROR: <file>: overlapping exclude directives. Found LCOV_EXCL_START at line N - but no matching LCOV_EXCL_STOP for LCOV_EXCL_START at line M That error short-circuits the LCOV capture path in `test/infra/control/run-unit-tests-asan-coverage.bash` so `coverage/lcov.info` is never produced. The downstream "Upload coverage LCOV file" workflow step has been silently warning "No files found" for at least three runs (compare artifact listings of the three most recent CI-unit-tests-asan-coverage successes -- all zero artifacts), which in turn means the new Codecov upload step in #5817 also had nothing to send. Fix: rewrite the @note paragraph in both files so it describes what the directives do without literally containing the token strings, and explicitly call out that this is intentional. The wrapper directives around the actual `assert(0)` are unchanged. A clean lcov capture validates with `make ubuntu24-tap` followed by `docker compose run --rm --entrypoint bash ubuntu24_dbg_build -lc \ 'cd /opt/proxysql && test/infra/control/run-unit-tests-asan-coverage.bash'` which should produce a non-empty `coverage/lcov.info`.	1 month ago
Rene Cannao	a315ebe0be	fix(set-walker #150): document algo-3-vs-algo-0/1/2 whitespace divergence ... #150 in pgsql-set_parameter_validation_test-t (`SET search_path = '"$user" , public'`) sees algorithm-dependent SHOW output because of an existing destructive behaviour in the regex SET parser path: PgSQL_Set_Stmt_Parser::set_query() runs remove_spaces() on the whole SET query, which collapses every whitespace run -- INCLUDING runs inside string literals -- to a single space before parsing. So under algorithms 0/1/2, PG receives `'"$user" , public'` and SHOW returns `"""$user"" , public"`. Under algorithm 3 (ParserSQL walker) the query is parsed as-is, the original `'"$user" , public'` reaches PG, and SHOW returns `"""$user"" , public"`. Algo 3 is the more correct behaviour (PG would never see the destructive collapse) but diverges from algorithms 0/1/2's historical behaviour. Earlier this session I considered making the walker call remove_spaces() too just to bring algo 3 in line. That would have papered over the issue by propagating the pre-existing data-mangling bug into the new code path. Better: - Leave the walker honest -- it stays the more correct path. - Document the algo-0/1/2 quirk inline in PgSQL_Session.cpp at the parser-selection site so future readers know why the two paths don't match byte-for-byte. - Teach the test about the divergence: SetCommandTest gains an optional `expected_value_algo3` field, populated only for the affected case (#150). At test startup we query `pgsql-set_parser_algorithm` from proxysql admin once and pick the appropriate expected value per case. A real fix would be to make remove_spaces() string-literal-aware (or stop running it on SET commands altogether). That's a larger change with risk to other tests calibrated against today's behaviour, so it's deferred to its own ticket; this commit keeps the regression green on both algorithm groups (PASS 1/345, FAIL 0/345 locally for both pgsql-set_parameter_validation_test-t under set_parser_algorithm_3-g1 and legacy-g6).	1 month ago
Rene Cannao	35f84c384b	fix(pgsql-session): don't lock hostgroup on parse fail for tracked vars ... When the SET parser returns an empty map for a SET that targets a proxysql-tracked variable (search_path, datestyle, ...), the previous behaviour called unable_to_parse_set_statement(), which set *lock_hostgroup = true. From that point on, the entire SET-handling function was short-circuited for the rest of the session, so every subsequent SET -- including ones that DO parse and would have been caught by the per-variable validator -- bypassed proxysql entirely and went straight to PG. That cascade is what pgsql-set_parameter_validation_test-t case #184 hit: case #172's malformed `SET search_path = "unclosed_quote, public` caused the parser to (correctly) reject, locked the hostgroup, and from then on case #184's 64-char delimited identifier `SET search_path TO "1234...64chars..."` skipped the search_path validator's "> 63 chars -> reject" check and reached PG, which silently accepts/truncates such identifiers. The test then saw the SET succeed unexpectedly and reported the misleading "Value changed unexpectedly from '"$user"' to ''" message (test_ok flipped before the SHOW value was read; new_value defaults to "" in that branch). The fix: at this specific call site (inside the `match_regexes[1]->match(dig)` block, i.e. we already know the SET targets a tracked variable), drop the unable_to_parse_set_statement() call. Log a warning and return false so the SET is forwarded to PG normally. PG itself rejects truly malformed SETs (which was already the observed behaviour for cases #172-#183), and subsequent valid SETs on the same connection continue to flow through proxysql's validator. The locking branch at the outer `else` (when the regex didn't match -- i.e. genuinely unknown SET) is unchanged. End-to-end verification gates on CI -- there's no local PG infra in this build -- but the in-process unit test TestWalkerToValidatorChain184 (added in the previous commit) already confirmed that walker + validator together reject the 64-char input, which is exactly the path #184 now reaches with the lock removed. Diagnostic SET-VALIDATE proxy_info logging from the previous commit is removed.	1 month ago
Rene Cannao	950f6415b5	test+debug(set_param_validation #184): chain test + temp validator logging ... Investigation of pgsql-set_parameter_validation_test-t case #184 (64-char delimited identifier > 63-char PG limit) has been blocked by divergence between static analysis and observed CI behaviour: - Static analysis: walker emits 66-char `"<64 digits>"` string, the search_path validator's quoted-branch counts effective_len=64, triggers `> 63` reject, returns false. Session error path generates an error packet to the client. SET should fail. - Observed in CI: no `[ERROR] invalid value for parameter "search_path"` log line at the time of case #184, despite the same log capturing every other validator-reject (client_encoding, DateStyle, IntervalStyle, etc.). Test reports `actual_success=true` (SET unexpectedly succeeded). The `"Value changed unexpectedly from '"$user"' to ''"` message is the test framework's artefact when test_ok flips before the value-comparison read. To pin where the disconnect actually is: * setparser_parsersql_test.cpp adds an inline copy of pgsql_variable_validate_search_path plus 3 strict cases plus a walker -> session-join -> validator chain test exercising the exact 64-char input. All 5 new assertions pass locally, confirming walker + validator are individually correct (the validator does reject 64 chars; walker does emit 66-char output). Chain test produces what production should be feeding the validator. * PgSQL_Session.cpp adds two `proxy_info` lines around the validator call in the SET-tracking block (pre and post). These will print value1 + validator return to the CI proxysql.log so we can see what proxysql actually feeds the validator for #184 and what it gets back. To be reverted once #184 is fixed.	1 month ago
Rene Cannao	4eb52abf2c	fix(pgsql,test): PG multi-value SET dispatch + test-fixture cleanup ... Follow-up to the cluster-2 ParserSQL 1.0.3 bump (this same PR), addressing review feedback from coderabbit + gemini. ## 1. PgSQL_Session: accept multi-value SET vectors (coderabbit, Major) The previous commit `5a48004af` taught the adapter to walk every RHS sibling so PG multi-value lists (e.g. `SET search_path TO "$user", public`) come through as a `std::vector<std::string>` of every value. But `lib/PgSQL_Session.cpp:4516` still had `size() > 2` → `unable_to_parse_set_statement()`, which would have caused 3+ value PG SETs to lock the hostgroup instead of being processed. Worse, the downstream value-consumer at the same site used only `it->second.front()`, silently dropping every value past the first for valid 2-value SETs too. Fix: * Drop the upper bound from the cardinality check (keep `< 1` so empty value lists still fail loudly). * Join every value in the vector into `value1` with the comma-separated form PG itself uses for these parameters (`a, b, c`). Single-value SETs hit the loop with `size()==1` and take just `it->second.front()`, preserving the previous behaviour. The validator / hash-compare / tracker / param_status code below the join is unchanged — it now receives the full joined value instead of the truncated first element. ## 2. setparser_parsersql_test: drop INTERVAL test that locked in incomplete behaviour (subagent correctness review, Major) `parsersql_pgsql_time_zone` previously asserted that `SET TIME ZONE INTERVAL '7' HOUR` produces `timezone = "INTERVAL"`, which is the *current* observable output of ParserSQL — the expression parser does not yet consume the `'7' HOUR` modifier chain. Encoding that as expected output would flip the test red when ParserSQL is later improved to capture the full interval expression, conflating an improvement with a regression. Removed the case; left a comment in the fixture explaining the gap so a future contributor adds the case back together with the ExpressionParser fix. ## 3. Diag-output cleanup (gemini, cosmetic) * `%lu` → `%zu` for `size_t` (portability — `size_t` is not always `unsigned long` on 32-bit / Windows). * Extract the per-vector " | "-separated diag-string builder into a single `join_values_for_diag()` helper used by both `TestParse` (now also dumps full vector on mismatch, where it previously printed just `vals[0]`) and `TestParsePgsql`. ## Verification * `./setparser_parsersql_test`: 260/270 ok; the 10 remaining not-ok are unchanged pre-existing failures (`sql_mode 23-25`, `Set1_v2 0-1`), nothing new from this commit. * `pgsql_time_zone` group: 6 assertions (3 cases × 2) — confirms the INTERVAL case is removed cleanly. * `pgsql_search_path` group: 16 assertions still all pass — the multi-value join in PgSQL_Session does not affect the parser- side test (which goes through the adapter, not the session).	1 month ago
Rene Cannao	5b5b9373e3	feat(pgsql,mysql): gate ProcessAllSessions_Partition by pool-acquire NULL ratio ... ProcessAllSessions_Partition (Pass 1, the 3-way A/B/C session classification) is cheap and beneficial when block B (sessions waiting for a backend) is non-empty: it lets block A finish queries and release conns to block B waiters in the same outer iteration. When no sessions are contending for backends, partition is unnecessary work. Detect the contention state with a per-worker counter pair tracked at the get_MyConn_from_pool() call site: every call increments partition_pool_attempts; calls that returned NULL (couldn't acquire, session must wait) also increment partition_pool_nulls. At the top of each process_all_sessions outer iteration, the worker computes the NULL ratio and updates a hysteresis state machine: 3 consecutive iterations with ratio >= 5% flips partition_active to true; 3 consecutive iterations below the threshold flips it back. Mechanism: * Cheapest possible signal: 1 int increment per pool acquire, 1 branch per outer iteration. * Hysteresis (3 consecutive iterations either direction) filters transient bursts without needing min-sample / asymmetric-threshold tuning. * Single threshold 5% works in both directions by virtue of the streak filter -- the ratio has to be sustained near the boundary for flapping to occur, which doesn't match any realistic workload. Empirical verification at 1 worker thread, 500c clients, 4 KB rows, 50-conn backend pool, SSL on, 120 s: 20c (no contention): 1576 tps, 13 ms avg latency, gate stays OFF 100c (some): 1487 tps, 67 ms, gate stays ON 500c (heavy): 1487 tps, 336 ms, gate stays ON The gate transitions automatically; no admin variable to learn. Applied symmetrically to MySQL_Thread / MySQL_Session and PgSQL_Thread / PgSQL_Session. See #5791 for analysis.	1 month ago
Rene Cannao	c68f60f04b	merge: merge v3.0 into feature/parsersql-integration ... Resolve conflict in test/tap/tests/unit/Makefile: keep v3.0's refactored linking (WHOLE_LIBPROXYSQL to avoid double-link under ASAN) and add -lsqlparser for parsersql integration.	2 months ago
Rene Cannao	6927682d08	fix: add null checks before deleting server_myds in backend reset and ping handlers ... SonarCloud flagged potential double-free in mysql_backend.cpp (S3520). While the callers ensure single invocation, adding null checks and nullifying after delete is defensive. Applied same pattern to MySQL_Session and PgSQL_Session ping handlers for consistency.	2 months ago
René Cannaò	222a5d1ec8	Merge branch 'v3.0' into feature/parsersql-integration ... Signed-off-by: René Cannaò <rene@proxysql.com>	2 months ago
Rene Cannao	deb1c46173	fix: address code review findings C1-C3, I1, I3 ... C1: NODE_SET_CHARSET map key corrected to character_set_results C2: query_processor_parser==1 now also routes SET through ParserSQL C3: qp->digest initialized to 0 on parse failure I1: header docs corrected (tokenizer-based, not libinjection) I3: defensive NODE_SET_TRANSACTION case added to SET walker	2 months ago
Rene Cannao	dffa89de7a	fix(ci): repair non-PROXYSQL40 dbg builds + genai unit-test linking ... CI surfaced two separate build breakages that had been latent since the carve-out steps but only became visible once they ran on the right matrix variants. # lib/MySQL_Session.cpp + lib/PgSQL_Session.cpp: chassis dispatch # block needs PROXYSQL40 guard The query-hook dispatch I added in Step 2.2 references chassis types (`ProxySQL_PluginProtocol`, `ProxySQL_PluginQueryHookPayload`, `ProxySQL_PluginQueryHookResult`, etc.) without an `#ifdef PROXYSQL40` guard. Those types are only declared when ProxySQL_Plugin.h is compiled with `-DPROXYSQL40` — v3.0/v3.1 dbg builds (debian12,-dbg and ubuntu22,-tap on the CI matrix) define neither, so the lib step fails: MySQL_Session.cpp:4477:127: error: 'ProxySQL_PluginProtocol' has not been declared MySQL_Session.cpp:4477:85: error: 'proxysql_has_configured_plugin _query_hook' was not declared in this scope [+ 7 more identical errors in PgSQL_Session.cpp] Wrap both dispatch blocks (MySQL_Session.cpp:~4477-4509, PgSQL_Session.cpp:~2414-2446) in `#ifdef PROXYSQL40`. The block was only meaningful when the chassis is present anyway — the call to `proxysql_has_configured_plugin_query_hook` short-circuits to false when no plugin is loaded, but the SYMBOL doesn't exist at all in non-chassis builds. # test/tap/tests/unit/Makefile: PROXYSQLGENAI autodetect probe was # defining the macro without a way to satisfy the resulting linker # requirements Pre-Step-7 the autodetect probed for `MCP_Tool_Handler` in libproxysql.a; this symbol is plugin-side now, so the probe never matches and PROXYSQLGENAI stays empty. My previous attempt to "fix" the autodetect (probing `invoke_register_schemas_phase`) made it match again — but that defines PROXYSQLGENAI for the WHOLE unit- test build, including tests like `genai_llm_clients_unit-t` and `genai_mcp_endpoint_unit-t` that pull in plugin headers AND instantiate plugin classes. Those classes' definitions live in plugin .cpp files that the unit-test build does NOT link, so the result was: genai_llm_clients_unit-t.cpp:201: undefined reference to `LLM_WriteCallback(void*, unsigned long, ...)' [+5 more] Solution: never auto-define PROXYSQLGENAI for the unit-test build. Tests that legitimately exercise plugin internals (`genai_plugin_anomaly_unit-t`, `genai_plugin_backend_client_unit-t`, `genai_plugin_load_unit-t`, `genai_fts_string_unit-t`) declare a new GENAI_PLUGIN_DEFINES variable on their dedicated rule and add the specific plugin .cpp files they need to the link line. Every other genai_*_unit-t test guards its body in `#ifdef PROXYSQLGENAI ... #else SKIP #endif`, so falling through to the SKIP path keeps CI green. Coverage for those tests in non-genai builds was already zero — the autodetect was never activating PROXYSQLGENAI for them in practice (the probed symbol moved out in Step 6). This commit documents that reality instead of pretending to maintain coverage that didn't exist. # test/tap/tests/unit/genai_fts_string_unit-t.cpp: stub for # MySQL_Tool_Handler::execute_query The new dedicated rule for genai_fts_string_unit-t compiles plugins/genai/src/MySQL_FTS.cpp directly into the test binary. MySQL_FTS::index_table calls MySQL_Tool_Handler::execute_query — but this test only exercises the pure string helpers (sanitize_name / escape_*) and never reaches index_table, so the call site is unreachable at runtime. The linker still needs a body, so we provide an empty stub directly in the test .cpp. The real implementation lives in MySQL_Tool_Handler.cpp which we can't link without dragging in the full plugin runtime stack.	2 months ago
Rene Cannao	1c84a376fe	feat(session): Step 2.2 — call plugin query hook on the COM_QUERY hot path ... Wires the Step 2.1 plugin query-hook ABI into the actual session hot paths. Both MySQL_Session and PgSQL_Session now consult any plugin- registered query hook just before the query dispatches to a backend, and short-circuit with an error to the client if the hook returns DENY. Hot-path discipline: - Gate is proxysql_has_configured_plugin_query_hook(proto), a lock-free atomic load + one pointer-sized read. When no plugin has registered a hook for the protocol (the overwhelmingly common case), the only cost is that single load + branch. - The dispatch helper takes the plugin-manager mutex; that path is only entered after the gate says "maybe". The gate is documented as spurious-true-tolerant: a concurrent unload can null the manager between the gate and the dispatch, in which case the dispatch helper re-checks under the lock and returns false. - Payload assembly uses the pointers already on hand in the session (client_myds->{addr.addr, myconn->userinfo->{username,schemaname}}). No allocation, no copy. Placement: - MySQL_Session.cpp: same block as the existing PROXYSQLGENAI in-core anomaly detector check. The two coexist during the carve-out transition; the in-core detector goes away in Step 3 once Anomaly_Detector lives inside the plugin and consumes the hook. - PgSQL_Session.cpp: parallel block right after the existing SQLi detector check, the equivalent location for pgsql. Error response on DENY: - MySQL: ER 1313 / SQLSTATE HY000 with the plugin's message (or "Query blocked by plugin" if the plugin returned an empty message), via generate_pkt_ERR + RequestEnd, matching the existing in-core anomaly-detector blocking style. - PgSQL: ERRCODE_INSUFFICIENT_PRIVILEGE via generate_error_packet + RequestEnd. End-to-end smoke verified: with the genai skeleton plugin loaded (no hook registered yet), an admin SELECT 1+1 returns the right answer -- the no-hook fast path is invisible. All 59 unit-test binaries pass. Note: hot-path tests with a real registered hook live in plugin_query_hook_unit-t (Step 2.1). An end-to-end TAP test that exercises the call site through a real backend is deferred to the group-tests work; not in scope for Step 2.	2 months ago
Rene Cannao	146a977c26	feat: wire ParserSQL adapter into query processor and sessions	2 months ago
Rene Cannao	6d8dff2939	docs(chassis): mark query-hook ABI scaffold-state explicit + add TODO markers ... The chassis ABI 2 query-hook surface (ProxySQL_PluginQueryHookPayload / Result / Action plus register_query_hook / dispatch_configured_plugin_ query_hook) is wired through ProxySQL_PluginManager and exercised end- to-end by unit tests. But the production data plane never calls proxysql_dispatch_configured_plugin_query_hook — neither MySQL_Session nor PgSQL_Session has the integration point. Net effect today: a plugin can register a query hook successfully and unit tests can drive dispatch through it, but a real client query arriving over MySQL or PgSQL will never consult the hook. The DENY contract from the public ABI ("DENY prevents a query from dispatching") is currently a promise the production path doesn't keep. # Why scaffold-state instead of immediate full integration Wiring the dispatch into MySQL_Session::handler___status_WAITING_ CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY_qpo (and the COM_STMT_ PREPARE / COM_STMT_EXECUTE entries that share the qpo machinery) is mechanical but touches a 95k-LOC hot-path file. A bug in the integration would manifest as a query-blocking regression that's hard to debug and easy to ship. Same for PgSQL_Session. The integration deserves its own focused PR with TAP coverage that exercises both ALLOW and DENY paths against a fake plugin — not a side-quest in this PR which is already large. # What this commit does 1. Updates the ABI doc block in include/ProxySQL_Plugin.h to call out the scaffold state explicitly. A plugin author reading the header today would reasonably assume the hook fires in production; that assumption is wrong, and the doc block now says so. References the precise grep target ("TODO(plugin-query-hook)") that points at the missing injection sites. 2. Adds two TODO(plugin-query-hook) markers to MySQL_Session.cpp: - line ~3387 (COM_STMT_PREPARE → COM_QUERY shared codepath) - line ~5403 (the main COM_QUERY execution path) Each immediately follows the GloMyQPro->process_query call, where CurrentQuery is populated and the dispatch payload would be ready. 3. Adds one TODO(plugin-query-hook) marker to PgSQL_Session.cpp at line ~2387, following the GloPgQPro->process_query call. # What this commit deliberately does NOT do The actual dispatch wiring. The next person picking this up has: - A precise injection site (3 in MySQL, 1+ in PgSQL). - Pre-built ABI surface (proxysql_has_configured_plugin_query_hook for the lock-free fast-path probe; proxysql_dispatch_configured_ plugin_query_hook for the actual hook invocation under shared lock). - Existing test_helpers/fake_plugin.cpp with a HOOK_DENY env toggle that exercises the dispatch path under unit test today — extensible to TAP integration tests once the production wiring lands. Caught by an external review pass, finding #5. Filing as a deferred fix with explicit scaffold acknowledgement rather than rushing the integration.	2 months ago
Rene Cannao	ae88c07089	Fix GTID range validation	2 months ago
Rene Cannao	f5e8732afb	fix(pgsql): review-driven fixes for tx_poisoned feature ... Consolidated feedback from four independent reviews (correctness, security, test-coverage, code-quality). Changes, in four logical groups: 1. Extend coverage to clean-ErrorResponse backend-death path (correctness review #2). The most common real-world mid-tx backend-death scenarios — graceful postgres shutdown (57P01), crash shutdown (57P02), connect-now-refused (57P03) — deliver a clean ErrorResponse on an otherwise-reusable connection, so they reach handler_minus1_HandleErrorCodes, not handler_minus1_ClientLibraryError. Pre-fix, poison only fired on the connection-broken path. Now it fires on both. The caller at PgSQL_Session.cpp:~3222 grew a tx_poisoned check that mirrors the one already added at :3214 for the reusable-state=false path. 2. Tighten the poison preflight to true in-tx only (security F1). Previously gated on myconn->IsActiveTransaction(), which returns true via the disconnect-heuristic set by 68c76eb42's compute_unknown_transaction_status — even for a plain autocommit statement that happened to be killed mid-query. Now gates on is_in_transaction() via transaction_state_manager, which tracks explicit BEGIN/COMMIT/ROLLBACK. A non-tx statement killed mid-flight no longer gets a misleading "current transaction is aborted" synthesized error. 3. Harden the simple-query classifier (correctness #3/#4/#5, test review 1.11). - Trim the trailing null terminator from the Simple Query payload before applying the "no trailing content" check, otherwise valid statements like "ROLLBACK\0" / "ROLLBACK WORK\0" fall through to KIND_REJECT. - Skip SQL line comments (-- to EOL) and block comments (/* */ non-nesting) in leading whitespace, so ORM-prepended comments don't block recovery. - After matching ROLLBACK/ABORT/COMMIT/END, require the rest of the statement to be a plain whole-tx end (optional WORK/ TRANSACTION noise word + optional trailing ';'). Reject: * ROLLBACK TO SAVEPOINT (partial rollback — can't honor on a dead backend) * ROLLBACK AND CHAIN (chained tx — same) * ROLLBACK; SELECT 1; (multi-statement — we'd silently drop the second stmt) Rejection is ERROR 25P02 + stay poisoned. 4. Code-quality refactor (simplify review #1, correctness #7/#8, security F2/F3). - Centralize the ErrorResponse(25P02) + ReadyForQuery('E') packet construction via write_tx_poisoned_error(). Four sites collapse to one-liners; message string lives in one place. - Fix NoticeResponse SQLSTATEs: WARNING-class notices now use ERRCODE_WARNING (01000, generic) for the backend- context notice and ERRCODE_NO_ACTIVE_SQL_TRANSACTION (25P01) for the "there is no transaction in progress" notice on the COMMIT-on-poisoned path. Previously both used ERRCODE_SUCCESSFUL_COMPLETION (00000) which is semantically wrong for a WARNING. - Cap backend error text to 4 KB and strip embedded NUL bytes before re-embedding in a NoticeResponse. Prevents memory-amplification if a compromised/rogue backend sends multi-MB error text, and prevents silent truncation at the first NUL (PG_pkt::put_string -> strlen). - Enrich the poison-event proxy_warning with hostgroup, backend addr:port, backend PID, client addr, user, dbname, query digest, and the full backend error code+message. Mirrors handler_minus1_LogErrorDuringQuery so operators can correlate poison events with the triggering query. Related to PR #5659 and issue #5658.	2 months ago
Rene Cannao	100e417179	test(pgsql): extended-query coverage for tx_poisoned behavior ... New sibling to pgsql-tx_poisoned_recovery-t, plus a one-line semantics fix in the extended-query poison gate. Semantics fix (lib/PgSQL_Session.cpp): Before, the pgsql_tx_poisoned_rejected_statements_total counter was incremented on EVERY swallowed ext-query wire packet — one per Parse / Bind / Describe / Close / Execute / Sync. That made a single logical PQexecParams call count as 4+, inconsistent with the simple-query path where a 'Q' packet bumps it exactly once. Now P/B/D/C/E are silently swallowed (no counter bump), and Sync is the single accounting point — the counter value matches the number of client-visible "rejected statements" regardless of protocol. New test pgsql-tx_poisoned_extended_query-t (plan(11)): * Poison the session via a simple-query pg_sleep + mid-tx kill. * While poisoned: - PQexecParams("SELECT 42", 0 params) returns 25P02. - PQprepare("stmt_poisoned", "SELECT 43") returns 25P02. - PQexecPrepared on that name returns 25P02. * The rejected counter increments by exactly 3 (one per logical ext-query call), not by the 8-11 wire packets that libpq actually sends. This assertion IS the mechanical proof that the semantics fix above took effect. * Simple-query ROLLBACK recovers the session. * Post-recovery PQexecParams and PQprepare+PQexecPrepared both execute cleanly and return rows. Registered alongside the simple-query sibling in legacy-g2. Verified against a live legacy-g2 infra: both tests pass (11/11 and 15/15), counter deltas exactly match expected per-statement semantics.	2 months ago
Rene Cannao	fb690e7095	feat(pgsql): preserve client session on mid-tx backend death ... Previously, when a PgSQL backend died mid-transaction, ProxySQL's handler_minus1_ClientLibraryError refused to silently retry the failed statement on a fresh backend (correct, per 68c76eb42) but then also tore down the client session by returning handler_ret=-1. The application had to reconnect before it could make forward progress, even though Postgres's wire protocol has enough granularity to let us signal the client "your transaction is aborted, send ROLLBACK to recover" while keeping the socket open. This change introduces that alternative path, gated behind a new admin variable pgsql-preserve_client_on_broken_backend_in_tx (default true). When the backend dies mid-transaction with the admin var on and the result-set transfer to the client has not yet started, ProxySQL now synthesizes on the client OUT queue: * ErrorResponse severity=ERROR sqlstate=25P02 "current transaction is aborted, commands ignored until end of transaction block" * NoticeResponse carrying the backend's original error-message text (no leak of the original 57P01 sqlstate — per design, the only client-visible sqlstate is the synthesized 25P02) * ReadyForQuery('E') — libpq now reports PQTRANS_INERROR. The session is marked tx_poisoned. handler_special_queries gains an earliest gate that dispatches to handler_poisoned_simple_query before query rules / digests / routing: * ROLLBACK / ROLLBACK TO SAVEPOINT / ABORT -> synthesize CommandComplete('ROLLBACK') + ReadyForQuery('I'), clear the flag. * COMMIT / END -> match Postgres native: emit NoticeResponse with "there is no transaction in progress" then the same synthesized ROLLBACK, clear the flag. * RELEASE SAVEPOINT -> ERROR 25P02 (matches Postgres native). * Anything else -> ERROR 25P02, stay poisoned. Extended Query (Parse/Bind/Describe/Close/Execute/Sync) while poisoned is rejected wholesale in V1: P/B/D/C/E are swallowed, Sync emits ErrorResponse + ReadyForQuery('E'). The client can always recover via a Simple-Query ROLLBACK. Extended-query recovery is deliberate future work (documented in the accompanying issue). The fallback paths stay intact. With the admin var off, or when the result-set transfer has already started to the client, the session is terminated as before — preserving pre-feature behavior for operators who rely on session termination as an app-layer signal. Three counters expose the feature's activity on stats_pgsql_global: pgsql_tx_poisoned_total pgsql_tx_poisoned_recovered_total pgsql_tx_poisoned_rejected_statements_total Each PgSQL thread maintains its own lock-free copy; PgSQL_Threads_Handler aggregates across threads when the admin table is rendered. Prometheus wiring for these counters is deliberately not added here — the existing PgSQL Prometheus pipeline (p_gauge_array) is incomplete even for active_transactions. Follow-up work. tx_poisoned resets on PgSQL_Session::reset(). No retry-loop interacts with the flag; session handoff to a fresh backend only happens after the client itself sends ROLLBACK/COMMIT/ABORT to clear it. Related: issue #5658.	2 months ago
Rahim Kanji	7aaf126a07	Skip RE2 for non-COPY queries via strcasestr pre-check ... Use strcasestr on the query digest to detect "COPY " before calling CopyCmdMatcher::match(). Non-COPY queries short-circuit without running the regex. Falls back to RE2 when digest is unavailable. Refs #5568	3 months ago
Rahim Kanji	ca45405942	Merge remote-tracking branch 'Master/v3.0-moveInfra' into v3.0_pgsql-pipeline-set-reset-discard-fix-5432	4 months ago
Rahim Kanji	000a9794e4	Don't short-circuit in pipeline mode, fall through to backend execution	4 months ago
Rahim Kanji	abb66cc19f	Addressed AI review feeback	4 months ago
Rene Cannao	6b46b26931	chore: implement PR review feedback for release tier system ... - Makefile: normalized GIT_VERSION by stripping 'v' prefix before arithmetic. - Makefile: corrected .PHONY declaration for build_deps_debug_default. - Admin Handler: properly guarded TSDB commands to return error when disabled. - REST API: removed redundant nested guards inside tsdb_resource class. - Statistics: implemented rigorous input validation for set_variable. - Statistics: added return value checks and rollback logic for TSDB migrations. - Statistics: implemented bounded concurrency (batching) for backend probes. - FFTO: implemented cumulative buffer limit enforcement in MySQL and PgSQL. - FFTO: ensured initial packet capture in PgSQL_Session by early initialization. - FFTO: improved reassembly buffer compaction logic. - Headers: default-initialized ffto_bypassed to false. - Code cleanup: merged redundant PROXYSQLFFTO guards.	4 months ago
Rene Cannao	c2f678a9de	feat: implement release tiers and feature guards for v3.0, v3.1, and v4.0 ... This commit introduces a structured release tier system controlled by environment variables, allowing three distinct ProxySQL versions to be built from the same codebase while maintaining a clear upgrade path. Key Changes: - Tier 1 (v3.0.x Stable): Default build including bug fixes and core enhancements. - Tier 2 (v3.1.x Innovative): Enabled via PROXYSQL31=1. Adds FFTO and TSDB. - Tier 3 (v4.0.x AI/MCP): Enabled via PROXYSQLGENAI=1. Adds the AI stack. Implementation Details: - Makefile: Implemented dynamic versioning logic that increments the minor version for v3.1 and the major version for v4.0. Added tier documentation. - Feature Guards: Introduced PROXYSQLFFTO and PROXYSQLTSDB guards across ProxySQL_Admin, ProxySQL_Statistics, REST API, and HTTP server. - FFTO: Guarded Fast Forward Traffic Observer in MySQL and PgSQL sessions, including thread-local variables and configuration parameters. - TSDB: Guarded the entire Time Series Database subsystem, including SQLite table definitions, background loops, and REST API endpoints. - Build Fixes: * Fixed 'proxy_debug_func' linkage by adding extern "C" and moving the definition outside of DEBUG guards to ensure non-debug builds link correctly. * Corrected misplaced/nested DEBUG and PROXYSQLFFTO guards. * Verified build stability for all tiers (unset, PROXYSQL31=1, PROXYSQLGENAI=1).	4 months ago
Rahim Kanji	e84433159b	Merge remote-tracking branch 'v3.0' into v3.0_pgsql-pipeline-set-reset-discard-fix-5432	4 months ago
Rahim Kanji	96bc668735	Fix pipeline mode transaction state handling and SET/RESET/DISCARD support ... Addresses issues with PostgreSQL Extended Query Protocol (Pipeline mode) in ProxySQL: Transaction State Handler Migration: - Moved handle_transaction_state() from PgSQL_Session::RequestEnd() to PgSQL_Session::handler() - Now called only for client queries (PROCESSING_QUERY/PROCESSING_STMT_EXECUTE) - Prevents internal SET commands from triggering transaction logic incorrectly - Added transaction state reset at 7 locations for clean connection pool return SET/RESET/DISCARD Pipeline Mode Support (lib/PgSQL_Session.cpp): - SET statements now execute properly in pipeline mode via RunQuery() - RESET ALL and DISCARD ALL blocked with proper error messages in pipeline mode - Added startup parameter mismatch detection for RESET with locked hostgroup	4 months ago
Rahim Kanji	742ff05ecc	Fix PostgreSQL transaction state management for pipeline mode ... Move transaction state tracking from RequestEnd to connection handler to enable proper binary pipeline mode support: - Call handle_transaction_state() at ASYNC_QUERY_END/ASYNC_STMT_EXECUTE_END in PgSQL_Connection instead of RequestEnd - Add reset_state() calls in 7 locations to prevent stale state: * Session reset * Connection detach (finishQuery) * Error handlers (4 paths) * Base session housekeeping - Refactor destructor to use reset_state() Fixes transaction state tracking in pipeline mode where connections can be returned to pool mid-transaction.	4 months ago
Rahim Kanji	4d86e6949c	Add defensive null guard before reading PSarrayIN->len	4 months ago
Rahim Kanji	8c9af56f38	Address AI review feedback	4 months ago
Rahim Kanji	05b5821fbc	Fix integer underflow in PostgreSQL query packet handling and null terminator checking ... * Add validation to reject packets smaller than 6 bytes before processing * Add null terminator check	4 months ago
Rahim Kanji	866c3593bb	Set query retries for PostgreSQL extended query protocol operations ... Add query_retries_on_failure configuration for PARSE, DESCRIBE, and EXECUTE operations in PostgreSQL extended query protocol.	4 months ago
Rahim Kanji	27119169e4	Reset extended query frame if an unsupported message type is received	4 months ago
Rahim Kanji	176362c77d	Addressed feedback from AI review	4 months ago
Rahim Kanji	a82d653c0b	Handle late COPY protocol messages after session exits fast_forward mode ... When a COPY FROM STDIN operation encounters an error, the session switches back to normal mode. However, the client may have already pipelined CopyData('d'), CopyDone('c'), or CopyFail('f') messages that are still in the input queue. Previously, these messages fell through to the default case, generating a spurious "Feature not supported" error. This change adds explicit handling to discard these messages when session_fast_forward == SESSION_FORWARD_TYPE_NONE, preventing the race condition from causing errors. The client does not expect a response for these messages in this scenario.	4 months ago
Rene Cannao	e53d2c76f9	Address PR #5410 review findings across FFTO, sessions, TAP, and docs ... Apply actionable bot review feedback from PR #5410 without behavior drift:\n- fix MySQL/PgSQL FFTO digest normalization and max-length hashing\n- add early FFTO packet/message size guards for bypass safety\n- restore MySQL session GTID reset and harden GenAI async error ownership\n- add null-check for stripped KILL query normalization path\n- call FFTO on_close() before observer reset/bypass in session flows\n- fix processlist query-options default pagination limit semantics\n- fix TAP noise_failures locking race and noise-aware plan count\n- extend wait_get_enpoint_ready with optional basic-auth passthrough\n- update MCP architecture docs from observe->stats naming\n- clean corrupted character in pgsql query-cache test comment\n\nValidation run:\n- git diff --check\n- make -C test/tap/tap tap.o utils_mariadb.o utils_mysql57.o utils_mysql8.o -j4\n- make -C lib obj/MySQLFFTO.oo obj/PgSQLFFTO.oo obj/MySQL_Session.oo obj/PgSQL_Session.oo obj/Query_Processor.oo -j4	4 months ago
Rene Cannao	1258a1f2f2	Address PR reviews: improve performance, safety, and robustness of FFTO	4 months ago
Rahim Kanji	194e20b3e8	Revert "Reset current_hostgroup to default when no query rule matches" ... This reverts commit 1b3d20388e.	4 months ago
Rene Cannao	3a3be5a559	Implement Fast Forward Traffic Observer (FFTO) for MySQL and PostgreSQL ... - Added TrafficObserver base interface - Implemented MySQLFFTO and PgSQLFFTO classes for protocol-aware observation - Integrated FFTO hooks into MySQL and PostgreSQL session handlers in FAST_FORWARD state - Added configuration variables: mysql-ffto_enabled, mysql-ffto_max_buffer_size, pgsql-ffto_enabled, pgsql-ffto_max_buffer_size - Implemented query digest extraction and reporting parity with internal stats - Added session-level bypass when packet size exceeds buffer threshold - Included architectural documentation in doc/ffto_design.md	4 months ago
Rahim Kanji	1b3d20388e	Reset current_hostgroup to default when no query rule matches ... In the extended query protocol flow (Parse/Bind/Execute/Sync messages), queries without matching query rules were incorrectly retaining the current_hostgroup value from the previous query instead of being reset to the default_hostgroup. This happened because: - Simple query protocol goes through STATE_SLEEP between queries, which resets current_hostgroup to default_hostgroup - Extended query protocol does NOT go through STATE_SLEEP between Parse/Bind/Execute messages, so current_hostgroup was never reset	4 months ago
Rene Cannao	af58865b06	Fix uninitialized memory in QueryParserArgs.buf ... Use memset to zero-initialize the entire QueryParserArgs struct instead of individually setting only digest_text and first_comment to NULL. This fixes Valgrind errors about use of uninitialised values in strcasestr() when reading from the uninitialized buf array. Fixes errors like: Use of uninitialised value of size 8 at tolower (ctype.c:46) by strcasestr by MySQL_Connection::ProcessQueryAndSetStatusFlags(char*)	4 months ago
René Cannaò	2637d28f36	Merge pull request #5299 from sysown/v3.0_pg-cancel-terminate-backend-param-support_5298 ... Add parameterized PID support for pg_cancel_backend/pg_terminate_backend in extended query protocol	5 months ago
Rahim Kanji	9ec045ca74	Fix PostgreSQL deadlock with Close Statement flood exceeding threshold_resultset_size ... Bug Description: ProxySQL would deadlock when processing extended query frames where: 1. Many Close Statement messages accumulate responses in PSarrayOUT 2. Total response size exceeds pgsql-threshold_resultset_size 3. A backend operation (Describe/Execute) follows in the same frame Root Cause: - Close Statement operations are handled locally by ProxySQL (no backend routing) - Their CloseComplete responses accumulate in PSarrayOUT - When threshold_resultset_size is exceeded, ProxySQL stops reading from backend - Subsequent backend operations (Describe/Execute) need backend responses to complete - This creates a deadlock: ProxySQL won't read, backend operation can't complete - Extended query frame never finishes, query times out The Fix: When PSarrayOUT exceeds threshold_resultset_size and a backend operation is pending, ProxySQL now flushes all accumulated data in PSarrayOUT to the client first, then continues processing backend operations. This breaks the deadlock by clearing the buffer before attempting to read more data from the backend.	5 months ago
Rahim Kanji	67cbe46450	Simplify PID extraction ... Using bind message to obtain parameter information, rather than determining whether the query is parameterized from the query itself. Multiple parameters are not possible in this case, as PostgreSQL itself rejects multi-parameter pg_cancel_backend() and pg_terminate_backend() and only accepts a single parameter for these functions.	5 months ago
Rahim Kanji	5066ddd181	Removed isdigit	5 months ago
Rahim Kanji	ce42c188f5	Improvements	5 months ago
Rahim Kanji	a1e10e3055	Add parameterized PID support for pg_cancel_backend/pg_terminate_backend ... This commit extends the existing pg_cancel_backend() and pg_terminate_backend() support to work with parameterized queries in the extended query protocol. While literal PID values were already supported in both simple and extended query protocols, this enhancement adds support for parameterized queries like SELECT pg_cancel_backend($1).	5 months ago
Rahim Kanji	556b1023c4	Removed change_user_auth_switch flag	6 months ago
René Cannaò	0f7ff1f374	Merge branch 'v3.0' into v3.0_pgsql-query-digest-gen-5253 ... Signed-off-by: René Cannaò <rene@proxysql.com>	6 months ago
Rahim Kanji	42864e8867	Improved Tokenizer for PostgreSQL ... - Added `process_pg_typecast()` to handle PostgreSQL type cast syntax (::) - Recognizes type casts in various contexts: 'value'::type, column::type, etc. - Added `process_array_literal()` for PostgreSQL array processing - Handles both ARRAY[] constructor and {} literal syntax - Processes multi-dimensional arrays and nested array structures - Added `process_literal_prefix_type()` for PostgreSQL prefixed literals - Processes E'' escape string constants with backslash escapes - Handles U&'' Unicode string literals with optional UESCAPE clauses - Supports x'' hex string literals and b'' bit string literals - Manages B'' bit strings and bytea literals (\\xDEADBEEF format) - Added `process_replace_boolean()` for boolean literal replacement - Replaces TRUE and FALSE literals with parameter placeholders - Maintains case-insensitive matching (true, True, TRUE, etc.) - Preserves boolean context in expressions and WHERE clauses	6 months ago