790 Commits (19be159ffbf5b56d8ea52eb13bb679439d90f198)

Author	SHA1	Message	Date
René Cannaò	3e7b77456e	Merge branch 'v3.0' into v3.0_partition-gate	1 month ago
Rene Cannao	031f6fa1f5	fix(build): adapt callers of dns_cache counters to std::atomic ... The previous commit promoted MySQL_Monitor::dns_cache_{queried, lookup_success,record_updated} to std::atomic<unsigned long long> but missed several call sites that pass them through implicit copy: * lib/MySQL_Thread.cpp sprintf("%llu", GloMyMon->dns_cache_*) — varargs cannot copy std::atomic. * lib/PgSQL_Thread.cpp sprintf("%llu", GloPgMon->dns_cache_*) — same, for the PgSQL counters. * lib/MySQL_Monitor.cpp p_update_counter(..., GloMyMon->dns_cache_*) — works via implicit operator T(), but explicit .load() is clearer and matches the rest of the file. Promote PgSQL_Monitor::dns_cache_* to std::atomic for the same race that motivated the MySQL side (resolver workers increment, stats readers and p_update_metrics read from another thread), and add .load() at every read site so we don't depend on implicit conversion. Verified with a local 'make -j32 debug' build in the integration worktree.	1 month ago
Rahim Kanji	878ba3b1e6	refactor(partition-gate): consolidate into Base_Thread; drain every iteration; widen ratio to uint64_t ... Three closely related changes: * Move the four partition_* members and note_pool_attempt() from MySQL_Thread / PgSQL_Thread into Base_Thread. Eliminates 22 lines of duplicated gate logic across the two protocol implementations. * Run update_partition_gate() once per process_all_sessions outer iteration, before the sess_sort / len > 3 / partition_active check. Per-tick counters can no longer accumulate stale across ticks where partition is skipped (sess_sort off, IDLE_THREADS maintenance thread, or session count below threshold), which in 5b5b937 could cause spurious gate flips after sess_sort toggled back on. * Widen the ratio comparison to uint64_t so the multiplications cannot overflow unsigned int (reachable in combination with the staleness issue above). Function-local HI_NUM/HI_DEN/STREAK locals are promoted to named static constexpr class members (PARTITION_GATE_NULL_RATIO_NUM/DEN/STREAK). No new behavior introduced -- purely correctness fixes plus a behavior-preserving refactor.	1 month ago
Rahim Kanji	dcd143d39e	revert: drop unrelated push_MyConn_local debug-leftover split ... The two-line split of `MySrvC *mysrvc=(MySrvC *)c->parent;` into a NULL-init plus assignment landed in 5b5b937 as an unrelated debug- session artifact. Reverting to the pre-5b5b937 single-line form.	1 month ago
Rene Cannao	5b5b9373e3	feat(pgsql,mysql): gate ProcessAllSessions_Partition by pool-acquire NULL ratio ... ProcessAllSessions_Partition (Pass 1, the 3-way A/B/C session classification) is cheap and beneficial when block B (sessions waiting for a backend) is non-empty: it lets block A finish queries and release conns to block B waiters in the same outer iteration. When no sessions are contending for backends, partition is unnecessary work. Detect the contention state with a per-worker counter pair tracked at the get_MyConn_from_pool() call site: every call increments partition_pool_attempts; calls that returned NULL (couldn't acquire, session must wait) also increment partition_pool_nulls. At the top of each process_all_sessions outer iteration, the worker computes the NULL ratio and updates a hysteresis state machine: 3 consecutive iterations with ratio >= 5% flips partition_active to true; 3 consecutive iterations below the threshold flips it back. Mechanism: * Cheapest possible signal: 1 int increment per pool acquire, 1 branch per outer iteration. * Hysteresis (3 consecutive iterations either direction) filters transient bursts without needing min-sample / asymmetric-threshold tuning. * Single threshold 5% works in both directions by virtue of the streak filter -- the ratio has to be sustained near the boundary for flapping to occur, which doesn't match any realistic workload. Empirical verification at 1 worker thread, 500c clients, 4 KB rows, 50-conn backend pool, SSL on, 120 s: 20c (no contention): 1576 tps, 13 ms avg latency, gate stays OFF 100c (some): 1487 tps, 67 ms, gate stays ON 500c (heavy): 1487 tps, 336 ms, gate stays ON The gate transitions automatically; no admin variable to learn. Applied symmetrically to MySQL_Thread / MySQL_Session and PgSQL_Thread / PgSQL_Session. See #5791 for analysis.	1 month ago
Rene Cannao	e5beb0d65a	fix(pgsql,mysql): bounded local connection cache 1-in-N (#5791) ... The per-thread cached_connections cache in PgSQL_Thread/MySQL_Thread greedily kept every released backend conn local to the thread that released it. At high client count this self-reinforced into one thread hoarding all backend conns while sibling worker threads starved on the shared pool, manifesting as a throughput cliff at ~200 clients with 2 worker threads and a 50-conn backend pool. Change push_MyConn_local() to cache only 1 release in every N (N = mysql_threads / pgsql_threads); push the rest straight to the shared HGM pool. At N=1 (single worker) the behaviour is unchanged. At N>1 the cache still amortizes the bulk of the HGM wrlock cost while ensuring released conns flow to peer threads regularly enough that the pool is never drained on one side. See #5791 for the empirical analysis. Mechanism eliminates the 2-thread cliff at c=200 and recovers ~80% throughput at c=200, ~50% at c=1000 vs the always-cache baseline.	1 month ago
Rahim Kanji	ca202ceba1	Refactor: partition sessions by backend state instead of sorting ... Replace ProcessAllSessions_SortingSessions with ProcessAllSessions_Partition. The old function did an implicit one-pass swap-sort over mysql_sessions by max_connect_time; the new one explicitly partitions sessions into three contiguous blocks in pdata according to their backend state: [0, running_end) A: running a query against the backend [running_end, idle_begin) B: acquiring a backend (max_connect_time != 0) [idle_begin, len) C: idle (no backend, or holding a conn while parked in WAITING_CLIENT_DATA)	2 months ago
Rene Cannao	0f96ae1fe7	fix: resolve all 12 SonarCloud issues (S3776, S134, S1231, M23_090) ... - Extract skip_quoted_char() from extract_paren_expr() to reduce cognitive complexity (S3776, S134) in Query_Processor_ParserSQL.cpp - Extract resolve_var_value() and finalize_var_value() from walk_set_stmt() to reduce complexity from 39 to under 25 - Extract qp_digest_parsersql() and qp_digest_legacy() from query_parser_init() to reduce complexity from 30 to under 25 - Add NOSONAR annotations for C-style casts on string literals that follow existing codebase patterns (M23_090) - Add NOSONAR annotations for free() calls on C-allocated memory in test code (S1231)	2 months ago
René Cannaò	222a5d1ec8	Merge branch 'v3.0' into feature/parsersql-integration ... Signed-off-by: René Cannaò <rene@proxysql.com>	2 months ago
Rene Cannao	62ed0660a7	feat(plugins/genai): Step 7 — drop PROXYSQLGENAI from core build ... Last step of the GenAI plugin carve-out. The PROXYSQLGENAI macro is no longer compiled into core: lib/libproxysql.a and the proxysql binary build identically with or without `PROXYSQLGENAI=1`. The user-facing build flag stays — it now means "also build the genai plugin alongside proxysql" rather than "compile GenAI into the binary". What this commit does plugins/genai/Makefile Hardcodes -DPROXYSQLGENAI in the plugin's CXXFLAGS — the plugin IS the GenAI surface, so the macro is always-on inside its own compilation unit. Lets the existing `#ifdef PROXYSQLGENAI` guards inside plugin source files keep working without a massive guard-stripping pass (deferred follow-up cleanup). lib/Makefile, src/Makefile Drop the `PSQLGA := -DPROXYSQLGENAI` setter — core no longer sees the macro on the compile line. PSQLGA stays as an empty variable so the existing `$(PSQLGA)` cascades in CXXFLAGS references stay structurally identical (cleanup deferred). Also: lib/Makefile no longer compiles MCP_Tool_Handler.oo (the base class moved to the plugin in this commit too — see below). Top-level Makefile Plugin-build rules switched from `$(if $(filter 1,$(PROXYSQL40))` to `$(if $(filter 1,$(PROXYSQLGENAI))` — `PROXYSQL40=1` enables the chassis but not necessarily the genai plugin. Skip-message text updated to match. `PROXYSQLGENAI=1` still implies `PROXYSQL40=1` per the existing tier hierarchy, so the user invocation stays unchanged. MCP_Tool_Handler base class Moved to plugins/genai/ (`include/MCP_Tool_Handler.h` -> plugin include, `lib/MCP_Tool_Handler.cpp` -> plugin tool_handlers). Was the last GenAI-tier .cpp in core — only existed there because AI/RAG_Tool_Handler still inherited from it before Step 5. Now plugin-internal. lib/proxy_sqlite3_symbols.cpp Hook for sqlite_vec_init unconditional now (sqlite-vec extension object always linked into proxysql, since the genai plugin relies on it via this hook at dlopen time). lib/MySQL_Thread.cpp / lib/PgSQL_Thread.cpp `variables.ffto_enabled` default no longer flips to true when PROXYSQLGENAI is set (the gate was: PROXYSQLFFTO + PROXYSQLGENAI -> true; else false). With PROXYSQLGENAI removed, the default is false everywhere; operators opt in via admin SQL. CLAUDE.md Feature-tier table now lists Plugin Chassis (PROXYSQL40) and AI/MCP (PROXYSQLGENAI) separately, with a note explaining what PROXYSQLGENAI=1 means post-carve-out. Conditional Components section calls out that GenAI/MCP/RAG/LLM lives in plugins/genai/. What this commit does NOT do (deferred) - Stripping the `#ifdef PROXYSQLGENAI` guards from the plugin's own source files (LLM_Bridge.cpp, GenAI_Thread.cpp, etc. all have `#ifdef PROXYSQLGENAI ... #endif` brackets). The plugin's Makefile force-defines the macro so they keep expanding. A guard-stripping pass would be ~50 file edits with zero behaviour change. - Stripping the dead `#ifdef PROXYSQLGENAI` blocks from core (lib/Admin_Handler.cpp etc. still have FIXME-stubbed blocks inside `#ifdef PROXYSQLGENAI` from the 4.C / 5 surgery). With the macro undefined in core's compile line, those blocks silently expand to nothing — they're dead code already. Cleanup follow-up. - Removing the now-no-op stubs themselves (init_mcp_variables, flush_mcp_variables___*, load_mcp_server, stats___mcp_*, etc.). They linger as dead wrappers around early-returns; their declarations in proxysql_admin.h linger too. Cleanup follow-up. All three plugin unit tests pass: genai_plugin_load_unit-t : 25/25 genai_plugin_anomaly_unit-t : 6/6 genai_plugin_backend_client : 27/27 Carve-out scoreboard, FINAL: Step 0-2 : infrastructure (chassis cherry-pick, ABI extensions) Step 3 : ~1.1 K LOC Anomaly_Detector Step 4.A : ~1 K LOC GENAI: / LLM: query-prefix removal Step 4.B : ~0.5 K LOC backend_client helper Step 4.C : ~13 K LOC MCP subsystem + 8 tool handlers + MySQL_FTS Step 4.F : n/a runtime-reconfig admin SQL via plugin commands Step 4.G : n/a MCP admin/config/stats tables via plugin ABI Step 5 : ~7 K LOC GenAI/LLM/AI/RAG Step 6 : ~5 K LOC AI_Vector_Storage / Discovery_Schema / MySQL_Catalog / Static_Harvester pair Step 7 : n/a PROXYSQLGENAI macro dropped from core build Net core surface change: ~28 K LOC moved out of lib/ + include/ into plugins/genai/. src/proxysql binary shrunk from 197 MB (pre-carve-out, all PROXYSQLGENAI compiled in) to 152 MB; plugin .so grew to ~56 MB. Net process RSS roughly neutral.	2 months ago
Rene Cannao	f1c0ea2dbc	feat(genai): Step 3 — carve Anomaly_Detector out of core into plugins/genai/ ... Per docs/superpowers/specs/2026-04-16-genai-plugin-carveout-design.md "Migration sequence — Step 3", this is the first step that actually moves GenAI surface out of libproxysql.a. The Anomaly_Detector now lives entirely inside the genai plugin and is exercised through the plugin query-hook ABI added in Step 2; core has zero references to it. Files moved (git mv preserves history): include/Anomaly_Detector.h -> plugins/genai/include/Anomaly_Detector.h lib/Anomaly_Detector.cpp -> plugins/genai/src/Anomaly_Detector.cpp Plugin-side adjustments: - Drop the `#ifdef PROXYSQLGENAI` guard from both files (the file only compiles inside the plugin now; the macro remains a core build flag but the carve-out gates these files by *location*, not by macro). - Update the json-include relative path to the new depth. - Replace `Anomaly_Detector::get_query_embedding` with a stub that returns an empty vector. The previous implementation called `GloGATH->embed_documents`, but `GloGATH` (the GenAI_Threads_Handler) is still a core symbol at this point in the carve-out. The embedding-similarity stage was already nullptr-guarded on `vector_db`, so an empty embedding short-circuits cleanly to "no anomaly". Step 5 reattaches the embedding back-end when GenAI_Thread moves into the plugin. Plugin lifecycle wiring (plugins/genai/src/plugin_main.cpp): - `genai_init` now: registers two Prometheus counters (`proxysql_genai_detected_anomalies_total`, `proxysql_genai_blocked_queries_total`) against the shared registry exposed by Step 2.3, registers the query hook for both MySQL and PgSQL protocols, and constructs the Anomaly_Detector. - `genai_start`: flips the started flag (the detector has no separate start phase). - `genai_stop`: closes and deletes the detector; counters stay registered (prometheus-cpp has no Unregister). Plugin query-hook adapter (plugins/genai/src/plugin_hooks.cpp, NEW): - Bridges ProxySQL_PluginQueryHookPayload to Anomaly_Detector::analyze. - Increments the appropriate Prometheus counter on detection / block. - Translates AnomalyResult into ALLOW or DENY(message) per the ABI. Core removals (the symmetric carve-out): - lib/MySQL_Session.cpp: delete the in-core `#ifdef PROXYSQLGENAI` anomaly-detection block in COM_QUERY (lines around 5370-5377 pre-edit) and the `handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY_detect_ai_anomaly` method. Same logic now runs inside the plugin via the query hook added in Step 2.2. - include/MySQL_Session.h: drop the matching method declaration. - lib/AI_Features_Manager.{cpp,h}: remove the anomaly_detector field, init_anomaly_detector(), close_anomaly_detector(), get_anomaly_detector(), and the corresponding init() / shutdown() call sites. AI_Features_Manager now owns only the LLM bridge and vector_db (LLM_Bridge moves in Step 5; vector_db in Step 6). - lib/AI_Tool_Handler.{cpp,h}: drop the anomaly_detector field and the constructor parameter that fed it -- the field was stored but never read, dead code that was awkward to keep alive across the carve-out. - lib/ProxySQL_MCP_Server.cpp: update the AI_Tool_Handler ctor call site accordingly (one less argument). - include/MySQL_Thread.h: remove `st_var_ai_detected_anomalies` / `st_var_ai_blocked_queries` from `enum status_variable_id` and `ai_detected_anomalies` / `ai_blocked_queries` from `enum p_th_counter`. The plugin's Prometheus counters replace these (the old ones were per-thread integer increments surfaced by the legacy stats path; the plugin uses Prometheus directly). - lib/MySQL_Thread.cpp: drop the matching entries from MySQL_Thread_status_variables_counter_array and from th_metrics_map (the prometheus registration table). - include/proxysql.h, include/cpp.h: drop the now-stale `#include "Anomaly_Detector.h"` lines. - lib/Makefile: drop Anomaly_Detector.oo from the libproxysql object list. Tests: - Delete the in-core TAP tests that linked against the removed symbols: anomaly_detection-t, anomaly_detection_integration-t, anomaly_detector_unit-t (Docker-dependent, won't link), plus the unit test genai_anomaly_unit-t (linked against libproxysql's copy). - Add test/tap/tests/unit/genai_plugin_anomaly_unit-t.cpp: replaces the deleted unit test by compiling the plugin's Anomaly_Detector.cpp directly into the test binary. The friend-class helper pattern (Anomaly_Detector_TestHelper) is preserved so private methods (normalize_query, check_sql_injection) remain testable. Six assertions cover the same surface plus a regression test that analyze() now runs cleanly without a vector_db. Plugin Makefile changes: - Add Anomaly_Detector.cpp + plugin_hooks.cpp to SRCS. - Pull in the include paths Anomaly_Detector.cpp's transitively- included headers need (mariadb client for mysql.h, libev, libscram, libusual, postgresql, jemalloc, microhttpd, libhttpserver, curl, re2, pcre, zstd, libconfig, libdaemon, ssl). These mirror the set already used by test/tap/tests/unit -- the test harness pulls the same headers in transitively from libproxysql.a's headers. Doxygen documentation added for every new function and for the new header structures (GenAIPluginContext fields, genai_query_hook, register_prometheus_counters, all three lifecycle callbacks, plus file-level @brief blocks for plugin_main.cpp, plugin_hooks.cpp, genai_plugin.h, and the moved Anomaly_Detector.{h,cpp}). Verified end-to-end: - Full clean rebuild of libproxysql.a + proxysql + ProxySQL_GenAI_Plugin.so. - All 60 unit-test binaries pass (genai_anomaly_unit-t replaced by genai_plugin_anomaly_unit-t; net same). - Smoke test: launching proxysql with plugins=("...ProxySQL_GenAI_Plugin.so") shows "Anomaly: Initializing Anomaly Detector v0.1.0" / "Anomaly Detector initialized with 11 injection patterns" in the log on startup, and "Anomaly Detector closed" on SIGTERM -- the plugin's lifecycle owns the detector cleanly. Carve-out scoreboard so far (lines of GenAI surface moved out of core): Step 0-2 : 0 (infrastructure prep only) Step 3 : ~1100 (Anomaly_Detector.{h,cpp}) + counter/handler removals	2 months ago
Rene Cannao	dcfe687a6e	feat: register query_processor_parser variable, extend set_parser_algorithm to 3	2 months ago
Rene Cannao	2a7bddca28	MySrvC: rename server_backoff_time to session_track_backoff_until, gate hot-path read ... Context ------- PR #5166 introduced a per-'MySrvC' atomic timestamp that prevents the server from being chosen for new connections for a 30-second window. The timestamp is written in exactly one place, 'MySQL_Session::handle_session_track_capabilities()', and only when 'mysql-session_track_variables' is in ENFORCED mode and the backend is missing 'CLIENT_SESSION_TRACKING' or 'CLIENT_DEPRECATE_EOF'. The field was originally named 'server_backoff_time' and its only semantics were that ENFORCED-mode capability mismatches temporarily exclude the server from selection. Two issues with the original shape: 1. The name does not hint that the scope is session-tracking capability enforcement. A reader of 'MyHGC::get_random_MySrvC' or 'MySQL_Thread::get_MyConn_local' had no way to tell that this field is tied to one specific feature and not, say, a generic health backoff. 2. The atomic load was unconditional in both hot selection loops. Even though the atomic is relaxed (effectively a 'mov' on x86), per-iteration atomic reads inhibit hoisting and CSE that the compiler can otherwise apply to thread-local integer reads. 'get_random_MySrvC' runs the loop over every server in the hostgroup on every backend-connection acquisition, so this is the sort of place where small per-iteration costs accumulate. Changes ------- * Rename the field: MySrvC::server_backoff_time -> MySrvC::session_track_backoff_until Naming it after the feature makes it obvious that the field is specific to session-tracking enforcement and should not be overloaded for unrelated purposes (e.g. connect errors, monitor failures). If a future change needs a generic backoff mechanism, a new field -- or a rename at that point -- keeps intent clear rather than piggy-backing on this one silently. * Gate the atomic load on mode: In both 'MyHGC::get_random_MySrvC()' (lib/MyHGC.cpp) and 'MySQL_Thread::get_MyConn_local()' (lib/MySQL_Thread.cpp), the relaxed load is skipped entirely when 'mysql_thread___session_track_variables' is not ENFORCED. In 'get_MyConn_local' the check is computed once before the loop; in 'get_random_MySrvC' the check is inside the loop but is a thread-local int read, which is free. Correctness note: switching out of ENFORCED at runtime makes any stale 'session_track_backoff_until' values in the pool immediately ignored, which is the desired behavior -- servers rejected while ENFORCED was active become eligible again the moment the operator relaxes the mode, matching how ProxySQL handles other runtime-variable-dependent state. No explicit reset is required. * Updated the field documentation in include/MySQL_HostGroups_Manager.h with the monotonic-timestamp unit, the list of sites that read/write it, the explicit scope (ENFORCED mode only), and the reset policy (no explicit reset; deadline ages out by comparison against curtime). No functional change for DISABLED / OPTIONAL configurations (the default for all installations). ENFORCED mode behaviour is unchanged.	2 months ago
Wazir Ahmed	f815b77812	fix: make server_backoff_time atomic ... Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	2 months ago
Wazir Ahmed	885712c679	Merge branch 'v3.0' into session-track-system-variable	2 months ago
Rene Cannao	aa24529f39	feat: register mysql-zstd_compression_level variable in MySQL thread	2 months ago
Rene Cannao	7e1a78fd6b	lint: fix reserved identifiers and include guards ... - Rename __HGM_TABLES_SIZE -> HGM_TABLES_SIZE_ (Base/MySQL/PgSQL HGM) - Rename __TYPE_SIZE -> TYPE_SIZE_ (Base/MySQL/PgSQL HGM) - Fix include guard defines: __PROXYSQL_* -> PROXYSQL_* - Fix include guard comments: __CLASS_* -> PROXYSQL_*_H (29 files) - Add uint8_t underlying type to small enums	3 months ago
René Cannaò	f713d78e41	Merge pull request #5554 from sysown/copilot/add-flag-choose-ipv4-or-ipv6 ... Add `mysql-resolution_family` to control IPv4/IPv6 hostname resolution	3 months ago
Rene Cannao	0e1ea6a734	Address code review feedback for mysql-resolution_family ... - Add documentation comments to MySQL_Resolution.h explaining NULL fallback behavior for all three helper functions - Initialize mysql_thread___resolution_family to "system" in MySQL_Thread constructor to prevent NULL before first refresh_variables() call - Add explanatory comment in mysql_resolution_unit-t.cpp documenting the DNS cache bypass limitation for future contributors Review: #5554	3 months ago
Rene Cannao	424a3664ae	Use thread-local storage for resolution_family variable ... Replace direct GloMTH->get_variable_string() call in monitor_dns_resolver_thread() with the proper mysql_thread___ thread-local variable pattern: - Add __thread char* mysql_thread___resolution_family in proxysql_structs.h - Add REFRESH_VARIABLE_CHAR(resolution_family) in refresh_variables() - Free the thread-local variable in MySQL_Thread destructor - Read mysql_thread___resolution_family directly in the monitor thread instead of taking a lock via get_variable_string()	3 months ago
copilot-swe-agent[bot]	bc843a1a25	test: cover mysql resolution family helper cases ... Agent-Logs-Url: https://github.com/sysown/proxysql/sessions/7d0ebed6-886c-4e06-add5-d74b44eced4a Co-authored-by: renecannao <3645227+renecannao@users.noreply.github.com>	3 months ago
copilot-swe-agent[bot]	4a93861c6c	feat: add mysql resolution_family setting ... Agent-Logs-Url: https://github.com/sysown/proxysql/sessions/7d0ebed6-886c-4e06-add5-d74b44eced4a Co-authored-by: renecannao <3645227+renecannao@users.noreply.github.com>	3 months ago
Rene Cannao	61fc7a129c	Disable FFTO by default in 3.1.x, keep enabled in 4.0.x (#5539) ... Gate the ffto_enabled default on PROXYSQLGENAI so that: - 3.1.x (PROXYSQL31): FFTO compiled in but disabled by default - 4.0.x (PROXYSQLGENAI): FFTO enabled by default (unchanged) Users on 3.1.x can still opt in via SET mysql-ffto_enabled='true'.	3 months ago
Rene Cannao	3458cbd28b	fix: address zstd review feedback	3 months ago
copilot-swe-agent[bot]	ec44a15058	feat: add mysql protocol zstd compression support ... Co-authored-by: renecannao <3645227+renecannao@users.noreply.github.com>	3 months ago
Rene Cannao	6b46b26931	chore: implement PR review feedback for release tier system ... - Makefile: normalized GIT_VERSION by stripping 'v' prefix before arithmetic. - Makefile: corrected .PHONY declaration for build_deps_debug_default. - Admin Handler: properly guarded TSDB commands to return error when disabled. - REST API: removed redundant nested guards inside tsdb_resource class. - Statistics: implemented rigorous input validation for set_variable. - Statistics: added return value checks and rollback logic for TSDB migrations. - Statistics: implemented bounded concurrency (batching) for backend probes. - FFTO: implemented cumulative buffer limit enforcement in MySQL and PgSQL. - FFTO: ensured initial packet capture in PgSQL_Session by early initialization. - FFTO: improved reassembly buffer compaction logic. - Headers: default-initialized ffto_bypassed to false. - Code cleanup: merged redundant PROXYSQLFFTO guards.	4 months ago
Rene Cannao	c2f678a9de	feat: implement release tiers and feature guards for v3.0, v3.1, and v4.0 ... This commit introduces a structured release tier system controlled by environment variables, allowing three distinct ProxySQL versions to be built from the same codebase while maintaining a clear upgrade path. Key Changes: - Tier 1 (v3.0.x Stable): Default build including bug fixes and core enhancements. - Tier 2 (v3.1.x Innovative): Enabled via PROXYSQL31=1. Adds FFTO and TSDB. - Tier 3 (v4.0.x AI/MCP): Enabled via PROXYSQLGENAI=1. Adds the AI stack. Implementation Details: - Makefile: Implemented dynamic versioning logic that increments the minor version for v3.1 and the major version for v4.0. Added tier documentation. - Feature Guards: Introduced PROXYSQLFFTO and PROXYSQLTSDB guards across ProxySQL_Admin, ProxySQL_Statistics, REST API, and HTTP server. - FFTO: Guarded Fast Forward Traffic Observer in MySQL and PgSQL sessions, including thread-local variables and configuration parameters. - TSDB: Guarded the entire Time Series Database subsystem, including SQLite table definitions, background loops, and REST API endpoints. - Build Fixes: * Fixed 'proxy_debug_func' linkage by adding extern "C" and moving the definition outside of DEBUG guards to ensure non-debug builds link correctly. * Corrected misplaced/nested DEBUG and PROXYSQLFFTO guards. * Verified build stability for all tiers (unset, PROXYSQL31=1, PROXYSQLGENAI=1).	4 months ago
René Cannaò	80f9c1a079	Merge branch 'v3.0' into session-track-system-variable ... Signed-off-by: René Cannaò <rene@proxysql.com>	4 months ago
Rene Cannao	acef17a8c1	Merge branch 'v3.0-ff_inspect' into v3.0-unified-large-pr ... # Conflicts: # test/tap/tests/Makefile	4 months ago
Rene Cannao	e2ebc0c449	Merge branch 'v4.0-mcp-stats2' into v3.0-unified-large-pr	4 months ago
Rene Cannao	5519080ac2	mcp stats: address review findings and remove --genai CLI flag	4 months ago
Rene Cannao	1c25787b88	Introduce {mysql,pgsql}-query_processor_first_comment_parsing variable (#5384) ... This universal fix allows controlling when the first comment of a query is processed relative to the query rules. By setting this variable to 1, ProxySQL-specific annotations (like GTID) can be parsed immediately, allowing subsequent query rules to strip them from the query string. This prevents prepared statement cache bloat and improves backend statement reuse when annotations with unique literals are used.	4 months ago
Rene Cannao	1258a1f2f2	Address PR reviews: improve performance, safety, and robustness of FFTO	4 months ago
Rene Cannao	e762f91cda	MCP stats: add in-memory processlist filtering and PgSQL concurrency stress TAP ... Replace direct stats-schema reads in MCP show_processlist with in-memory processlist access via SQL3_Processlist query options. Add processlist query option types in proxysql_admin interfaces and implement filtering, sorting, and pagination post-processing in both MySQL and PgSQL thread implementations. Introduce configurable MCP cap mcp_stats_show_processlist_max_rows (default 200, max 1000), wire it through MCP variable loading, and expose it in default config. Expand Stats_Tool_Handler processlist handling to parse filter/sort arguments, enforce cap metadata, and return consistent MCP payload fields from in-memory snapshots. Add extensive doxygen documentation in the touched headers and source blocks to describe API contracts, filtering behavior, and runtime constraints. Add new TAP test mcp_pgsql_concurrency_stress-t that generates sustained concurrent PgSQL traffic (simple reads, read/write table workload, randomized pg_sleep) while polling MCP show_processlist and show_queries in parallel with filter and ordering assertions. Validation performed locally: mcp_pgsql_concurrency_stress-t (31 assertions) and mcp_show_queries_topk-t (12 assertions).	4 months ago
Rene Cannao	3a3be5a559	Implement Fast Forward Traffic Observer (FFTO) for MySQL and PostgreSQL ... - Added TrafficObserver base interface - Implemented MySQLFFTO and PgSQLFFTO classes for protocol-aware observation - Integrated FFTO hooks into MySQL and PostgreSQL session handlers in FAST_FORWARD state - Added configuration variables: mysql-ffto_enabled, mysql-ffto_max_buffer_size, pgsql-ffto_enabled, pgsql-ffto_max_buffer_size - Implemented query digest extraction and reporting parity with internal stats - Added session-level bypass when packet size exceeds buffer threshold - Included architectural documentation in doc/ffto_design.md	4 months ago
Rene Cannao	caf324f911	logger: harden buffered logging flush/rotation semantics ... This commit addresses three correctness regressions introduced by thread-local log buffering in PR #5364 (query logging performance): 1) stale logfile pointer/UAF race during concurrent rotate/close 2) stale logfile-open state after close 3) non-global flush behavior in admin/format-switch paths What was fixed - Make `flush_and_rotate()` consume the current logfile pointer under lock - Signature changed from `std::fstream*` to `std::fstream*&` - Prevents dereferencing a stale stream pointer captured before lock acquisition while another thread rotates/closes the file - Updated declaration/definition and all call sites - Add explicit synchronization for cross-thread buffer draining - Added `LogBufferThreadContext::buffer_lock` - Any path that appends or flushes a thread buffer now locks this mutex - Guarantees force-flush from admin/config paths cannot race with worker-thread appends on the same context - Restore global forced flush semantics where required - Extended `MySQL_Logger::flush` and `PgSQL_Logger::flush` to `flush(bool force = false)` - `force=false`: preserves existing low-overhead worker-loop behavior (per-thread timeout-based flush) - `force=true`: snapshots all known thread contexts and drains both events/audit buffers regardless of timeout - `flush_log()` now calls `flush(true)` before file rotation, so admin flush and format-switch operations no longer miss pending thread buffers - Avoid unintended rotation during forced draining - In `force=true` path, flush uses `rotate_fn=nullptr` - Drains buffered payload into the current file first - `flush_log()` then performs one controlled rotate/open step - Fix logfile state tracking after close - `events_close_log_unlocked()` and `audit_close_log_unlocked()` now set `logfile_open=false` when the stream is closed - Prevents write paths from treating a closed stream as open - Remove per-thread-context dependency during metadata header write - `events_open_log_unlocked()` now uses a local `LogBuffer` for metadata emission in format=1 instead of reusing a thread context buffer - Keeps open/rotate path independent from worker context lifecycle - Keep callers consistent and non-duplicative - Since `flush_log()` now force-drains internally, removed redundant explicit `flush()` calls from: - MySQL/PgSQL `eventslog_format` switch handlers - `ProxySQL_Admin::flush_logs` Behavioral outcome - No stale stream pointer use when close/rotate interleaves with flush - No false-positive logfile-open state after close - `FLUSH LOGS` and eventslog format switch now drain all known thread buffers before rotating, preventing dropped/misplaced buffered records Validation - Built modified objects directly - Ran full debug build with GENAI enabled: make clean && export PROXYSQLGENAI=1 && make debug -j24 Build completed successfully.	4 months ago
René Cannaò	568fe1524e	Merge pull request #5364 from mevishalr/feature/query-logging-improvements ... Optimize query logging performance (#5243)	4 months ago
Rene Cannao	cc53ddca23	Merge branch 'v3.0-openssl-fix' into v3.0-test0213	5 months ago
vramesha	15a4ed66e9	Optimize query logging performance (#5243) ... Detailed Changes: - Introduced LogBuffer and LogBufferThreadContext to implement per-thread buffering, reducing lock contention. - Replaced localtime() with localtime_r() for improved performance and thread safety during timestamp generation. - Implemented configurable sampling for event logs to reduce I/O overhead. - Added global configuration variables to control flush size, flush interval, and sampling rate for event and audit logs.	5 months ago
Rene Cannao	778e01174c	Add protocol labels to Thread Handler metrics and enable PostgreSQL metrics ... This commit extends PR #5069 by adding protocol labels to distinguish MySQL and PostgreSQL metrics in the Thread Handler modules. Changes: - Add { "protocol", "mysql" } labels to 65 MySQL_Thread.cpp metrics - Add { "protocol", "pgsql" } labels to 60 PgSQL_Thread.cpp metrics - Enable GloPTH (PostgreSQL threads handler) metrics export - Document Query Cache limitation (template base class collision) This resolves Issue #5068 by ensuring PostgreSQL prometheus metrics are properly exported with protocol labels, distinguishing them from MySQL metrics. Related: #5068, #5069	5 months ago
Rene Cannao	4c788dcbb3	fix: signed/unsigned comparison warnings (-Wsign-compare) ... Fix two HIGH priority signed/unsigned comparison warnings: 1. MySQL_HostGroups_Manager.cpp:1792 - Change loop variable from 'int j' to 'unsigned int j' to match the unsigned comparison with servers->len. 2. MySQL_Thread.cpp:4357 - Cast int to size_t when comparing with getBufferSize() which returns size_t. These changes prevent potential logic errors from negative values being interpreted as large unsigned values. Found during compiler warning analysis with 'make debug'.	5 months ago
René Cannaò	75a62f334d	Merge branch 'v4.0' into v3.1-vec ... Signed-off-by: René Cannaò <rene@proxysql.com>	5 months ago
Rene Cannao	52a70b0b09	feat: Implement AI-based Anomaly Detection for ProxySQL ... Phase 3: Anomaly Detection Implementation This commit implements a comprehensive multi-stage anomaly detection system for real-time SQL query security analysis. **Core Detection Methods:** 1. **SQL Injection Pattern Detection** (lib/Anomaly_Detector.cpp) - Regex-based detection of 11 SQL injection patterns - Suspicious keyword detection (11 patterns) - Covers: tautologies, union-based, comment-based, stacked queries 2. **Query Normalization** (lib/Anomaly_Detector.cpp:normalize_query) - Converts to lowercase - Removes SQL comments - Replaces string/numeric literals with placeholders - Normalizes whitespace 3. **Rate Limiting** (lib/Anomaly_Detector.cpp:check_rate_limiting) - Per user/host query rate tracking - Configurable time windows (3600s default) - Auto-block on threshold exceeded - Prevents DoS and brute force attacks 4. **Statistical Anomaly Detection** (lib/Anomaly_Detector.cpp:check_statistical_anomaly) - Z-score based outlier detection - Abnormal execution time detection (>5s) - Large result set detection (>10000 rows) - Behavioral profiling per user 5. **Embedding-based Similarity** (lib/Anomaly_Detector.cpp:check_embedding_similarity) - Placeholder for vector similarity search - Framework for sqlite-vec integration - Detects novel attack variations **Query Flow Integration:** - Added `detect_ai_anomaly()` to MySQL_Session (line 3626) - Integrated after libinjection SQLi detection (line 5150) - Blocks queries when risk threshold exceeded (default: 0.70) - Sends error response with anomaly details **Status Variables Added:** - `ai_detected_anomalies`: Total anomalies detected - `ai_blocked_queries`: Total queries blocked - Available via: `SELECT * FROM stats_mysql_global` **Configuration (defaults):** - `enabled`: true - `risk_threshold`: 70 (0-100) - `similarity_threshold`: 85 (0-100) - `rate_limit`: 100 queries/hour - `auto_block`: true - `log_only`: false **Detection Pipeline:** ``` Query → SQLi Check → AI Anomaly Check → [Block if needed] → Execute (libinjection) (Multi-stage) ``` **Files Modified:** - include/MySQL_Session.h: Added detect_ai_anomaly() declaration - include/MySQL_Thread.h: Added AI status variables - lib/Anomaly_Detector.cpp: Full implementation (700+ lines) - lib/MySQL_Session.cpp: Integration and query flow - lib/MySQL_Thread.cpp: Status variable definitions **Next Steps:** - Add unit tests for each detection method - Add integration tests with sample attacks - Add user and developer documentation Related: Phase 1-2 (NL2SQL foundation and testing) Related: Phase 4 (Vector storage for embeddings)	5 months ago
René Cannaò	f82bcdff38	Merge branch 'v3.0' into session-track-system-variable ... Signed-off-by: René Cannaò <rene@proxysql.com>	6 months ago
Rene Cannao	253591d262	Add experimental GenAI EMBED: query support for MySQL ... This commit adds experimental support for generating embeddings directly from MySQL queries using a special EMBED: syntax. Changes: - Add MYDS_INTERNAL_GENAI to MySQL_DS_type enum for GenAI connections - Add handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___genai_embedding() - Implement EMBED: query detection and JSON parsing for document arrays - Build CSV resultset with embeddings (1 row per document, 1 column) - Add myconn NULL check in MySQL_Thread for INTERNAL_GENAI type - Add "debug_genai" name to debug module array - Remove HAVE_LIBCURL checks (libcurl is always statically linked) - Use static curl header: "curl/curl.h" instead of <curl/curl.h> - Remove curl_global_cleanup() from GenAI module (should only be in main()) Query format: EMBED: ["doc1", "doc2", ...] Result format: 1 row per document, 1 column with CSV embeddings Error handling uses MySQL ERR_Packet instead of resultsets.	6 months ago
Rene Cannao	b5598d8d53	Add comprehensive ProxySQL_Poll usage documentation throughout codebase ... Enhance ProxySQL_Poll class documentation with detailed usage patterns: - lib/ProxySQL_Poll.cpp: Enhanced file-level documentation with architecture overview, template specialization, memory management, and event processing pipeline explanations - lib/MySQL_Thread.cpp: Added usage documentation for listener registration, removal patterns, client session setup, and main poll loop - lib/PgSQL_Thread.cpp: Added equivalent PostgreSQL usage documentation mirroring MySQL patterns with protocol-specific details - lib/mysql_data_stream.cpp: Documented cleanup, receive activity tracking, and send activity tracking patterns - lib/PgSQL_Data_Stream.cpp: Documented equivalent PostgreSQL data stream patterns for cleanup and activity tracking All documentation is placed directly where code is used, avoiding specific line numbers for better maintainability. Includes comprehensive explanations of when, why, and how ProxySQL_Poll methods are used throughout ProxySQL's event-driven architecture. [skip-ci]	6 months ago
Rene Cannao	0699c7ba11	Add comprehensive ProxySQL_Poll usage documentation throughout codebase ... Enhance ProxySQL_Poll class documentation with detailed usage patterns: - lib/ProxySQL_Poll.cpp: Enhanced file-level documentation with architecture overview, template specialization, memory management, and event processing pipeline explanations - lib/MySQL_Thread.cpp: Added usage documentation for listener registration, removal patterns, client session setup, and main poll loop - lib/PgSQL_Thread.cpp: Added equivalent PostgreSQL usage documentation mirroring MySQL patterns with protocol-specific details - lib/mysql_data_stream.cpp: Documented cleanup, receive activity tracking, and send activity tracking patterns - lib/PgSQL_Data_Stream.cpp: Documented equivalent PostgreSQL data stream patterns for cleanup and activity tracking All documentation is placed directly where code is used, avoiding specific line numbers for better maintainability. Includes comprehensive explanations of when, why, and how ProxySQL_Poll methods are used throughout ProxySQL's event-driven architecture. [skip-ci]	6 months ago
Rene Cannao	366164ab26	Fix #5272: Add mysql-select_version_forwarding variable for SELECT VERSION() ... Since ProxySQL 3.0.4, SELECT VERSION() queries were intercepted and returned ProxySQL's mysql-server_version variable instead of proxying to backends. This broke SQLAlchemy for MariaDB which expects "MariaDB" in the version string. This commit adds a new variable `mysql-select_version_forwarding` with 4 modes: - 0 = never: Always return ProxySQL's version (3.0.4+ behavior) - 1 = always: Always proxy to backend (3.0.3 behavior) - 2 = smart (fallback to 0): Try backend connection, else ProxySQL version - 3 = smart (fallback to 1): Try backend connection, else proxy (default) The implementation includes: - New global variable mysql_thread___select_version_forwarding - New function get_backend_version_for_hostgroup() to peek at backend connection versions without removing them from the pool - Modified SELECT VERSION() handler to support all 4 modes - ProxySQL backend detection to avoid recursion Mode 3 (default) ensures SQLAlchemy always gets the real MariaDB version string while maintaining fast response when connections are available.	6 months ago
Wazir Ahmed	f0ac86c317	Enable session tracking based on server capabilities ... Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago
Wazir Ahmed	4b9c083209	MySQL: Track session-specific system variables changes ... - Add new mysql variable `mysql-session_track_variables`. - Configure `session_track_system_variables` and `session_track_state_change` on backend connections if the mysql variable is enabled. - Utilize notifications from backend servers to capture system variable changes that cannot be handled by `MySQL_Set_Stmt_Parser` - Update both client and server variable maps based on backend responses. - TAP test to verify this patch. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago