1031 Commits (f42c3ee1ab66ec2ec408976c3078bb66a954d5e2)

Author	SHA1	Message	Date
Rene Cannao	f42c3ee1ab	feat(chassis): add register_runtime_view ABI for module-owned state ... Plugins that declare admin-side runtime views of their own in-memory state need a way to register a projection callback the chassis can invoke when admin SELECTs against the registered table. Without this hook, the only way for a plugin to surface its runtime state to admin operators was to *persist* a duplicate copy of the data into admin_db, which violates the separation of duties between Admin (owns configuration tables and views) and the module (owns runtime state). Refs #5687. ABI surface (in include/ProxySQL_Plugin.h): - struct ProxySQL_PluginRuntimeView { table_name, refresh, opaque } - new services.register_runtime_view callback - bumps PROXYSQL_PLUGIN_ABI_VERSION to 3 The new field is appended at the end of ProxySQL_PluginServices, so ABI-2 plugins keep working — they neither set nor read past the previous layout. ABI 3 plugins set abi_version=3 and use the new field. Loader accepts ABI 1, 2, and 3. Plumbing (in lib/ProxySQL_PluginManager.cpp): - register_runtime_view_service() free fn wires through to the manager's runtime_views_ vector - ProxySQL_PluginManager::register_runtime_view() rejects empty table names, null callbacks, and duplicate registrations - sql_references_table_ci() does whole-identifier substring match, so a SELECT on `runtime_mysqlx_users` doesn't fire the refresh for `runtime_mysqlx_users_extra` if both ever coexist - refresh_runtime_views_for_query() iterates registered views and invokes the matching refresh callbacks - proxysql_refresh_configured_plugin_runtime_views() is the Admin-callable wrapper that takes the manager shared lock Admin-handler integration (in lib/ProxySQL_Admin.cpp): - the existing pre-SELECT refresh block (where runtime_mysql_users triggers save_mysql_users_runtime_to_database(true), etc.) now also calls proxysql_refresh_configured_plugin_runtime_views(). This is the SAME refresh point the canonical core tables use, so plugin views are guaranteed to refresh before the admin query actually executes against admindb. Doc block at the bottom of ProxySQL_Plugin.h rewritten to spell out the separation-of-duties contract explicitly, replacing the previous "copy_table guidance" that incorrectly endorsed plugins persisting their runtime state to admin_db tables. This commit is the foundation only — no plugin uses the new API yet. The follow-up commits convert the mysqlx plugin's four entity pairs (users / routes / endpoints / variables) to the canonical pattern.	4 weeks ago
Rene Cannao	04bccec51e	chore(plugin-chassis): tighten gating, drop dead paths, gate forgery setters ... Five clean-up items from the independent review of PR #5651. None of these change behaviour on a normal run; they each fix a concrete way the current code is misleading or unnecessarily exposed. 1) lib/ProxySQL_Admin.cpp: gate the three plugin-DB-handle getters (proxysql_plugin_get_admindb / _configdb / _statsdb) under #ifdef PROXYSQL40. Previously these were defined unconditionally and emitted symbols into v3.0/v3.1 binaries. The chassis is a v4.0 feature; the user explicitly required that v3.x builds carry no plugin-aware code. Wrap in PROXYSQL40 so they are entirely absent from v3.x linkage. ProxySQL_PluginManager.cpp's extern declarations (lines 23-25) are already inside the file-wide PROXYSQL40 gate and so resolve only when the gate is active. 2) lib/ProxySQL_PluginManager.cpp + include/ProxySQL_PluginManager.h: delete the dead `#else /* !PROXYSQL40 */` branches. Both files are wrapped in a top-level `#ifdef PROXYSQL40` covering the entire body. Inner `#ifdef PROXYSQL40 ... #else ... #endif` blocks therefore had unreachable `#else` arms — 30+ lines of "pre-chassis two-phase loader" in the .cpp, plus a redundant declaration of proxysql_load_configured_plugins in the .h. The dead arms read as load-bearing alternative implementations on review and that is exactly the wrong signal. Drop them; the single PROXYSQL40 path is the only one. 3) lib/Admin_Bootstrap.cpp + include/proxysql_admin.h + src/main.cpp: remove ProxySQL_Admin::materialize_plugin_tables(). `Admin::init()` already merges plugin-declared schemas into tables_defs_{admin,config,stats} (~line 944) and runs the DDL via check_and_build_standard_tables() (~line 994), all on the same first-boot/reload code path the core tables use. The follow-up call to GloAdmin->materialize_plugin_tables() in main.cpp ran a second name-dedup pass that found everything already present and produced an empty new-rows set — i.e. the post-init helper was a no-op disguised as load-bearing infrastructure. Delete the helper, the header declaration, and the main.cpp call site, and update the comments in main.cpp + ProxySQL_PluginManager.cpp to point at Admin::init() as the single canonical materialization site. Leave a NOTE in Admin_Bootstrap.cpp at the old call site so anyone re-adding a similar helper sees why the prior one was removed. 4) plugins/mysqlx/include/mysqlx_session.h + plugins/mysqlx/src/mysqlx_session.cpp: gate the two genuine forgery vectors behind MYSQLX_TEST_BUILD. inject_identity_for_test() bypasses the full auth flow — no credential check, no capability negotiation, just sets identity_ to a caller-supplied MysqlxResolvedIdentity. resolve_backend_target_ for_test() drives a private routing helper without an authenticated identity. Both are necessary for unit tests but should not be reachable at all in shipped binaries; an in-process exploit reaching the session can call inject_identity_for_test() to forge an authenticated identity. Wrap them in #ifdef MYSQLX_TEST_BUILD; define -DMYSQLX_TEST_BUILD in the test Makefile only. The remaining target_*_for_test() getters are read-only state observers and are left unconditional — they cannot mutate the session and a debugger could observe the same state regardless. 5) test/tap/tests/unit/Makefile: define -DMYSQLX_TEST_BUILD on every unit test compile line via OPT. This is the test-only knob that re-enables the gated forgery methods so unit tests still compile. plugins/mysqlx/Makefile does NOT define this macro, so the production .so does not compile in the entry points. Verified locally: - plugins/mysqlx/ProxySQL_MySQLX_Plugin.so builds clean with PROXYSQL40=1 PROXYSQL31=1 PROXYSQLFFTO=1 PROXYSQLTSDB=1 PROXYSQLGENAI=1 (no MYSQLX_TEST_BUILD). - test/tap/tests/unit/mysqlx_robustness_unit-t builds and runs: 74/74 assertions pass, including the ones that exercise inject_identity_for_test (visible only because the test Makefile defines MYSQLX_TEST_BUILD).	4 weeks ago
Rene Cannao	3ba92815f8	fix(plugin-chassis): make chassis fully invisible in v3.x builds ... Review feedback identified that the chassis was leaking into the v3.0 stable build via unfenced references in core code: * Admin_Bootstrap.cpp had `if (auto* m = proxysql_get_plugin_manager())` and a full materialize_plugin_tables() implementation outside any PROXYSQL40 fence * proxysql_admin.h declared materialize_plugin_tables() * proxysql_glovars.hpp held GloVars.plugin_modules and proxysql_load_plugin_modules_from_config() * ProxySQL_GloVars.cpp had three unfenced .clear()/use sites * ProxySQL_Admin.cpp::dispatch_plugin_admin_command (template + two instantiations) was unfenced * src/main.cpp's GloPluginManager, LoadConfiguredPlugins, InitConfiguredPlugins, StartConfiguredPlugins, StopConfiguredPlugins, UnloadPlugins teardown, the four-phase orchestration block, and the proxysql_load_plugin_modules_from_config call were all unfenced Separately, Admin_Handler.cpp held 16 hardcoded MYSQLX alias vectors (LOAD_MYSQLX_USERS_FROM_MEMORY, etc.) and a 16-deep if-ladder that called resolve_admin_alias_to_canonical(). This embedded plugin-specific knowledge in core code -- core should know nothing about any specific plugin's commands. The agreed architecture: v3.0/v3.1 builds have NO plugin loader at all; the entire plugin chassis (including its public manager surface, its loader, and its dispatch helpers) only exists under PROXYSQL40. v3.0 core code therefore holds zero plugin-related code paths. Changes: * include/ProxySQL_Plugin.h, include/ProxySQL_PluginManager.h, lib/ProxySQL_PluginManager.cpp: wrap entire file body in `#ifdef PROXYSQL40 ... #endif`. Including the header from a v3.x translation unit yields nothing; the .cpp compiles to an empty object file. Removed redundant inner PROXYSQL40 fences inside the now-empty v3.x body of Plugin.h (kept the few that are inside the file-wide outer for clarity). PROXYSQL_PLUGIN_ABI_VERSION is now unconditionally 2 (the macro is no longer reachable from a v3.x compile, so the PROXYSQL40-vs-1 split is moot). * include/proxysql_glovars.hpp, lib/ProxySQL_GloVars.cpp: gate GloVars.plugin_modules field, all .clear() sites, the proxysql_load_plugin_modules_from_config declaration + definition. * include/proxysql_admin.h: gate materialize_plugin_tables and dispatch_plugin_admin_command<S> declarations. * lib/Admin_Bootstrap.cpp: gate `#include "ProxySQL_PluginManager.h"`, the `if (auto* m = proxysql_get_plugin_manager())` merge block, and the materialize_plugin_tables definition. * lib/Admin_Handler.cpp: delete the 16 LOAD_/SAVE_MYSQLX_* const std::vector<std::string> declarations entirely. Delete the helper resolve_admin_alias_to_canonical(). Delete the 16-deep if-ladder + the !PROXYSQL40 fallback dispatch branch. The remaining plugin dispatch is one generic call to proxysql_resolve_configured_plugin_admin_alias() inside an #ifdef PROXYSQL40 block; v3.x compiles it out entirely. Core now holds no plugin-specific knowledge of any kind. * lib/ProxySQL_Admin.cpp: gate dispatch_plugin_admin_command template body + both explicit instantiations. * src/main.cpp: gate `#include "ProxySQL_PluginManager.h"`, GloPluginManager declaration, all four wrapper functions (LoadConfiguredPlugins / InitConfiguredPlugins / StartConfiguredPlugins / StopConfiguredPlugins), the StopConfiguredPlugins call inside UnloadPlugins, the four-phase orchestration in ProxySQL_Main_init_phase2___not_started, the GloVars.plugin_modules.clear() and the proxysql_load_plugin_modules_from_config call in ProxySQL_Main_process_global_variables. v3.x builds reduce that whole block to a single ProxySQL_Main_init_Admin_module call. * plugins/mysqlx/src/mysqlx_plugin.cpp, plugins/mysqlx/src/mysqlx_admin_schema.cpp: drop the !PROXYSQL40 fallback paths. The plugin only loads under PROXYSQL40 now (no loader exists in v3.x), so the legacy combined-init code path and the 16 fallback register_command() calls were dead code. The descriptor unconditionally wires register_schemas; mysqlx_init holds only context setup. * Makefile (top-level): the four build_src_* recipes that recurse into plugins/mysqlx are wrapped in `$(if $(filter 1,$(PROXYSQL40)),...)`. v3.x builds emit "[skip] mysqlx plugin (PROXYSQL40 not set)" instead of building the .so. Clean targets stay unconditional. * test/tap/tests/unit/mysqlx_admin_alias_resolution_unit-t.cpp: delete the test (it covered resolve_admin_alias_to_canonical, which is itself deleted). Remove from unit Makefile UNIT_TESTS list. Verification: * v3.0 stable build (`make`, no flags): - Produces 147 MB src/proxysql binary - `nm src/proxysql | grep -cE 'invoke_register_schemas_phase|GloPluginManager|materialize_plugin_tables'` -> 0 - mysqlx plugin .so is correctly skipped: "[skip] mysqlx plugin (PROXYSQL40 not set)" * PROXYSQLGENAI=1 build (cascades PROXYSQL40/31/FFTO/TSDB): - Produces 200 MB src/proxysql binary with chassis symbols - mysqlx plugin .so built (9.2 MB), contains mysqlx_register_schemas * All 5 plugin unit tests pass under PROXYSQL40: plugin_config_unit-t 48/48 plugin_dispatch_unit-t 52/52 plugin_lifecycle_unit-t 26/26 plugin_query_hook_unit-t 46/46 plugin_prometheus_unit-t 10/10	1 month ago
Rene Cannao	5c1549dcc6	merge: resolve conflicts with v3.0 branch ... Merge origin/v3.0 into ProtocolX. One conflict in test/tap/tests/unit/Makefile resolved by keeping both mysqlx unit tests (ProtocolX) and pgsql_servers_ssl_params unit test + GENAI vec.o link (v3.0).	1 month ago
René Cannaò	669ab91494	Merge pull request #5594 from sysown/v3.0-lint ... lint + static analysis: clang-tidy and cppcheck fixes	2 months ago
Rene Cannao	f1826cf206	lint: fix reserved identifier in member variables ... Rename __active -> active_, __frontend -> frontend_, __backend -> backend_ in authentication and hostgroups classes to comply with C++ reserved identifier rules. Includes updates to: - MySQL_Authentication.hpp - ClickHouse_Authentication.hpp - PgSQL_Authentication.h - MySQL_HostGroups_Manager.h - Base_HostGroups_Manager.h - Corresponding .cpp files	2 months ago
Rene Cannao	ac0f9304b0	fix: address code review findings for pgsql cluster sync ... 1. Remove dead declarations and implementations: PullOperationConfig, pull_from_peer_unified, fetch_query_with_metrics, compute_*_checksum, ScopedChar* wrappers, and all safe_* memory helpers (-690 lines) 2. Document mutex sharing: pgsql pull functions intentionally reuse MySQL mutexes since pulls are sequential in the monitoring thread 3. Fix variable shadowing: rename local 'error' to 'exec_error' in Admin_Handler.cpp pgsql_servers_v2 fallback to avoid shadowing the outer error variable 4. Fix misleading debug message in process_component_checksum: the else branch fires when peer checksum is unchanged, not different 5. Fix indentation in ProxySQL_Admin.cpp (3 lines with double-tab) and Admin_FlushVariables.cpp (NULL-guard block with extra indent)	2 months ago
Rene Cannao	80ff2242e6	merge: resolve v3.0 conflict and add pgsql_servers_ssl_params cluster sync ... v3.0 added pgsql_servers_ssl_params table. Integrate into pgsql cluster sync path: - Add CLUSTER_QUERY_PGSQL_SERVERS_SSL_PARAMS to Admin_Handler interception block alongside other pgsql cluster queries - Add ssl_params fetch query to pull_pgsql_servers_v2_from_peer - Add update_pgsql_servers_ssl_params() for applying synced data - Update convert_pgsql_servers_resultsets to handle 5 result sets - Include ssl_params in v2 checksum computation (4 tables)	2 months ago
René Cannaò	93eabfc7bc	fix: address code review issues in PostgreSQL cluster sync ... - Fix fetch_and_store incrementing success counter on failure - Fix sqlite3_stmt leak in pull_pgsql_variables_from_peer (use RAII) - Convert get_peer_to_sync_pgsql_query_rules and _servers_v2 to unified framework delegates (fixes atomic data race on .load()) - Merge dual scan loops in set_checksums to respect enabled_check - Remove spurious "deprecated" label from new checksum_pgsql_variables - Replace sprintf with snprintf in pgsql diffs_before_sync getters - Fix indentation on cluster_pgsql_variables blocks - Fix TAP plan mismatch: replace MYSQL_QUERY with inline mysql_query - Fix trivially-true table accessibility assertions - Add mysql_close on failed connect in test - Move test_cluster_sync_pgsql-t to legacy-g5 group	2 months ago
Rahim Kanji	e68f8bb74a	refactor: simplify pgsql_servers_ssl_params schema and pre-parse TLS range ... - Drop ssl_capath and ssl_cipher columns (no libpq equivalents) - Pre-parse ssl_protocol_version_range into ssl_min/max_protocol_version at construction time instead of on every backend connection - Fix Servers_SslParams 3-arg delegating constructor - Update tests, docs, and all consumers (Connection, Monitor, Kill, Admin)	2 months ago
René Cannaò	8a61232517	fix: decouple MySQL and PostgreSQL checksum/sync controls ... Two bugs in the PostgreSQL cluster sync admin variables: 1. Cross-protocol coupling (critical): Disabling a MySQL checksum flag (e.g. SET admin-checksum_mysql_query_rules='false') incorrectly zeroed out the corresponding PostgreSQL cluster sync threshold (cluster_pgsql_query_rules_diffs_before_sync). An operator disabling MySQL checksums would unexpectedly halt PostgreSQL cluster sync. Fix: MySQL checksum setters now only affect MySQL sync thresholds. The warning messages no longer mention pgsql side effects. 2. Single-gate for all pgsql modules (operational limitation): All four PostgreSQL cluster sync thresholds (cluster_pgsql_{query_rules,servers,users,variables}_diffs_before_sync) were gated by a single flag (checksum_pgsql_variables). This meant you could not independently control checksum computation per-module. By contrast, MySQL has per-module flags (checksum_mysql_query_rules, checksum_mysql_servers, checksum_mysql_users, checksum_mysql_variables). Fix: Add three new per-module admin variables: - admin-checksum_pgsql_query_rules - admin-checksum_pgsql_servers - admin-checksum_pgsql_users Each gates only its corresponding cluster_*_diffs_before_sync setter, mirroring the MySQL pattern. The existing admin-checksum_pgsql_variables now only gates cluster_pgsql_variables_diffs_before_sync. Disabling admin-checksum_pgsql_variables no longer cascades to zero out query_rules/servers/users sync thresholds. Files changed: - include/proxysql_admin.h: Add checksum_pgsql_query_rules, checksum_pgsql_servers, checksum_pgsql_users to checksum_variables struct. - lib/ProxySQL_Admin.cpp: Add variable names, constructor defaults, get/set handlers for new flags. Update diffs_before_sync guards to use per-module flags. Remove pgsql zeroing from MySQL checksum setters. Fix checksum_pgsql_variables setter to only affect pgsql variables sync threshold.	2 months ago
Rene Cannao	47758a468d	merge: resolve groups.json conflict with v3.0 ... Take v3.0's groups.json as base (legacy-gN naming, mysql84-gN groups, new test entries) and add test_cluster_sync_pgsql-t from this branch. Drop duplicate pgsql_admin_metacmds-t (already exists as pgsql-admin_metacmds-t in v3.0).	2 months ago
Rene Cannao	9b87260b70	fix: harden plugin init failures and command dispatch	2 months ago
Rene Cannao	fd5f029474	fix: expose plugin db handles during startup	2 months ago
Rene Cannao	cd15afdd1e	feat: support plugin-owned admin tables and commands	2 months ago
Rahim Kanji	93d0ddf477	fix: add pgsql_servers_ssl_params to tablenames vector and fix monitor fallback ... - Add pgsql_servers_ssl_params to pgsql_servers_tablenames so SAVE/LOAD PGSQL SERVERS TO/FROM DISK correctly persists and restores SSL params. - Fix monitor SSL fallback to use per-server params directly (empty fields omitted) instead of per-field fallback to globals, matching the backend connection behavior.	2 months ago
Rahim Kanji	516fa468ba	feat: add admin load/save/dump for pgsql_servers_ssl_params	2 months ago
René Cannaò	efedd93c92	Rename stats_global to stats_proxysql_global to avoid strstr substring collision ... stats_global is a substring of stats_mysql_global and stats_pgsql_global, causing the strstr-based dispatch to match incorrectly. Exclusion guards don't work either because a query referencing both tables would suppress the stats_proxysql_global refresh. Renaming to stats_proxysql_global follows the existing naming convention (stats_mysql_global, stats_pgsql_global) and eliminates the substring collision entirely.	2 months ago
René Cannaò	9850d9daeb	Fix stats_global substring collision and reload failure tracking ... 1. stats_global strstr match: add exclusion guards for stats_mysql_global and stats_pgsql_global to prevent spurious stats___global() calls and unnecessary ssl_mutex contention on every MySQL/PgSQL global stats query. 2. TLS reload failure tracking: set tls_last_load_ok=false and update tls_load_count/tls_last_load_timestamp on non-bootstrap reload failures. Previously, a failed PROXYSQL RELOAD TLS would leave stale SUCCESS status in stats_global.	2 months ago
copilot-swe-agent[bot]	e274862857	feat: introduce stats_global table and move TLS metrics out of stats_mysql_global; add TAP tests ... - Add STATS_SQLITE_TABLE_GLOBAL definition (stats_global table for non-MySQL/PgSQL metrics) - Register stats_global in Admin_Bootstrap.cpp - Add stats___global() implementation and declaration; document with Doxygen comment - Remove TLS_* variables from stats___mysql_global() - they were misplaced there - Move all TLS tracking metrics to stats___global() under ssl_mutex - Wire up stats_global query detection and refresh in GenericRefreshStatistics() - Add TAP test test_tls_stats-t.cpp: - Verifies stats_global contains all 6 TLS tracking variables - Checks value ranges and validity of each TLS variable - Verifies stats_tls_certificates has 2 rows (server + ca) with correct fields - Verifies TLS_Load_Count increments and TLS_Last_Load_Timestamp increases after PROXYSQL RELOAD TLS - Confirms TLS variables are absent from stats_mysql_global Co-authored-by: renecannao <3645227+renecannao@users.noreply.github.com>	2 months ago
copilot-swe-agent[bot]	4f88740b20	feat: add stats_tls_certificates table and TLS tracking metrics ... - Add STATS_SQLITE_TABLE_TLS_CERTIFICATES definition with cert_type, file_path, subject_cn, issuer_cn, serial_number, not_before, not_after, days_until_expiry, sha256_fingerprint, and loaded_at columns - Add TLS tracking fields to GloVars.global: tls_load_count, tls_last_load_timestamp, tls_last_load_ok, tls_cert_file, tls_ca_file, tls_key_file - Update ProxySQL_create_or_load_TLS() to populate tracking fields on success - Register stats_tls_certificates table in Admin_Bootstrap - Add stats___tls_certificates() function that reads certs from disk at query time, extracts metadata (CN, serial, validity dates, SHA-256 fingerprint, days_until_expiry) and populates the table - Add TLS variables to stats___mysql_global(): TLS_Load_Count, TLS_Last_Load_Timestamp, TLS_Last_Load_Result, TLS_Server_Cert_File, TLS_CA_Cert_File, TLS_Key_File - Wire up query detection and refresh in GenericRefreshStatistics() Co-authored-by: renecannao <3645227+renecannao@users.noreply.github.com>	2 months ago
Rene Cannao	859c89b30e	cluster: harden PostgreSQL cluster sync parity paths after real two-node validation ... This commit turns the current working branch follow-up into one coherent changeset after reviewing the PostgreSQL cluster sync implementation against real multi-node behavior instead of relying only on checksum presence and basic TAP accessibility checks. The main implementation change is that PostgreSQL cluster sync now updates the replica admin-memory tables in the places where the previous branch state was still weaker than the established MySQL behavior. In the PostgreSQL servers path, the replica now rewrites pgsql_servers, pgsql_replication_hostgroups, and pgsql_hostgroup_attributes before promoting the fetched state into runtime. That closes the most important parity gap that showed up during replica-side validation, because the cluster sync operation no longer leaves runtime state correct while replica admin-memory tables lag behind. In the PostgreSQL query rules path, the fetched query-rules and fast-routing resultsets are now hashed using the same converted representation that runtime loading consumes, then passed directly into load_pgsql_query_rules_to_runtime(), and finally written back from runtime into the replica admin-memory tables. In the PostgreSQL users path, the branch now relies on PostgreSQL-specific runtime checksum logic and persists accepted rows into the replica pgsql_users admin-memory table. This commit also fixes a real correctness bug that was discovered only after running a true two-node manual validation flow. The refactored module table in ProxySQL_Cluster::set_checksums() passed the full module names "pgsql_variables" and "ldap_variables" into pull_global_variables_from_peer(), but that shared helper only accepts the short dispatch keys "pgsql" and "ldap". In practice, changing a PostgreSQL variable on the primary caused the replica to abort on an assertion as soon as the cluster monitor attempted the variable pull. This commit restores the correct short dispatch keys so the PostgreSQL and LDAP variable sync paths actually execute instead of crashing. On the TAP side, the PostgreSQL cluster sync test is strengthened so it checks what the implementation is supposed to guarantee rather than what the earlier test happened to probe. The server sync test now mutates the real pgsql_servers admin table, then validates replica runtime_pgsql_servers, replica pgsql_servers, and disk.pgsql_servers when save_to_disk is enabled. The users sync test now validates runtime, main, and optional disk state on the replica. The query-rules sync test now validates both query rules and fast-routing rows across runtime, main, and optional disk persistence. The polling helpers were also hardened to support fresh replica admin sessions, because during local validation a long-lived admin session did not reliably observe synced rows even though fresh sessions and the manual harness did. The test now asserts the externally observable replica state seen by new admin clients. The documentation is updated to reflect the branch as it actually exists after this review pass. The PostgreSQL cluster sync status document now records the remaining non-CI TODO, the module-by-module behavior, the real two-node test plan, the outcomes for save_to_disk=false and save_to_disk=true, the variable sync dispatcher crash that was found and fixed locally, and the current MySQL parity assessment. This makes the branch easier to continue without having to reconstruct the reasoning from terminal logs or partial memory. Validation performed for this commit: - git diff --check - make build_src -j2 - PostgreSQL cluster-sync TAP two-node validation with save_to_disk=false - PostgreSQL cluster-sync TAP two-node validation with save_to_disk=true - manual two-node validation of pgsql_servers, pgsql_replication_hostgroups, pgsql_hostgroup_attributes, and a real PostgreSQL variable with save_to_disk=false - manual two-node validation of the same paths with save_to_disk=true Residual caveat intentionally documented but not treated as a blocker in this commit: - a long-lived replica admin session did not reliably observe synced PostgreSQL rows during polling, while fresh admin sessions and the manual validation harness did observe the expected state - the underlying cluster-sync apply path now validates correctly in real two-node runs, but the admin-session visibility quirk should still be understood better before calling the branch completely finished	2 months ago
Rene Cannao	c2f678a9de	feat: implement release tiers and feature guards for v3.0, v3.1, and v4.0 ... This commit introduces a structured release tier system controlled by environment variables, allowing three distinct ProxySQL versions to be built from the same codebase while maintaining a clear upgrade path. Key Changes: - Tier 1 (v3.0.x Stable): Default build including bug fixes and core enhancements. - Tier 2 (v3.1.x Innovative): Enabled via PROXYSQL31=1. Adds FFTO and TSDB. - Tier 3 (v4.0.x AI/MCP): Enabled via PROXYSQLGENAI=1. Adds the AI stack. Implementation Details: - Makefile: Implemented dynamic versioning logic that increments the minor version for v3.1 and the major version for v4.0. Added tier documentation. - Feature Guards: Introduced PROXYSQLFFTO and PROXYSQLTSDB guards across ProxySQL_Admin, ProxySQL_Statistics, REST API, and HTTP server. - FFTO: Guarded Fast Forward Traffic Observer in MySQL and PgSQL sessions, including thread-local variables and configuration parameters. - TSDB: Guarded the entire Time Series Database subsystem, including SQLite table definitions, background loops, and REST API endpoints. - Build Fixes: * Fixed 'proxy_debug_func' linkage by adding extern "C" and moving the definition outside of DEBUG guards to ensure non-debug builds link correctly. * Corrected misplaced/nested DEBUG and PROXYSQLFFTO guards. * Verified build stability for all tiers (unset, PROXYSQL31=1, PROXYSQLGENAI=1).	3 months ago
Wazir Ahmed	5421a382ab	fix: Load TSDB variables from disk during startup ... - TSDB variables stored on disk were not being loaded into runtime during startup. Add the missing `flush_tsdb_variables___database_to_runtime()` and invoke it during init. - Inline the existing load/save wrappers in the header as thin delegations to the underlying flush functions Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Rene Cannao	5c7e616f9b	PR5297: resolve remaining actionable review findings for PGSQL checksum sync and TAP assertions ... This commit addresses the review findings that are still reproducible on top of fix/postgresql-cluster-sync_2 after the v3.0 merge, while leaving already-resolved or no-longer-applicable reports untouched. Changes included: - Admin_FlushVariables: restore active PGSQL checksum generation path in flush_pgsql_variables___database_to_runtime(). - Remove the stale commented-out checksum block. - After releasing GloPTH write lock, compute runtime PGSQL vars snapshot and call flush_GENERIC_variables__checksum__database_to_runtime("pgsql", checksum, epoch) under checksum mutex, matching the mysql/admin pattern. - ProxySQL_Admin::set_variable(checksum_pgsql_variables): when disabling, reset all PGSQL *_diffs_before_sync thresholds (variables/query_rules/servers/users) in both admin variables and GloProxyCluster state, and update warning text accordingly. - TAP test hardening in test_cluster_sync_pgsql-t: - Ensure each checksum-loop failure path emits an explicit failing ok(false, ...) before continue, preventing TAP plan/accounting mismatches. - Simplify pgsql_servers_v2 test tuple shape by removing the unused int field and using the comment field consistently. Validation performed: - make -C lib -j4 (pass) - make -C test/tap/tests test_cluster_sync_pgsql-t -j4 (pass) Note: - The previously reported double wrunlock() in flush_pgsql_variables__from_memory_to_disk() is not present in current HEAD; function has a single wrlock/wrunlock pair.	3 months ago
Rene Cannao	6dbd0e9cb1	Merge branch 'v3.0' into fix/postgresql-cluster-sync_2 ... Resolve merge conflicts in: - lib/ProxySQL_Cluster.cpp - test/tap/tap/Makefile Conflict resolution details: - Kept PostgreSQL/MySQL cluster-sync updates from this branch (SQLQueries delete constants and pgsql sync flow changes) while adopting upstream SQLite prepare_v2 RAII usage where introduced in v3.0 conflict regions. - Combined TAP archive fixes from this branch (archive recreation via rm -f to prevent stale objects) with upstream v3.0 mcp_client.o additions to libtap archives. Validation performed after conflict resolution: - make -C test/tap/tap debug -j4 (success) - make -C lib -j4 (fails in current workspace due missing sqlite-vec.h/vec.o dependency in broader v3.0 content, not conflict markers) - make -C test/tap/tests admin-listen_on_unix-t -j4 (fails in current workspace due missing deps/sqlite3/sqlite3/vec.o, same dependency issue) No conflict markers remain.	3 months ago
Rene Cannao	4cdf61cbae	cluster: complete PostgreSQL servers sync flow and fix TAP libtap stale-archive linker failures ... This commit finalizes the remaining PostgreSQL cluster-sync implementation work and includes a targeted TAP build fix that addresses the unresolved perform_simple_get() linker error seen while building debug TAP tests. PostgreSQL cluster sync changes: - Add conversion support to transform fetched pgsql resultsets into SQLite3_result objects suitable for admin/runtime loaders. - Rework runtime pgsql server pull path to: - use peer-selected checksums, - fetch runtime rows from the selected peer, - commit runtime state through PgHGM in only-runtime mode, - persist runtime state and keep locking semantics around pgsql server updates. - Rework pgsql_servers_v2 pull path to fetch and validate all related datasets together: - pgsql_servers_v2, - pgsql_replication_hostgroups, - pgsql_hostgroup_attributes, - runtime_pgsql_servers (when applicable). - Ensure checksum validation includes all fetched tables and rejects partial/inconsistent peer payloads. - Load the actual incoming converted datasets into runtime instead of placeholder/empty local resultsets. Test updates: - Extend test_cluster_sync_pgsql-t with additional verification for synchronized pgsql data and optional replica/runtime sync checks. - Adjust TAP planning and resultset handling so all executed queries are validated and cleaned up consistently. TAP build/linker fix: - Update test/tap/tap/Makefile archive targets (libtap_mariadb.a/libtap_mysql57.a/libtap_mysql8.a) to remove existing archives before recreating them. - Add Makefile as a prerequisite for those archive targets so this behavior is applied on rebuild after pulling. - This prevents stale archive members (for example old noise_utils*.o objects) from being retained and reintroduced into libtap.so, which was causing undefined references to an obsolete 4-argument perform_simple_get() symbol. Validation performed in this branch after these changes: - make -C lib -j4 - make -C src -j4 - make -C test/tap/tests test_cluster_sync_pgsql-t -j4 - make -C test/tap/tests admin-listen_on_unix-t -j4	3 months ago
Rene Cannao	e2ebc0c449	Merge branch 'v4.0-mcp-stats2' into v3.0-unified-large-pr	3 months ago
René Cannaò	9c6945fae8	Merge branch 'v3.0' into v3.0-tsdb-feature ... Signed-off-by: René Cannaò <rene@proxysql.com>	3 months ago
Rene Cannao	70b61a86e3	MCP stats: implement in-memory show_queries Top-K path ... Replace stats table reads in show_queries with Query Processor in-memory aggregation to avoid stale/empty admin-db snapshots.\n\nIntroduce Top-K digest filtering primitives and wire them into Stats_Tool_Handler so filtering/sorting/limit are executed against live digest memory. Add runtime MCP variable mcp-stats_show_queries_max_rows with validation, defaulting, and configuration plumbing, plus cap metadata in the tool response.\n\nEnforce configurable hard cap semantics while preserving deterministic ordering and pagination behavior for MCP clients. Add internal DEBUG validator path (PROXYSQLTEST 56) in admin tests to stress generated digest data and validate the Top-K/filter pipeline against in-memory state.\n\nKeep the validator callable from startup/testing workflows without depending on MCP linkage. Add TAP coverage in test/tap/tests/mcp_show_queries_topk-t.cpp for:\n- MCP endpoint setup and reachability\n- digest generation + optional DEBUG internal validator\n- payload validation and cap metadata assertions\n- descending count order validation\n- match_digest_text filter verification\n\nMake payload parsing backward-compatible in TAP (direct tool result object and legacy wrapped success/result payloads).	3 months ago
René Cannaò	4a6809b60f	Merge branch 'v3.0' into v3.0-tsdb-feature ... Signed-off-by: René Cannaò <rene@proxysql.com>	3 months ago
Rene Cannao	ef397f9f1d	Fix TSDB variable patterns and backend monitor runtime source	3 months ago
Rene Cannao	b4e85f1799	Expose PostgreSQL eventslog metrics in stats and Prometheus	3 months ago
Rene Cannao	b100235fbd	Add PGSQL eventslog dump commands and periodic disk sync scheduling	3 months ago
Rene Cannao	2a46c239be	Add PostgreSQL advanced eventslog schema and variable scaffolding	3 months ago
Rene Cannao	700459f67b	tsdb: address AI code review feedback, fix memory safety and logic bugs ... Summary of changes: - Fixed allocator mismatches (sqlite3_free vs free) across multiple files. - Converted manual SQL construction to prepared statements for TSDB variables. - Resolved memory leaks in statistics and monitoring loops. - Fixed critical 'LOAD TSDB VARIABLES FROM CONFIG' command mismatch. - Improved command parsing robustness using string searching. - Standardized SQL LIKE patterns with escaped wildcards (\%). - Switched to data-driven variable management in ProxySQL_Statistics. - Improved web dashboard security with URL encoding and const correctness. - Clarified documentation regarding configuration lifecycle and schema names. - Updated TAP tests for better reliability and correct persistence verification.	3 months ago
Rene Cannao	b8c6b4f573	tsdb: address code review comments and improve stability ... - Refactored variable flushing to use prepared statements for safety. - Replaced brittle command parsing with robust string searching. - Extracted TSDB dashboard HTML/JS to separate source file for maintainability. - Added NULL guards for GloProxyStats in Admin and REST API. - Implemented batched metric insertions using transactions for performance. - Refactored backend probing to use non-blocking connect() with timeout. - Fixed potential integer overflows in timer calculations. - Improved error handling and fixed memory leaks in statistics reporting. - Updated TAP tests to verify UI and REST API functionality.	3 months ago
Rene Cannao	6a788e48c4	mcp: make /mcp/query self-healing when targets/backends appear after startup ... Problem addressed: - MCP query endpoint could stay unusable with 'Tool Handler not initialized' after restart/reload flows. - This was triggered when Query_Tool_Handler could not build an executable pool at init time (for example profiles loaded before ONLINE servers), leaving runtime commands like 'LOAD MCP QUERY RULES FROM MEMORY' blocked behind a NULL query tool handler. - Users had to manually toggle MCP enablement to recover, which is the opposite of expected self-healing behavior. What this commit changes: 1) Query_Tool_Handler pool init is now resilient and idempotent - init_connection_pool() now starts with close() so reinitialization fully resets stale mysql/pgsql pool state before rebuilding from runtime profile + server tables. - If no executable targets are available, init_connection_pool() now returns success with a warning instead of hard failure. This allows the query tool handler (and /mcp/query endpoint) to stay initialized even before backends are ready. 2) Lazy auto-rebuild on first query usage - get_connection() and get_pgsql_connection() now: - refresh target registry before resolution, - attempt to use an existing pooled connection, - if unavailable, trigger a full pool rebuild (init_connection_pool()) and retry once. - This provides automatic recovery when hostgroups/servers/profiles are loaded or changed after MCP startup, without requiring manual MCP disable/enable. 3) Admin runtime rule load path attempts MCP recovery - load_mcp_query_rules_to_runtime() now detects NULL query tool handler and calls load_mcp_server() once before failing. - This turns a hard, immediate admin error into a self-recovery attempt consistent with MCP runtime semantics. Behavioral impact: - /mcp/query endpoint remains online even when there are temporarily zero executable targets. - As soon as compatible runtime targets/backends exist, run_sql_readonly/explain_sql can recover automatically on demand. - LOAD MCP QUERY RULES TO/FROM RUNTIME no longer fails immediately on first NULL handler condition; it retries after MCP server recovery. Validation performed: - Recompiled modified objects successfully: - lib/obj/Query_Tool_Handler.oo - lib/obj/ProxySQL_Admin.oo - Full TAP runtime test execution is not possible in this sandbox due blocked local TCP socket creation; validation should be run in the normal test environment where ProxySQL/MySQL/PGSQL are reachable.	3 months ago
Rene Cannao	998bd82387	MCP TAP startup: fix tool-handler initialization order, improve MCP PROFILES observability, and seed monitor users ... This change fixes recurring MCP TAP failures where `/mcp/query` returned: Tool Handler not initialized for endpoint: query and where backend monitor auth failures flooded logs. Problem summary - MCP server startup can occur before runtime target/auth profiles and backend server mappings are loaded. - If that happens, Query_Tool_Handler initialization sees no executable targets and remains NULL. - MCP endpoint resources bind the handler pointer at creation time, so a NULL query handler at startup breaks `/mcp/query` until server restart. Code changes 1) Add explicit admin command logging for MCP PROFILES commands - Added `Received <command>` logging in the MCP PROFILES command block, matching behavior of other admin command handlers. - File: `lib/Admin_Handler.cpp` 2) Trigger MCP server refresh after `LOAD MCP PROFILES TO RUNTIME` - After copying profiles into runtime and rebuilding target/auth map, call `ProxySQL_Admin::load_mcp_server()`. - This allows MCP to self-heal when profiles become available after initial startup. - File: `lib/Admin_Handler.cpp` 3) Restart MCP server when query handler is missing - Extended `ProxySQL_Admin::load_mcp_server()` restart checks to include: - running server + `query_tool_handler == NULL` - Restart reason now includes tool handler initialization mismatch. - File: `lib/ProxySQL_Admin.cpp` 4) Fix TAP configurator load order to avoid early MCP startup - Reordered `test/tap/tests/mcp_rules_testing/configure_mcp.sh` runtime sequence: - `LOAD MYSQL SERVERS TO RUNTIME` - `LOAD PGSQL SERVERS TO RUNTIME` (best effort) - `LOAD MCP PROFILES TO RUNTIME` - `LOAD MCP VARIABLES TO RUNTIME` (last) - This ensures MCP starts only after routing/auth context is present. 5) Seed monitor credentials in AI local infra pre-hook - Added backend user/role creation for default monitor credentials `monitor/monitor`: - MySQL: create user + monitor-relevant grants - PostgreSQL: create role + `pg_monitor` + DB connect grants - Reduces monitor auth noise in local AI TAP dockerized setup. - File: `test/tap/groups/ai/pre-proxysql.bash` 6) Mark new TAP phase scripts executable - `test_phase10_eval_explain.sh` - `test_phase11_pgsql_target.sh` Expected outcome - MCP query endpoint no longer stays stuck with an uninitialized tool handler after TAP configuration. - MCP query-rules admin commands stop failing due to missing Query_Tool_Handler. - MCP profile command flow is visible in logs for easier debugging. - Local AI TAP infra no longer emits continuous monitor authentication failures for default monitor credentials.	3 months ago
Rene Cannao	af0411bd46	MCP: add target-aware rules/stats tests, explain_sql rule coverage, and AI local docker TAP infra ... This commit completes end-to-end MCP query-rules validation for multi-target routing and introduces a self-contained TAP infra for the `ai` group that can run both in Jenkins and manually. Main MCP/runtime changes: - Extended MCP query-rule matching context to include both `target_id` and resolved backend `username`. - Added `target_id` column to `mcp_query_rules` and `runtime_mcp_query_rules` table definitions. - Extended `stats_mcp_query_rules` to include `username` and `target_id` alongside `rule_id` and `hits`. - Updated load/save/runtime refresh paths to persist and rehydrate the expanded MCP rule schema. - Wired the rule engine into `explain_sql` so MCP rules apply consistently across `run_sql_readonly` and `explain_sql`. - Included startup-order fix in `src/main.cpp` to initialize MCP/GenAI thread handlers early, preventing startup crashes in PROXYSQLGENAI builds. Test coverage changes: - Updated existing MCP TAP phases to assert target-aware and username-aware behavior: - `test_phase4_stats.sh` - `test_phase6_eval_block.sh` - Added new MCP TAP phases: - `test_phase10_eval_explain.sh` (rule engine coverage for `explain_sql`) - `test_phase11_pgsql_target.sh` (pgsql target routing/rule/stats coverage; graceful skip if no pgsql target is configured) - Updated `test_mcp_query_rules-t.sh` runner to execute new phases. AI group isolated infra (manual + CI compatible): - Added `test/tap/groups/ai/docker-compose.yml` with MySQL 9.0 and PostgreSQL 16 backends. - Added lifecycle scripts: - `docker-compose-init.bash` - `docker-compose-destroy.bash` - `pre-proxysql.bash` - `post-proxysql.bash` - Extended `test/tap/groups/ai/env.sh` with local default ports/credentials/target IDs used by MCP TAP tests. - Added `test/tap/groups/ai/README.md` documenting manual execution flow outside Jenkins. Outcome: - MCP tests now validate routing-aware rule enforcement and stats attribution for both MySQL and PostgreSQL targets. - The `ai` TAP group can be run with an isolated local backend stack without relying on external Jenkins infra repositories.	3 months ago
Rene Cannao	013864b36f	MCP: introduce profile-based target/auth routing and unified LOAD/SAVE MCP PROFILES commands ... This commit introduces the core MCP routing architecture that decouples client-facing target selection from backend connection details. Architecture introduced: - New MCP auth profile table to store backend credentials/policy (`mcp_auth_profiles`). - New MCP target profile table to map opaque `target_id` -> protocol/hostgroup/auth profile (`mcp_target_profiles`). - Runtime mirrors for both profile tables. Behavioral changes: - MCP query tools resolve execution context by `target_id` and route internally by protocol + hostgroup. - Client no longer needs backend-specific connection details; only logical target identifiers. - Unified admin commands for profile lifecycle management: - `LOAD MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` - `SAVE MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` Design goals: - Support multi-backend MCP execution (MySQL + PostgreSQL) behind one endpoint. - Provide server-side credential management and cleaner operational model. - Replace legacy one-backend POC configuration with scalable target/auth abstractions. Result: - MCP routing is now dynamic, protocol-agnostic at the client layer, and fully managed through profile tables and unified load/save commands.	3 months ago
Rene Cannao	5f5b7d604f	tsdb: improve administrative interface and command support ... - Move 'stats_tsdb' to 'main' schema for better visibility in 'SHOW TSDB STATUS'. - Implement 'PROXYSQL TSDB DOWNSAMPLE' command to trigger manual metrics aggregation. - Ensure 'SHOW TSDB STATUS' triggers a statistics refresh before returning data. - Integrate 'tsdb-' variables into 'runtime_global_variables' refresh logic. - Update TAP tests to verify new commands and improve troubleshooting output.	3 months ago
Rene Cannao	6587bfdd40	tsdb: trigger stats refresh on SHOW TSDB STATUS	3 months ago
Rene Cannao	2d51aeff64	tsdb: correctly implement SHOW TSDB commands and variable initialization	3 months ago
Rene Cannao	6404d7b365	tsdb: implement SHOW TSDB VARIABLES and SHOW TSDB STATUS	3 months ago
Rene Cannao	4e0f94a345	tsdb: fix compilation error in ProxySQL_Admin.cpp	3 months ago
Rene Cannao	a236a13787	tsdb: dedicated module for TSDB variables with tsdb- prefix and optimized background tasks	3 months ago
Rene Cannao	da877c378b	tsdb: align admin variable lifecycle, full prometheus ingestion, docs and tap coverage	3 months ago
Rene Cannao	200fdecb8d	Apply AI agent review fixes to PR #5374 ... Critical Fixes: - Fix NULL pointer dereferences in MySQL_Logger.cpp: * Add null checks for myds->myconn->parent in log_audit_entry() * Fix cleanup code in log_request() and log_audit_entry() * Change sprintf to snprintf for buffer safety - Remove redundant pthread_rwlock_init in debug.cpp: * filters_rwlock already initialized with PTHREAD_RWLOCK_INITIALIZER * Re-initialization caused undefined behavior per POSIX Major Improvements: - Add bounded wait helper wait_for_glo_mth() in proxysql_utils.h: * Replaces indefinite while loops with 10s timeout * Applied to 26 locations across 4 files Build System: - Require C++17 in Makefile (remove C++11 fallback): * Code uses C++17 structured bindings extensively * Cleanest solution vs adding fallback guards to 27+ locations Compilation fixes: - Add unistd.h include for usleep - Remove conflicting GloMyMon extern declaration Files changed: - Makefile - include/proxysql_utils.h - lib/MySQL_Logger.cpp - lib/debug.cpp - lib/MySQL_Monitor.cpp - lib/ProxySQL_Admin.cpp - lib/ClickHouse_Server.cpp - src/SQLite3_Server.cpp	3 months ago
Rene Cannao	3c85c593cf	Add GloMTH initialization wait to all monitor and server threads ... Fix race condition where threads were accessing GloMTH before it was fully initialized, causing Valgrind errors in get_variable_int() and related functions. Added wait loop pattern (while GloMTH==NULL { usleep(50000); }) followed by usleep(100000) to ensure GloMTH is fully initialized before use: In lib/MySQL_Monitor.cpp: - monitor_connect(), monitor_ping(), monitor_read_only() - monitor_group_replication(), monitor_galera(), monitor_replication_lag() - monitor_AWS_Aurora_thread_HG(), monitor_aws_aurora() - init_mysql_thread_struct() In src/SQLite3_Server.cpp: - SQLite3_Server_session_handler In lib/ProxySQL_Admin.cpp: - admin_session_handler In lib/ClickHouse_Server.cpp: - ClickHouse_Server_session_handler The pattern ensures threads wait for GloMTH initialization during startup, then check for NULL during shutdown to exit cleanly.	3 months ago