proxysql

Commit Graph

Author	SHA1	Message	Date
René Cannaò	669ab91494	Merge pull request #5594 from sysown/v3.0-lint lint + static analysis: clang-tidy and cppcheck fixes	2 months ago
Rene Cannao	f1826cf206	lint: fix reserved identifier in member variables Rename __active -> active_, __frontend -> frontend_, __backend -> backend_ in authentication and hostgroups classes to comply with C++ reserved identifier rules. Includes updates to: - MySQL_Authentication.hpp - ClickHouse_Authentication.hpp - PgSQL_Authentication.h - MySQL_HostGroups_Manager.h - Base_HostGroups_Manager.h - Corresponding .cpp files	2 months ago
Rene Cannao	ac0f9304b0	fix: address code review findings for pgsql cluster sync 1. Remove dead declarations and implementations: PullOperationConfig, pull_from_peer_unified, fetch_query_with_metrics, compute__checksum, ScopedChar wrappers, and all safe_* memory helpers (-690 lines) 2. Document mutex sharing: pgsql pull functions intentionally reuse MySQL mutexes since pulls are sequential in the monitoring thread 3. Fix variable shadowing: rename local 'error' to 'exec_error' in Admin_Handler.cpp pgsql_servers_v2 fallback to avoid shadowing the outer error variable 4. Fix misleading debug message in process_component_checksum: the else branch fires when peer checksum is unchanged, not different 5. Fix indentation in ProxySQL_Admin.cpp (3 lines with double-tab) and Admin_FlushVariables.cpp (NULL-guard block with extra indent)	2 months ago
Rene Cannao	80ff2242e6	merge: resolve v3.0 conflict and add pgsql_servers_ssl_params cluster sync v3.0 added pgsql_servers_ssl_params table. Integrate into pgsql cluster sync path: - Add CLUSTER_QUERY_PGSQL_SERVERS_SSL_PARAMS to Admin_Handler interception block alongside other pgsql cluster queries - Add ssl_params fetch query to pull_pgsql_servers_v2_from_peer - Add update_pgsql_servers_ssl_params() for applying synced data - Update convert_pgsql_servers_resultsets to handle 5 result sets - Include ssl_params in v2 checksum computation (4 tables)	2 months ago
René Cannaò	93eabfc7bc	fix: address code review issues in PostgreSQL cluster sync - Fix fetch_and_store incrementing success counter on failure - Fix sqlite3_stmt leak in pull_pgsql_variables_from_peer (use RAII) - Convert get_peer_to_sync_pgsql_query_rules and _servers_v2 to unified framework delegates (fixes atomic data race on .load()) - Merge dual scan loops in set_checksums to respect enabled_check - Remove spurious "deprecated" label from new checksum_pgsql_variables - Replace sprintf with snprintf in pgsql diffs_before_sync getters - Fix indentation on cluster_pgsql_variables blocks - Fix TAP plan mismatch: replace MYSQL_QUERY with inline mysql_query - Fix trivially-true table accessibility assertions - Add mysql_close on failed connect in test - Move test_cluster_sync_pgsql-t to legacy-g5 group	2 months ago
Rahim Kanji	e68f8bb74a	refactor: simplify pgsql_servers_ssl_params schema and pre-parse TLS range - Drop ssl_capath and ssl_cipher columns (no libpq equivalents) - Pre-parse ssl_protocol_version_range into ssl_min/max_protocol_version at construction time instead of on every backend connection - Fix Servers_SslParams 3-arg delegating constructor - Update tests, docs, and all consumers (Connection, Monitor, Kill, Admin)	2 months ago
René Cannaò	8a61232517	fix: decouple MySQL and PostgreSQL checksum/sync controls Two bugs in the PostgreSQL cluster sync admin variables: 1. Cross-protocol coupling (critical): Disabling a MySQL checksum flag (e.g. SET admin-checksum_mysql_query_rules='false') incorrectly zeroed out the corresponding PostgreSQL cluster sync threshold (cluster_pgsql_query_rules_diffs_before_sync). An operator disabling MySQL checksums would unexpectedly halt PostgreSQL cluster sync. Fix: MySQL checksum setters now only affect MySQL sync thresholds. The warning messages no longer mention pgsql side effects. 2. Single-gate for all pgsql modules (operational limitation): All four PostgreSQL cluster sync thresholds (cluster_pgsql_{query_rules,servers,users,variables}_diffs_before_sync) were gated by a single flag (checksum_pgsql_variables). This meant you could not independently control checksum computation per-module. By contrast, MySQL has per-module flags (checksum_mysql_query_rules, checksum_mysql_servers, checksum_mysql_users, checksum_mysql_variables). Fix: Add three new per-module admin variables: - admin-checksum_pgsql_query_rules - admin-checksum_pgsql_servers - admin-checksum_pgsql_users Each gates only its corresponding cluster_*_diffs_before_sync setter, mirroring the MySQL pattern. The existing admin-checksum_pgsql_variables now only gates cluster_pgsql_variables_diffs_before_sync. Disabling admin-checksum_pgsql_variables no longer cascades to zero out query_rules/servers/users sync thresholds. Files changed: - include/proxysql_admin.h: Add checksum_pgsql_query_rules, checksum_pgsql_servers, checksum_pgsql_users to checksum_variables struct. - lib/ProxySQL_Admin.cpp: Add variable names, constructor defaults, get/set handlers for new flags. Update diffs_before_sync guards to use per-module flags. Remove pgsql zeroing from MySQL checksum setters. Fix checksum_pgsql_variables setter to only affect pgsql variables sync threshold.	2 months ago
Rene Cannao	47758a468d	merge: resolve groups.json conflict with v3.0 Take v3.0's groups.json as base (legacy-gN naming, mysql84-gN groups, new test entries) and add test_cluster_sync_pgsql-t from this branch. Drop duplicate pgsql_admin_metacmds-t (already exists as pgsql-admin_metacmds-t in v3.0).	2 months ago
Rahim Kanji	93d0ddf477	fix: add pgsql_servers_ssl_params to tablenames vector and fix monitor fallback - Add pgsql_servers_ssl_params to pgsql_servers_tablenames so SAVE/LOAD PGSQL SERVERS TO/FROM DISK correctly persists and restores SSL params. - Fix monitor SSL fallback to use per-server params directly (empty fields omitted) instead of per-field fallback to globals, matching the backend connection behavior.	2 months ago
Rahim Kanji	516fa468ba	feat: add admin load/save/dump for pgsql_servers_ssl_params	2 months ago
René Cannaò	efedd93c92	Rename stats_global to stats_proxysql_global to avoid strstr substring collision stats_global is a substring of stats_mysql_global and stats_pgsql_global, causing the strstr-based dispatch to match incorrectly. Exclusion guards don't work either because a query referencing both tables would suppress the stats_proxysql_global refresh. Renaming to stats_proxysql_global follows the existing naming convention (stats_mysql_global, stats_pgsql_global) and eliminates the substring collision entirely.	2 months ago
René Cannaò	9850d9daeb	Fix stats_global substring collision and reload failure tracking 1. stats_global strstr match: add exclusion guards for stats_mysql_global and stats_pgsql_global to prevent spurious stats___global() calls and unnecessary ssl_mutex contention on every MySQL/PgSQL global stats query. 2. TLS reload failure tracking: set tls_last_load_ok=false and update tls_load_count/tls_last_load_timestamp on non-bootstrap reload failures. Previously, a failed PROXYSQL RELOAD TLS would leave stale SUCCESS status in stats_global.	2 months ago
copilot-swe-agent[bot]	e274862857	feat: introduce stats_global table and move TLS metrics out of stats_mysql_global; add TAP tests - Add STATS_SQLITE_TABLE_GLOBAL definition (stats_global table for non-MySQL/PgSQL metrics) - Register stats_global in Admin_Bootstrap.cpp - Add stats___global() implementation and declaration; document with Doxygen comment - Remove TLS_* variables from stats___mysql_global() - they were misplaced there - Move all TLS tracking metrics to stats___global() under ssl_mutex - Wire up stats_global query detection and refresh in GenericRefreshStatistics() - Add TAP test test_tls_stats-t.cpp: - Verifies stats_global contains all 6 TLS tracking variables - Checks value ranges and validity of each TLS variable - Verifies stats_tls_certificates has 2 rows (server + ca) with correct fields - Verifies TLS_Load_Count increments and TLS_Last_Load_Timestamp increases after PROXYSQL RELOAD TLS - Confirms TLS variables are absent from stats_mysql_global Co-authored-by: renecannao <3645227+renecannao@users.noreply.github.com>	2 months ago
copilot-swe-agent[bot]	4f88740b20	feat: add stats_tls_certificates table and TLS tracking metrics - Add STATS_SQLITE_TABLE_TLS_CERTIFICATES definition with cert_type, file_path, subject_cn, issuer_cn, serial_number, not_before, not_after, days_until_expiry, sha256_fingerprint, and loaded_at columns - Add TLS tracking fields to GloVars.global: tls_load_count, tls_last_load_timestamp, tls_last_load_ok, tls_cert_file, tls_ca_file, tls_key_file - Update ProxySQL_create_or_load_TLS() to populate tracking fields on success - Register stats_tls_certificates table in Admin_Bootstrap - Add stats___tls_certificates() function that reads certs from disk at query time, extracts metadata (CN, serial, validity dates, SHA-256 fingerprint, days_until_expiry) and populates the table - Add TLS variables to stats___mysql_global(): TLS_Load_Count, TLS_Last_Load_Timestamp, TLS_Last_Load_Result, TLS_Server_Cert_File, TLS_CA_Cert_File, TLS_Key_File - Wire up query detection and refresh in GenericRefreshStatistics() Co-authored-by: renecannao <3645227+renecannao@users.noreply.github.com>	2 months ago
Rene Cannao	859c89b30e	cluster: harden PostgreSQL cluster sync parity paths after real two-node validation This commit turns the current working branch follow-up into one coherent changeset after reviewing the PostgreSQL cluster sync implementation against real multi-node behavior instead of relying only on checksum presence and basic TAP accessibility checks. The main implementation change is that PostgreSQL cluster sync now updates the replica admin-memory tables in the places where the previous branch state was still weaker than the established MySQL behavior. In the PostgreSQL servers path, the replica now rewrites pgsql_servers, pgsql_replication_hostgroups, and pgsql_hostgroup_attributes before promoting the fetched state into runtime. That closes the most important parity gap that showed up during replica-side validation, because the cluster sync operation no longer leaves runtime state correct while replica admin-memory tables lag behind. In the PostgreSQL query rules path, the fetched query-rules and fast-routing resultsets are now hashed using the same converted representation that runtime loading consumes, then passed directly into load_pgsql_query_rules_to_runtime(), and finally written back from runtime into the replica admin-memory tables. In the PostgreSQL users path, the branch now relies on PostgreSQL-specific runtime checksum logic and persists accepted rows into the replica pgsql_users admin-memory table. This commit also fixes a real correctness bug that was discovered only after running a true two-node manual validation flow. The refactored module table in ProxySQL_Cluster::set_checksums() passed the full module names "pgsql_variables" and "ldap_variables" into pull_global_variables_from_peer(), but that shared helper only accepts the short dispatch keys "pgsql" and "ldap". In practice, changing a PostgreSQL variable on the primary caused the replica to abort on an assertion as soon as the cluster monitor attempted the variable pull. This commit restores the correct short dispatch keys so the PostgreSQL and LDAP variable sync paths actually execute instead of crashing. On the TAP side, the PostgreSQL cluster sync test is strengthened so it checks what the implementation is supposed to guarantee rather than what the earlier test happened to probe. The server sync test now mutates the real pgsql_servers admin table, then validates replica runtime_pgsql_servers, replica pgsql_servers, and disk.pgsql_servers when save_to_disk is enabled. The users sync test now validates runtime, main, and optional disk state on the replica. The query-rules sync test now validates both query rules and fast-routing rows across runtime, main, and optional disk persistence. The polling helpers were also hardened to support fresh replica admin sessions, because during local validation a long-lived admin session did not reliably observe synced rows even though fresh sessions and the manual harness did. The test now asserts the externally observable replica state seen by new admin clients. The documentation is updated to reflect the branch as it actually exists after this review pass. The PostgreSQL cluster sync status document now records the remaining non-CI TODO, the module-by-module behavior, the real two-node test plan, the outcomes for save_to_disk=false and save_to_disk=true, the variable sync dispatcher crash that was found and fixed locally, and the current MySQL parity assessment. This makes the branch easier to continue without having to reconstruct the reasoning from terminal logs or partial memory. Validation performed for this commit: - git diff --check - make build_src -j2 - PostgreSQL cluster-sync TAP two-node validation with save_to_disk=false - PostgreSQL cluster-sync TAP two-node validation with save_to_disk=true - manual two-node validation of pgsql_servers, pgsql_replication_hostgroups, pgsql_hostgroup_attributes, and a real PostgreSQL variable with save_to_disk=false - manual two-node validation of the same paths with save_to_disk=true Residual caveat intentionally documented but not treated as a blocker in this commit: - a long-lived replica admin session did not reliably observe synced PostgreSQL rows during polling, while fresh admin sessions and the manual validation harness did observe the expected state - the underlying cluster-sync apply path now validates correctly in real two-node runs, but the admin-session visibility quirk should still be understood better before calling the branch completely finished	2 months ago
Rene Cannao	c2f678a9de	feat: implement release tiers and feature guards for v3.0, v3.1, and v4.0 This commit introduces a structured release tier system controlled by environment variables, allowing three distinct ProxySQL versions to be built from the same codebase while maintaining a clear upgrade path. Key Changes: - Tier 1 (v3.0.x Stable): Default build including bug fixes and core enhancements. - Tier 2 (v3.1.x Innovative): Enabled via PROXYSQL31=1. Adds FFTO and TSDB. - Tier 3 (v4.0.x AI/MCP): Enabled via PROXYSQLGENAI=1. Adds the AI stack. Implementation Details: - Makefile: Implemented dynamic versioning logic that increments the minor version for v3.1 and the major version for v4.0. Added tier documentation. - Feature Guards: Introduced PROXYSQLFFTO and PROXYSQLTSDB guards across ProxySQL_Admin, ProxySQL_Statistics, REST API, and HTTP server. - FFTO: Guarded Fast Forward Traffic Observer in MySQL and PgSQL sessions, including thread-local variables and configuration parameters. - TSDB: Guarded the entire Time Series Database subsystem, including SQLite table definitions, background loops, and REST API endpoints. - Build Fixes: * Fixed 'proxy_debug_func' linkage by adding extern "C" and moving the definition outside of DEBUG guards to ensure non-debug builds link correctly. * Corrected misplaced/nested DEBUG and PROXYSQLFFTO guards. * Verified build stability for all tiers (unset, PROXYSQL31=1, PROXYSQLGENAI=1).	3 months ago
Wazir Ahmed	5421a382ab	fix: Load TSDB variables from disk during startup - TSDB variables stored on disk were not being loaded into runtime during startup. Add the missing `flush_tsdb_variables___database_to_runtime()` and invoke it during init. - Inline the existing load/save wrappers in the header as thin delegations to the underlying flush functions Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	3 months ago
Rene Cannao	5c7e616f9b	PR5297: resolve remaining actionable review findings for PGSQL checksum sync and TAP assertions This commit addresses the review findings that are still reproducible on top of fix/postgresql-cluster-sync_2 after the v3.0 merge, while leaving already-resolved or no-longer-applicable reports untouched. Changes included: - Admin_FlushVariables: restore active PGSQL checksum generation path in flush_pgsql_variables___database_to_runtime(). - Remove the stale commented-out checksum block. - After releasing GloPTH write lock, compute runtime PGSQL vars snapshot and call flush_GENERIC_variables__checksum__database_to_runtime("pgsql", checksum, epoch) under checksum mutex, matching the mysql/admin pattern. - ProxySQL_Admin::set_variable(checksum_pgsql_variables): when disabling, reset all PGSQL *_diffs_before_sync thresholds (variables/query_rules/servers/users) in both admin variables and GloProxyCluster state, and update warning text accordingly. - TAP test hardening in test_cluster_sync_pgsql-t: - Ensure each checksum-loop failure path emits an explicit failing ok(false, ...) before continue, preventing TAP plan/accounting mismatches. - Simplify pgsql_servers_v2 test tuple shape by removing the unused int field and using the comment field consistently. Validation performed: - make -C lib -j4 (pass) - make -C test/tap/tests test_cluster_sync_pgsql-t -j4 (pass) Note: - The previously reported double wrunlock() in flush_pgsql_variables__from_memory_to_disk() is not present in current HEAD; function has a single wrlock/wrunlock pair.	3 months ago
Rene Cannao	6dbd0e9cb1	Merge branch 'v3.0' into fix/postgresql-cluster-sync_2 Resolve merge conflicts in: - lib/ProxySQL_Cluster.cpp - test/tap/tap/Makefile Conflict resolution details: - Kept PostgreSQL/MySQL cluster-sync updates from this branch (SQLQueries delete constants and pgsql sync flow changes) while adopting upstream SQLite prepare_v2 RAII usage where introduced in v3.0 conflict regions. - Combined TAP archive fixes from this branch (archive recreation via rm -f to prevent stale objects) with upstream v3.0 mcp_client.o additions to libtap archives. Validation performed after conflict resolution: - make -C test/tap/tap debug -j4 (success) - make -C lib -j4 (fails in current workspace due missing sqlite-vec.h/vec.o dependency in broader v3.0 content, not conflict markers) - make -C test/tap/tests admin-listen_on_unix-t -j4 (fails in current workspace due missing deps/sqlite3/sqlite3/vec.o, same dependency issue) No conflict markers remain.	3 months ago
Rene Cannao	4cdf61cbae	cluster: complete PostgreSQL servers sync flow and fix TAP libtap stale-archive linker failures This commit finalizes the remaining PostgreSQL cluster-sync implementation work and includes a targeted TAP build fix that addresses the unresolved perform_simple_get() linker error seen while building debug TAP tests. PostgreSQL cluster sync changes: - Add conversion support to transform fetched pgsql resultsets into SQLite3_result objects suitable for admin/runtime loaders. - Rework runtime pgsql server pull path to: - use peer-selected checksums, - fetch runtime rows from the selected peer, - commit runtime state through PgHGM in only-runtime mode, - persist runtime state and keep locking semantics around pgsql server updates. - Rework pgsql_servers_v2 pull path to fetch and validate all related datasets together: - pgsql_servers_v2, - pgsql_replication_hostgroups, - pgsql_hostgroup_attributes, - runtime_pgsql_servers (when applicable). - Ensure checksum validation includes all fetched tables and rejects partial/inconsistent peer payloads. - Load the actual incoming converted datasets into runtime instead of placeholder/empty local resultsets. Test updates: - Extend test_cluster_sync_pgsql-t with additional verification for synchronized pgsql data and optional replica/runtime sync checks. - Adjust TAP planning and resultset handling so all executed queries are validated and cleaned up consistently. TAP build/linker fix: - Update test/tap/tap/Makefile archive targets (libtap_mariadb.a/libtap_mysql57.a/libtap_mysql8.a) to remove existing archives before recreating them. - Add Makefile as a prerequisite for those archive targets so this behavior is applied on rebuild after pulling. - This prevents stale archive members (for example old noise_utils*.o objects) from being retained and reintroduced into libtap.so, which was causing undefined references to an obsolete 4-argument perform_simple_get() symbol. Validation performed in this branch after these changes: - make -C lib -j4 - make -C src -j4 - make -C test/tap/tests test_cluster_sync_pgsql-t -j4 - make -C test/tap/tests admin-listen_on_unix-t -j4	3 months ago
Rene Cannao	e2ebc0c449	Merge branch 'v4.0-mcp-stats2' into v3.0-unified-large-pr	3 months ago
René Cannaò	9c6945fae8	Merge branch 'v3.0' into v3.0-tsdb-feature Signed-off-by: René Cannaò <rene@proxysql.com>	3 months ago
Rene Cannao	70b61a86e3	MCP stats: implement in-memory show_queries Top-K path Replace stats table reads in show_queries with Query Processor in-memory aggregation to avoid stale/empty admin-db snapshots.\n\nIntroduce Top-K digest filtering primitives and wire them into Stats_Tool_Handler so filtering/sorting/limit are executed against live digest memory. Add runtime MCP variable mcp-stats_show_queries_max_rows with validation, defaulting, and configuration plumbing, plus cap metadata in the tool response.\n\nEnforce configurable hard cap semantics while preserving deterministic ordering and pagination behavior for MCP clients. Add internal DEBUG validator path (PROXYSQLTEST 56) in admin tests to stress generated digest data and validate the Top-K/filter pipeline against in-memory state.\n\nKeep the validator callable from startup/testing workflows without depending on MCP linkage. Add TAP coverage in test/tap/tests/mcp_show_queries_topk-t.cpp for:\n- MCP endpoint setup and reachability\n- digest generation + optional DEBUG internal validator\n- payload validation and cap metadata assertions\n- descending count order validation\n- match_digest_text filter verification\n\nMake payload parsing backward-compatible in TAP (direct tool result object and legacy wrapped success/result payloads).	3 months ago
René Cannaò	4a6809b60f	Merge branch 'v3.0' into v3.0-tsdb-feature Signed-off-by: René Cannaò <rene@proxysql.com>	3 months ago
Rene Cannao	ef397f9f1d	Fix TSDB variable patterns and backend monitor runtime source	3 months ago
Rene Cannao	b4e85f1799	Expose PostgreSQL eventslog metrics in stats and Prometheus	3 months ago
Rene Cannao	b100235fbd	Add PGSQL eventslog dump commands and periodic disk sync scheduling	3 months ago
Rene Cannao	2a46c239be	Add PostgreSQL advanced eventslog schema and variable scaffolding	3 months ago
Rene Cannao	700459f67b	tsdb: address AI code review feedback, fix memory safety and logic bugs Summary of changes: - Fixed allocator mismatches (sqlite3_free vs free) across multiple files. - Converted manual SQL construction to prepared statements for TSDB variables. - Resolved memory leaks in statistics and monitoring loops. - Fixed critical 'LOAD TSDB VARIABLES FROM CONFIG' command mismatch. - Improved command parsing robustness using string searching. - Standardized SQL LIKE patterns with escaped wildcards (\%). - Switched to data-driven variable management in ProxySQL_Statistics. - Improved web dashboard security with URL encoding and const correctness. - Clarified documentation regarding configuration lifecycle and schema names. - Updated TAP tests for better reliability and correct persistence verification.	3 months ago
Rene Cannao	b8c6b4f573	tsdb: address code review comments and improve stability - Refactored variable flushing to use prepared statements for safety. - Replaced brittle command parsing with robust string searching. - Extracted TSDB dashboard HTML/JS to separate source file for maintainability. - Added NULL guards for GloProxyStats in Admin and REST API. - Implemented batched metric insertions using transactions for performance. - Refactored backend probing to use non-blocking connect() with timeout. - Fixed potential integer overflows in timer calculations. - Improved error handling and fixed memory leaks in statistics reporting. - Updated TAP tests to verify UI and REST API functionality.	3 months ago
Rene Cannao	6a788e48c4	mcp: make /mcp/query self-healing when targets/backends appear after startup Problem addressed: - MCP query endpoint could stay unusable with 'Tool Handler not initialized' after restart/reload flows. - This was triggered when Query_Tool_Handler could not build an executable pool at init time (for example profiles loaded before ONLINE servers), leaving runtime commands like 'LOAD MCP QUERY RULES FROM MEMORY' blocked behind a NULL query tool handler. - Users had to manually toggle MCP enablement to recover, which is the opposite of expected self-healing behavior. What this commit changes: 1) Query_Tool_Handler pool init is now resilient and idempotent - init_connection_pool() now starts with close() so reinitialization fully resets stale mysql/pgsql pool state before rebuilding from runtime profile + server tables. - If no executable targets are available, init_connection_pool() now returns success with a warning instead of hard failure. This allows the query tool handler (and /mcp/query endpoint) to stay initialized even before backends are ready. 2) Lazy auto-rebuild on first query usage - get_connection() and get_pgsql_connection() now: - refresh target registry before resolution, - attempt to use an existing pooled connection, - if unavailable, trigger a full pool rebuild (init_connection_pool()) and retry once. - This provides automatic recovery when hostgroups/servers/profiles are loaded or changed after MCP startup, without requiring manual MCP disable/enable. 3) Admin runtime rule load path attempts MCP recovery - load_mcp_query_rules_to_runtime() now detects NULL query tool handler and calls load_mcp_server() once before failing. - This turns a hard, immediate admin error into a self-recovery attempt consistent with MCP runtime semantics. Behavioral impact: - /mcp/query endpoint remains online even when there are temporarily zero executable targets. - As soon as compatible runtime targets/backends exist, run_sql_readonly/explain_sql can recover automatically on demand. - LOAD MCP QUERY RULES TO/FROM RUNTIME no longer fails immediately on first NULL handler condition; it retries after MCP server recovery. Validation performed: - Recompiled modified objects successfully: - lib/obj/Query_Tool_Handler.oo - lib/obj/ProxySQL_Admin.oo - Full TAP runtime test execution is not possible in this sandbox due blocked local TCP socket creation; validation should be run in the normal test environment where ProxySQL/MySQL/PGSQL are reachable.	3 months ago
Rene Cannao	998bd82387	MCP TAP startup: fix tool-handler initialization order, improve MCP PROFILES observability, and seed monitor users This change fixes recurring MCP TAP failures where `/mcp/query` returned: Tool Handler not initialized for endpoint: query and where backend monitor auth failures flooded logs. Problem summary - MCP server startup can occur before runtime target/auth profiles and backend server mappings are loaded. - If that happens, Query_Tool_Handler initialization sees no executable targets and remains NULL. - MCP endpoint resources bind the handler pointer at creation time, so a NULL query handler at startup breaks `/mcp/query` until server restart. Code changes 1) Add explicit admin command logging for MCP PROFILES commands - Added `Received <command>` logging in the MCP PROFILES command block, matching behavior of other admin command handlers. - File: `lib/Admin_Handler.cpp` 2) Trigger MCP server refresh after `LOAD MCP PROFILES TO RUNTIME` - After copying profiles into runtime and rebuilding target/auth map, call `ProxySQL_Admin::load_mcp_server()`. - This allows MCP to self-heal when profiles become available after initial startup. - File: `lib/Admin_Handler.cpp` 3) Restart MCP server when query handler is missing - Extended `ProxySQL_Admin::load_mcp_server()` restart checks to include: - running server + `query_tool_handler == NULL` - Restart reason now includes tool handler initialization mismatch. - File: `lib/ProxySQL_Admin.cpp` 4) Fix TAP configurator load order to avoid early MCP startup - Reordered `test/tap/tests/mcp_rules_testing/configure_mcp.sh` runtime sequence: - `LOAD MYSQL SERVERS TO RUNTIME` - `LOAD PGSQL SERVERS TO RUNTIME` (best effort) - `LOAD MCP PROFILES TO RUNTIME` - `LOAD MCP VARIABLES TO RUNTIME` (last) - This ensures MCP starts only after routing/auth context is present. 5) Seed monitor credentials in AI local infra pre-hook - Added backend user/role creation for default monitor credentials `monitor/monitor`: - MySQL: create user + monitor-relevant grants - PostgreSQL: create role + `pg_monitor` + DB connect grants - Reduces monitor auth noise in local AI TAP dockerized setup. - File: `test/tap/groups/ai/pre-proxysql.bash` 6) Mark new TAP phase scripts executable - `test_phase10_eval_explain.sh` - `test_phase11_pgsql_target.sh` Expected outcome - MCP query endpoint no longer stays stuck with an uninitialized tool handler after TAP configuration. - MCP query-rules admin commands stop failing due to missing Query_Tool_Handler. - MCP profile command flow is visible in logs for easier debugging. - Local AI TAP infra no longer emits continuous monitor authentication failures for default monitor credentials.	3 months ago
Rene Cannao	af0411bd46	MCP: add target-aware rules/stats tests, explain_sql rule coverage, and AI local docker TAP infra This commit completes end-to-end MCP query-rules validation for multi-target routing and introduces a self-contained TAP infra for the `ai` group that can run both in Jenkins and manually. Main MCP/runtime changes: - Extended MCP query-rule matching context to include both `target_id` and resolved backend `username`. - Added `target_id` column to `mcp_query_rules` and `runtime_mcp_query_rules` table definitions. - Extended `stats_mcp_query_rules` to include `username` and `target_id` alongside `rule_id` and `hits`. - Updated load/save/runtime refresh paths to persist and rehydrate the expanded MCP rule schema. - Wired the rule engine into `explain_sql` so MCP rules apply consistently across `run_sql_readonly` and `explain_sql`. - Included startup-order fix in `src/main.cpp` to initialize MCP/GenAI thread handlers early, preventing startup crashes in PROXYSQLGENAI builds. Test coverage changes: - Updated existing MCP TAP phases to assert target-aware and username-aware behavior: - `test_phase4_stats.sh` - `test_phase6_eval_block.sh` - Added new MCP TAP phases: - `test_phase10_eval_explain.sh` (rule engine coverage for `explain_sql`) - `test_phase11_pgsql_target.sh` (pgsql target routing/rule/stats coverage; graceful skip if no pgsql target is configured) - Updated `test_mcp_query_rules-t.sh` runner to execute new phases. AI group isolated infra (manual + CI compatible): - Added `test/tap/groups/ai/docker-compose.yml` with MySQL 9.0 and PostgreSQL 16 backends. - Added lifecycle scripts: - `docker-compose-init.bash` - `docker-compose-destroy.bash` - `pre-proxysql.bash` - `post-proxysql.bash` - Extended `test/tap/groups/ai/env.sh` with local default ports/credentials/target IDs used by MCP TAP tests. - Added `test/tap/groups/ai/README.md` documenting manual execution flow outside Jenkins. Outcome: - MCP tests now validate routing-aware rule enforcement and stats attribution for both MySQL and PostgreSQL targets. - The `ai` TAP group can be run with an isolated local backend stack without relying on external Jenkins infra repositories.	3 months ago
Rene Cannao	013864b36f	MCP: introduce profile-based target/auth routing and unified LOAD/SAVE MCP PROFILES commands This commit introduces the core MCP routing architecture that decouples client-facing target selection from backend connection details. Architecture introduced: - New MCP auth profile table to store backend credentials/policy (`mcp_auth_profiles`). - New MCP target profile table to map opaque `target_id` -> protocol/hostgroup/auth profile (`mcp_target_profiles`). - Runtime mirrors for both profile tables. Behavioral changes: - MCP query tools resolve execution context by `target_id` and route internally by protocol + hostgroup. - Client no longer needs backend-specific connection details; only logical target identifiers. - Unified admin commands for profile lifecycle management: - `LOAD MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` - `SAVE MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` Design goals: - Support multi-backend MCP execution (MySQL + PostgreSQL) behind one endpoint. - Provide server-side credential management and cleaner operational model. - Replace legacy one-backend POC configuration with scalable target/auth abstractions. Result: - MCP routing is now dynamic, protocol-agnostic at the client layer, and fully managed through profile tables and unified load/save commands.	3 months ago
Rene Cannao	5f5b7d604f	tsdb: improve administrative interface and command support - Move 'stats_tsdb' to 'main' schema for better visibility in 'SHOW TSDB STATUS'. - Implement 'PROXYSQL TSDB DOWNSAMPLE' command to trigger manual metrics aggregation. - Ensure 'SHOW TSDB STATUS' triggers a statistics refresh before returning data. - Integrate 'tsdb-' variables into 'runtime_global_variables' refresh logic. - Update TAP tests to verify new commands and improve troubleshooting output.	3 months ago
Rene Cannao	6587bfdd40	tsdb: trigger stats refresh on SHOW TSDB STATUS	3 months ago
Rene Cannao	2d51aeff64	tsdb: correctly implement SHOW TSDB commands and variable initialization	3 months ago
Rene Cannao	6404d7b365	tsdb: implement SHOW TSDB VARIABLES and SHOW TSDB STATUS	3 months ago
Rene Cannao	4e0f94a345	tsdb: fix compilation error in ProxySQL_Admin.cpp	3 months ago
Rene Cannao	a236a13787	tsdb: dedicated module for TSDB variables with tsdb- prefix and optimized background tasks	3 months ago
Rene Cannao	da877c378b	tsdb: align admin variable lifecycle, full prometheus ingestion, docs and tap coverage	3 months ago
Rene Cannao	200fdecb8d	Apply AI agent review fixes to PR #5374 Critical Fixes: - Fix NULL pointer dereferences in MySQL_Logger.cpp: * Add null checks for myds->myconn->parent in log_audit_entry() * Fix cleanup code in log_request() and log_audit_entry() * Change sprintf to snprintf for buffer safety - Remove redundant pthread_rwlock_init in debug.cpp: * filters_rwlock already initialized with PTHREAD_RWLOCK_INITIALIZER * Re-initialization caused undefined behavior per POSIX Major Improvements: - Add bounded wait helper wait_for_glo_mth() in proxysql_utils.h: * Replaces indefinite while loops with 10s timeout * Applied to 26 locations across 4 files Build System: - Require C++17 in Makefile (remove C++11 fallback): * Code uses C++17 structured bindings extensively * Cleanest solution vs adding fallback guards to 27+ locations Compilation fixes: - Add unistd.h include for usleep - Remove conflicting GloMyMon extern declaration Files changed: - Makefile - include/proxysql_utils.h - lib/MySQL_Logger.cpp - lib/debug.cpp - lib/MySQL_Monitor.cpp - lib/ProxySQL_Admin.cpp - lib/ClickHouse_Server.cpp - src/SQLite3_Server.cpp	3 months ago
Rene Cannao	3c85c593cf	Add GloMTH initialization wait to all monitor and server threads Fix race condition where threads were accessing GloMTH before it was fully initialized, causing Valgrind errors in get_variable_int() and related functions. Added wait loop pattern (while GloMTH==NULL { usleep(50000); }) followed by usleep(100000) to ensure GloMTH is fully initialized before use: In lib/MySQL_Monitor.cpp: - monitor_connect(), monitor_ping(), monitor_read_only() - monitor_group_replication(), monitor_galera(), monitor_replication_lag() - monitor_AWS_Aurora_thread_HG(), monitor_aws_aurora() - init_mysql_thread_struct() In src/SQLite3_Server.cpp: - SQLite3_Server_session_handler In lib/ProxySQL_Admin.cpp: - admin_session_handler In lib/ClickHouse_Server.cpp: - ClickHouse_Server_session_handler The pattern ensures threads wait for GloMTH initialization during startup, then check for NULL during shutdown to exit cleanly.	3 months ago
Rene Cannao	ee7ee0b387	Fix multiple Valgrind-reported uninitialized memory issues 1. MySQL_Logger.cpp: Add NULL checks for myconn->parent before accessing address/port. Client connections (frontend) have parent=NULL, causing uninitialized value reads when logging. 2. debug.cpp: Add NULL check for GloVars.global.gdbg_lvl before accessing debug level configuration. Prevents use of uninitialized data when proxy_debug_func is called before init_debug_struct() completes. 3. ProxySQL_Admin.cpp: Add NULL checks for debug filter database fields before constructing filter strings. Prevents undefined behavior when constructing std::string from NULL pointers.	3 months ago
Rene Cannao	cc53ddca23	Merge branch 'v3.0-openssl-fix' into v3.0-test0213	3 months ago
Rene Cannao	746eca9f7c	More sprintf to snprintf conversions and indentation fixes - Replace remaining sprintf with snprintf in Admin_FlushVariables.cpp - Fix indentation in flush_GENERIC_variables__process__database_to_runtime - Apply same pattern to ProxySQL_Admin.cpp and ProxySQL_Admin_Stats.cpp	3 months ago
Rene Cannao	3e37337877	Use RAII for sqlite3 statements across codebase - Convert prepare_v2() calls to use unique_ptr RAII pattern - Remove manual sqlite3_finalize calls (now handled by RAII) - Update flush_*_variables functions to use new prepare_v2 return pattern - Apply changes to Admin, Cluster, Monitor, Catalog, and other modules	3 months ago
Rene Cannao	afd6dffb05	Fix admin shutdown races and implement graceful teardown - Add coordinated shutdown for PROXYSQL KILL/SHUTDOWN in debug builds - Track active admin client threads with admin_client_threads_active counter - Add shutdown checks to prevent new connections during shutdown - Wait for all admin client threads to finish before cleanup in admin_shutdown() - Handle accept failures gracefully	3 months ago
Rene Cannao	3ab964010d	Add protocol labels to Query Cache metrics and enable PostgreSQL QC metrics This commit resolves the prometheus metric collision between MySQL and PostgreSQL Query Cache instances by creating separate metric maps with protocol labels and using compile-time type trait selection. Changes: - Add #include <type_traits> to Query_Cache.cpp - Create qc_metrics_map_mysql with { "protocol", "mysql" } labels (8 metrics) - Create qc_metrics_map_pgsql with { "protocol", "pgsql" } labels (8 metrics) - Modify Query_Cache constructor to use if constexpr with std::is_same_v to select the appropriate metrics map based on derived type - Enable GloPgQC metrics export in update_modules_metrics() This allows MySQL and PostgreSQL query cache metrics to coexist without collision, following the same pattern as Thread Handler metrics. Related: #5068, #5069	4 months ago
Rene Cannao	778e01174c	Add protocol labels to Thread Handler metrics and enable PostgreSQL metrics This commit extends PR #5069 by adding protocol labels to distinguish MySQL and PostgreSQL metrics in the Thread Handler modules. Changes: - Add { "protocol", "mysql" } labels to 65 MySQL_Thread.cpp metrics - Add { "protocol", "pgsql" } labels to 60 PgSQL_Thread.cpp metrics - Enable GloPTH (PostgreSQL threads handler) metrics export - Document Query Cache limitation (template base class collision) This resolves Issue #5068 by ensuring PostgreSQL prometheus metrics are properly exported with protocol labels, distinguishing them from MySQL metrics. Related: #5068, #5069	4 months ago

1 2 3 4 5 ...

1024 Commits (4e39f3bdc4888ee7bd34b3fcba3b986503efe20b)