proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	2a1b19916e	test(tap): register pgsql_query_logging_autodump-t in test groups Add the new autodump E2E test to the default-g4 test group configuration.	4 days ago
Rene Cannao	82a28a2e62	fix(pgsql-eventslog): close remaining minor review findings in logger internals Address two minor correctness/style follow-ups from final review: 1) `PgSQL_Event::set_error()` overwrite safety - Free previous `sqlstate` / `errmsg` buffers when either ownership mode is active (`free_on_delete` or `free_error_on_delete`) before replacing pointers. - This avoids latent leaks if `set_error()` is called on deep-copied events. 2) Explicit atomic access for circular-buffer capacity - Keep `buffer_size` atomic and use explicit `load/store` operations with `std::memory_order_relaxed` in hot-path and helper methods. - Update log-path check, insert path, getter, and setter to use explicit atomic operations for clearer thread-safety semantics.	4 days ago
Rene Cannao	ab7d9187cb	test(pgsql-eventslog): improve autodump assertion diagnostics and runtime cleanup Test diagnostics: - Add explicit `diag()` output for successful-row assertion failures. - Include expected row count, observed row count, and query status for faster CI triage. Cleanup behavior: - Extend cleanup to restore: - `pgsql-eventslog_default_log=0` - `pgsql-eventslog_buffer_history_size=0` - Apply both runtime loads during cleanup: - `LOAD ADMIN VARIABLES TO RUNTIME` - `LOAD PGSQL VARIABLES TO RUNTIME` This avoids residual PGSQL eventslog settings leaking into subsequent TAP tests.	5 days ago
Rene Cannao	6db089883e	fix(admin): improve PGSQL eventslog dump failure reporting and stats insert path Admin command behavior: - `DUMP PGSQL EVENTSLOG ...` returned `OK 0` when `GloPgSQL_Logger` was null. - This masked unavailable logger state from operators. Change: - Return an explicit admin error (`PgSQL logger not initialized`) when logger is unavailable. - Preserve normal `OK` response path when logger is present. Stats path behavior: - PgSQL logger metrics insertion into `stats_pgsql_global` used per-row string formatting and `execute()` calls. Change: - Replace per-row dynamic SQL construction with a prepared-statement row insert path. - Reuse `sqlite3_global_stats_row_step()` for consistent binding/reset semantics.	5 days ago
Rene Cannao	0aa3c5e153	fix(pgsql-eventslog): harden event ownership and keep newest rows in memory table Problem 1: - `PgSQL_Event::set_error()` stored borrowed pointers from connection-owned strings. - This could create lifetime hazards before/around event serialization and buffering. Fix: - Add explicit ownership tracking for error fields via `free_error_on_delete`. - Make `set_error()` duplicate incoming SQLSTATE/error strings. - Free owned error strings in destructor when the event does not own all fields. - Keep copy-constructor behavior for deep-copied buffered events and disable copy-assignment to avoid accidental shallow assignment. Problem 2: - `processEvents()` could repopulate `stats_pgsql_query_events` using the beginning of the drained batch after retention pruning. - This keeps older rows instead of the most recent rows when truncation is required. Fix: - Insert the tail of the drained batch (`events.end() - numEventsToInsert`) so memory table retention always preserves latest events. - Tighten row retention arithmetic using `size_t` to avoid signed/unsigned mismatch in keep/delete calculations.	5 days ago
Rene Cannao	eba795d729	fix(pgsql-eventslog): make dump-to-memory sizing protocol-agnostic across admin ports Context: - Admin commands are exposed by one shared Admin module over two protocols: - MySQL admin endpoint (`6032`) - PostgreSQL admin endpoint (`6132`) - `DUMP PGSQL EVENTSLOG ...` must behave identically regardless of which admin protocol executes the command. Issue fixed: - PGSQL events dump path used thread-local PGSQL sizing variables when draining to memory (`stats_pgsql_query_events`). - When command execution happened from a non-PGSQL protocol admin worker, TLS values could be out of sync for PGSQL sizing decisions. Changes: - In `lib/PgSQL_Logger.cpp`, add runtime-size helper functions that read canonical values from `GloPTH->variables` with TLS fallback: - `eventslog_buffer_max_query_length` - `eventslog_table_memory_size` - Use these helpers in: - `PgSQL_Event` copy constructor (buffered query truncation) - `PgSQL_Logger::processEvents()` (in-memory retention for `stats_pgsql_query_events`) - Update internal architecture doc to explicitly state command availability from both admin protocol endpoints. Why: - Ensure `DUMP PGSQL EVENTSLOG ...` semantics remain consistent across `6032` and `6132`, matching the shared-admin-module design.	5 days ago
Rene Cannao	6f5a22f669	docs(internal): update PGSQL advanced logging architecture validation section with auto-dump E2E coverage Context: - The architecture document was rewritten as an as-built reference, but latest test coverage additions were not fully reflected. Changes: - Update validation mapping in `doc/internal/pgsql_advanced_query_logging_architecture.md` to include: - `pgsql_query_logging_autodump-t` TAP test - scheduler-driven auto-dump acceptance coverage - both relevant TAP group registrations Why: - Keep internal architecture/validation documentation aligned with implemented and wired test coverage. - Avoid stale validation references when reviewing branch readiness.	5 days ago
Rene Cannao	97e5cfee90	docs(doxygen): expand PostgreSQL logger API documentation for new advanced logging components Context: - The PostgreSQL advanced query logging implementation added new event, buffer, and metrics APIs. - Several newly introduced functions lacked Doxygen coverage. Changes: - Add Doxygen documentation blocks for newly introduced logging types and APIs in `include/PgSQL_Logger.hpp`: - metric enums and metric-map indexes - `PGSQL_LOG_EVENT_TYPE` - `PgSQL_Event` class and its new/updated lifecycle + serialization methods - `PgSQL_Logger_CircularBuffer` constructor/destructor and buffer operations - `PgSQL_Logger` constructor/destructor Why: - Improve maintainability and code navigation for the new PostgreSQL advanced logging feature set. - Align implementation with project expectation that introduced features include inline Doxygen documentation.	5 days ago
Rene Cannao	2586995478	tap(pgsql): add end-to-end test for eventslog automatic buffer-to-disk sync Context: - PostgreSQL advanced query logging already had end-to-end coverage for manual dump flows. - Scheduler-driven automatic dump path also needs explicit TAP coverage. Test added: - `test/tap/tests/pgsql_query_logging_autodump-t.cpp` - Connects to PGSQL admin and frontend interfaces. - Enables query logging and admin scheduler variable: `admin-stats_pgsql_eventslog_sync_buffer_to_disk=1` - Generates frontend workload (`SELECT 1` loop). - Polls `history_pgsql_query_events` to assert rows are flushed to disk by scheduler. - Validates successful rows are present (`sqlstate IS NULL`). - Restores scheduler variable to `0` during cleanup. Harness wiring: - Register `pgsql_query_logging_autodump-t` in `test/tap/groups/groups.json` so it runs with default PGSQL test group settings. Why: - Close e2e coverage gap between manual dump and auto-dump behavior for PGSQL events logging. - Catch regressions in scheduler-based buffer synchronization path.	5 days ago
Rene Cannao	20d6282a60	build(clean): ignore and remove generated MySQL parser artifacts Context: - Bison/Flex-generated parser files in `lib/` can be created locally during parser work. - These files are generated artifacts and should not be tracked. Changes: - Add generated parser artifacts to `.gitignore`: - `lib/MySQL_Lexer.yy.c` - `lib/MySQL_Parser.output` - `lib/MySQL_Parser.tab.c` - `lib/MySQL_Parser.tab.h` - Extend `lib/Makefile` `clean` target to remove the same generated files. Why: - Prevent accidental inclusion of regenerated parser outputs in feature commits. - Keep worktrees clean and make `make clean` deterministic for parser-related changes.	5 days ago
Rene Cannao	1edf1f8e86	Add PostgreSQL advanced logging TAP coverage and rewrite architecture doc	5 days ago
Rene Cannao	b4e85f1799	Expose PostgreSQL eventslog metrics in stats and Prometheus	5 days ago
Rene Cannao	b100235fbd	Add PGSQL eventslog dump commands and periodic disk sync scheduling	5 days ago
Rene Cannao	acbd7d0df3	Implement PostgreSQL eventslog circular buffer and sink pipeline	5 days ago
Rene Cannao	2a46c239be	Add PostgreSQL advanced eventslog schema and variable scaffolding	5 days ago
René Cannaò	10ff1322ba	Merge pull request #5386 from sysown/v3.0-MCP_multi MCP query+harvest hardening: HGM routing, weighted targets, pgsql harvest fix, TAP updates	5 days ago
Rene Cannao	253d098653	security: fix SQL injection vulnerabilities in SQLite catalog queries Address Gemini Code Assist review feedback on PR #5386: 1. Discovery_Schema::resolve_run_id - Replace string concatenation with replace_str escaping with proper parameterized queries using sqlite3_prepare_v2 and sqlite3_bind_* functions. 2. PgSQL_Static_Harvester - Remove unsafe escape_sql_string() function and replace its usages with: - New lookup_object_id() helper using parameterized queries - Parameterized UPDATE statements in harvest_view_definitions() These changes ensure complete protection against SQL injection for the catalog database, following the pattern already used in create_run().	5 days ago
Rene Cannao	0da772a4c1	tap groups: add new MCP test assignments to ai-g1 group Add test_mcp_claude_headless_flow-t, test_mcp_llm_discovery_phaseb-t, and test_mcp_static_harvest-t to the ai-g1 test group.	5 days ago
Rene Cannao	2fa1401b05	MCP query+harvest hardening: HGM-based target routing, weighted backend selection, pgsql schema fix, and TAP robustness This squashed commit consolidates the recent MCP stabilization work across query execution and static-discovery tests. 1) Query endpoint routing/executability now uses Hostgroup Manager runtime snapshots - File: lib/Query_Tool_Handler.cpp - Removed dependency on direct SQL probes against runtime_mysql_servers/runtime_pgsql_servers from inside the MCP query handler path. - Target backend resolution now reuses the same Hostgroup Manager runtime snapshot APIs that feed Admin runtime server visibility: - MyHGM->dump_table_mysql("mysql_servers") - PgHGM->dump_table_pgsql("pgsql_servers") - Non-executable diagnostics now summarize statuses from HGM snapshots instead of ad hoc runtime-table SQL. 2) Backend endpoint selection changed to weighted random - File: lib/Query_Tool_Handler.cpp - Selection remains constrained by: - matching target hostgroup_id - ONLINE backend status - Candidate choice now uses random selection with linear probability by weight, replacing deterministic ordering. 3) Read-only validation and guardrail adjustments - File: lib/Query_Tool_Handler.cpp - Fixed false negatives for valid read-only queries such as SELECT literals (no FROM clause required). - Added additional dangerous-query patterns for file-write variants: - INTO OUTFILE - INTO DUMPFILE 4) PostgreSQL static harvester schema query fix - File: lib/PgSQL_Static_Harvester.cpp - Fixed harvest_schemas() failure on PostgreSQL caused by querying non-existent information_schema.schemata.default_collation_name. - Replaced with PostgreSQL-safe projection while preserving the insert_schema() column contract. 5) TAP reliability improvements for MCP test phases - File: test/tap/tests/mcp_rules_testing/test_phase11_pgsql_target.sh - Isolated phase state by clearing MCP rules before setup. - Added verbose response prints for T11.1-T11.4. - Added LOGENTRY phase start/end markers and EXIT trap for guaranteed end watermark. - Replaced fragile grep/escaping checks with robust string assertions. - Fixed NULL-concat stats assertions with COALESCE(username,'') and COALESCE(target_id,''). - File: test/tap/tests/test_mcp_static_harvest-t.sh - Hardened escaped-JSON parsing in MCP response checks (run_id extraction + key/value assertions). - Reworked target/protocol/object assertions to avoid transport-format false negatives. - Corrected cross-target negative assertion to check MCP isError semantics. Observed result in user verification - Phase 11 pgsql target/rules/stats checks pass. - Static harvest phase-A checks pass for mixed mysql/pgsql target_id flow. This commit intentionally keeps runtime ONLINE-only target executability semantics and improves observability/diagnostic quality for future MCP routing/test failures.	6 days ago
Rene Cannao	9ffc3f8d71	mcp tests: add phase-B TAP coverage and optional real Claude CLI E2E runner Add two complementary TAP tests for phase-B validation and an optional real Claude CLI E2E helper, so we can validate both 'without Claude credentials' and 'with real Claude CLI' workflows. What was added: 1) CI-safe deterministic phase-B TAP - New: test/tap/tests/test_mcp_llm_discovery_phaseb-t.sh - Validates MCP phase-B primitives end-to-end without external LLM API: - list_targets - discovery.run_static (target_id-scoped setup) - catalog.list_objects - agent.run_start / agent.run_finish - llm.summary_upsert / llm.summary_get - llm.domain_upsert / llm.domain_set_members - llm.metric_upsert - llm.question_template_add - llm.search - Uses unique per-run markers to assert persisted artifacts are retrievable 2) Claude headless flow TAP smoke - New: test/tap/tests/test_mcp_claude_headless_flow-t.sh - Always validates integration path without external dependencies: - static_harvest.sh wrapper executes and yields run_id - two_phase_discovery.py --dry-run executes with target_id/run_id context - Optional real Claude execution: - enabled via TAP_RUN_REAL_CLAUDE=1 - skipped by default to keep CI deterministic 3) Manual real-CLI E2E runner - New: scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/run_real_claude_e2e.sh - Runs full two-step flow manually when credentials/CLI are available: - phase-A static harvest (or --skip-phase-a + --run-id) - phase-B real Claude run via two_phase_discovery.py 4) Documentation updates - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/README.md: - documents run_real_claude_e2e.sh usage - test/tap/groups/ai/README.md: - adds manual run instructions for: - phase-A static harvest test - phase-B deterministic TAP test - Claude headless smoke (with optional real mode) Validation run: - bash -n: - test_mcp_llm_discovery_phaseb-t.sh - test_mcp_claude_headless_flow-t.sh - run_real_claude_e2e.sh - static_harvest.sh - python3 -m py_compile: - two_phase_discovery.py	6 days ago
Rene Cannao	ade0130e67	discoveryagent: update Claude Code headless flow for target_id-scoped MCP tools Enhance the existing Claude Code custom-prompt discovery examples to match the new MCP contract introduced by multi-target routing and target-scoped catalog semantics. Changes included: - static_harvest.sh - added required --target-id argument (with MCP_TARGET_ID env fallback) - always sends target_id in discovery.run_static arguments - updated usage/help/examples to include target_id - two_phase_discovery.py - added required --target-id argument - static-harvest bootstrap call now passes target_id - improved run_id error guidance with target_id-aware commands - injects <TARGET_ID> placeholder into user prompt template - improved runtime logging to display target_id - Prompt templates - two_phase_discovery_prompt.md: - catalog/agent/llm tool signatures updated to include target_id - Stage 0 and workflow text updated to use provided target_id + run_id - removed contradictory instruction that asked to call discovery.run_static in Phase 2 - two_phase_user_prompt.md: - added target_id input section - all example calls updated to pass target_id - start instructions now require target_id + run_id - README and utility example - README quick-start curl and script examples now include target_id - test_catalog.sh now accepts/prints target_id and passes it to catalog/llm calls Validation: - bash -n passed for updated shell scripts - python3 -m py_compile passed for two_phase_discovery.py	6 days ago
Rene Cannao	2538e303cf	tap ai: add dual-backend static-harvest fixtures and target_id coverage test Implement phase-A (static harvesting) TAP coverage for MCP multi-target discovery by seeding deterministic schemas on both MySQL and PostgreSQL and validating discovery/catalog behavior per target_id. What this commit adds: - AI group deterministic seed datasets - Added test/tap/groups/ai/mysql-seed.sql with: - tap_mysql_static_customers - tap_mysql_static_orders (FK to customers) - Added test/tap/groups/ai/pgsql-seed.sql with: - tap_pgsql_static_accounts - tap_pgsql_static_events (FK to accounts) - pre-proxysql hook integration - Updated test/tap/groups/ai/pre-proxysql.bash to seed both backends after container startup: - seed_mysql_test_data() executes mysql-seed.sql via mysql CLI - seed_pgsql_test_data() executes pgsql-seed.sql via psql (or docker compose exec fallback) - Existing monitor-user/profile setup is preserved - New TAP test: test/tap/tests/test_mcp_static_harvest-t.sh - Validates MCP/ProxySQL reachability - Validates list_targets exposes both mysql and pgsql target_id entries - Runs discovery.run_static for MySQL target_id and validates run_id + protocol=mysql - Validates catalog.list_objects returns seeded MySQL table for that run - Runs discovery.run_static for PostgreSQL target_id and validates run_id + protocol=pgsql - Validates catalog.list_objects returns seeded PostgreSQL table for that run - Validates run isolation across targets (cross-target run_id lookup fails as expected) - Documentation update - Updated test/tap/groups/ai/README.md with seeded-table details and manual run instructions for the new static-harvest test Notes: - This commit focuses strictly on phase-A static harvesting, as requested. - Phase-B (LLM-driven discovery) tests are intentionally not included here.	6 days ago
Rene Cannao	9685cdaa4b	mcp discovery: enforce target-scoped run model and add protocol-aware static harvesting (mysql+pgsql) Refactor the MCP query/discovery stack to remove prototype-era MySQL-only assumptions and make discovery/catalog semantics explicitly target-scoped. This commit is intentionally not backward compatible with legacy single-target catalog metadata. Key implementation details: - Discovery_Schema: - runs table now stores target_id, protocol, and server_version - resolve_run_id() now requires target_id and resolves schemas within target scope - create_run() now records target/protocol/server version - added legacy schema detection + destructive catalog table rebuild when old runs layout is found - fixed resultset lifetime/escaping issues in target-aware run resolution - New PostgreSQL static harvester: - added PgSQL_Static_Harvester class and build wiring - harvests schemas/objects/columns/indexes/fks/view definitions into Discovery_Schema - maps pg metadata to object_id-based insert APIs used by Discovery_Schema - Query_Tool_Handler: - constructor simplified to catalog_path only (no mcp-mysql_* runtime ctor deps) - discovery.run_static now requires target_id and dispatches protocol-aware harvester at runtime - catalog., agent.run_start, and llm. tool contracts now require target_id when resolving run_id - all run_id resolution call sites switched to resolve_run_id(target_id, ...) - list_schemas catalog query now scoped by target_id via runs join - digest logging path now resolves run_id in target context - ProxySQL_MCP_Server: - updated Query_Tool_Handler construction to new signature - Docs updated: - added explicit target_id scoping guidance for discovery/catalog/agent/llm workflows - clarified protocol-aware routing and noted legacy examples still present in large script docs Build validation: - Recompiled changed objects with PROXYSQLGENAI=1: - Query_Tool_Handler.oo - Discovery_Schema.oo - PgSQL_Static_Harvester.oo - ProxySQL_MCP_Server.oo	6 days ago
Rene Cannao	7e54883ba5	mcp query debugging: surface rule-id context for blocks and add backend failure SQL details Observed confusion: - MCP failures like 'Error 100' / 'Error 101' were visible at endpoint level but gave no direct clue whether they came from query-rule blocks or backend execution failures. - Connection/firewall-style failures also lacked enough target/query context in lower-level logs. Changes in this commit: 1) Log MCP query-rule block/OK matches with rule id and context - In Discovery_Schema::evaluate_mcp_query_rules(): - when error_msg rule action is applied, log: rule_id, tool, target_id, schema, error_msg, query preview. - when OK_msg rule action is applied, log: rule_id, tool, target_id, schema, ok_msg. - This makes it explicit which runtime_mcp_query_rules row produced the returned error. 2) Add detailed backend execution error logs in Query_Tool_Handler - For MySQL and PostgreSQL execution paths (with and without schema switching), failures now log: - target_id - schema (when applicable) - backend error text - SQL query text - Added context for PG search_path failures too. - This disambiguates rule-block errors from actual backend connectivity/authorization/firewall failures. Behavioral notes: - API responses are unchanged (to avoid breaking tests expecting exact messages). - Logging is now sufficiently descriptive to trace each failure to either: - a specific MCP rule id/action, or - a concrete backend execution error. Validation: - Recompiled successfully: - lib/obj/Discovery_Schema.oo - lib/obj/Query_Tool_Handler.oo	6 days ago
Rene Cannao	8fbd570c79	mcp variables: stop writing runtime_global_variables during LOAD MCP VARIABLES TO RUNTIME Issue: - MCP variable load path was writing rows into runtime_global_variables. - For MCP this is incorrect and misleading; runtime_global_variables should not be used as a write target in this flow. - Logs showed explicit INSERT statements into runtime_global_variables for mcp-* entries. Fix implemented: 1) Removed runtime_global_variables population from MCP database->runtime flow - In flush_mcp_variables___database_to_runtime(), removed the block that called: flush_mcp_variables___runtime_to_database(..., runtime=true, ...) - LOAD MCP VARIABLES TO RUNTIME now applies mcp-* variables to GloMCPH and reloads MCP server state, without runtime_global_variables writes. 2) Removed runtime_global_variables writes from MCP runtime->database helper - In flush_mcp_variables___runtime_to_database(): - removed DELETE FROM runtime_global_variables WHERE variable_name LIKE 'mcp-%' - removed INSERT INTO runtime_global_variables(...) per variable - Function now persists MCP variables only to global_variables (main/disk paths), which is the intended save behavior. 3) Removed temporary MCP info-level debug noise related to runtime_global_variables inserts - Deleted MCP info logs that printed per-variable INSERT SQL into runtime_global_variables. Build validation: - Recompiled successfully: - lib/obj/Admin_FlushVariables.oo Expected result after this commit: - Executing LOAD MCP VARIABLES TO RUNTIME should no longer emit any INSERT/DELETE against runtime_global_variables for mcp-* variables. - MCP runtime behavior remains: variables are applied in-memory and MCP server reload logic is triggered via load_mcp_server().	6 days ago
Rene Cannao	f15460348c	mcp query diagnostics: include runtime hostgroup status breakdown when target lacks ONLINE backend Adds targeted debug context for non-executable MCP targets. When Query_Tool_Handler reports that a target has no ONLINE backend, it now appends a status summary read from the corresponding runtime server table for that hostgroup: - MySQL: runtime_mysql_servers - PostgreSQL: runtime_pgsql_servers Example diagnostics now include status aggregates such as: - ONLINE=0, SHUNNED=1 - OFFLINE_SOFT=1 - no rows in runtime_*_servers for hostgroup X This closes the observability gap where users saw 'target non-executable' but could not immediately tell why the hostgroup failed ONLINE eligibility. Validation: - Recompiled lib/obj/Query_Tool_Handler.oo successfully.	6 days ago
Rene Cannao	f4bc1943fb	mcp query diagnostics: restore strict ONLINE requirement and explain target non-executable failures This change does two things: 1) Reverts executable-target policy to strict ONLINE backend status - Query target resolution in Query_Tool_Handler is restored to require UPPER(status)='ONLINE' in runtime_mysql_servers/runtime_pgsql_servers. - This matches expected semantics: non-ONLINE backends should not be considered executable MCP targets. 2) Replaces generic non-executable errors with actionable diagnostics - Added Query_Tool_Handler::format_target_unavailable_error(target_id). - All query-tool paths that previously returned the generic: 'Target is not executable by this handler' now return a reasoned error with context, including: - target_id - protocol - hostgroup_id - auth_profile_id - concrete reason (for example: empty db_username in auth profile, or no ONLINE backend in runtime_*_servers) - Internal logs in get_connection()/get_pgsql_connection() now emit the same detailed reason. 3) Improves MCP endpoint failure logging with SQL/target context - In MCP_JSONRPC_Resource::handle_tools_call(), failed tool calls now log: - endpoint/tool/error - full arguments payload (existing) - additional parsed details when present: target_id, schema, sql (trimmed for safety) - This makes it explicit what SQL was requested when a tool call fails before execution. 4) Protocol normalization retained for robustness - MCP_Threads_Handler::load_target_auth_map() lowercases protocol values when loading runtime profile joins. - This avoids protocol-casing drift causing inconsistent routing behavior. Build validation: - Successfully recompiled modified objects: - lib/obj/Query_Tool_Handler.oo - lib/obj/MCP_Endpoint.oo - lib/obj/MCP_Thread.oo Resulting behavior expected on next test run: - If a target is non-executable, the response and logs will state exactly why (ONLINE/backend/auth reason), instead of a generic error. - Failure logs will also include the attempted SQL text (for tool calls that carry sql arguments), clarifying whether a backend query was actually executed or blocked before execution.	6 days ago
Rene Cannao	49f811a638	mcp query: stop misclassifying reachable targets as non-executable Observed failure: - MCP tools received valid target_id values, but /mcp/query returned 'Target is not executable by this handler'. - This prevented rule evaluation, digest accounting, and hit counters from advancing, causing multiple TAP phases to fail. Root cause addressed in this commit: - Target endpoint resolution required runtime server status to be exactly ONLINE. - In practice, reachable backends can temporarily appear in other statuses (for example monitor-related transitions), while MCP direct connections with profile credentials are still valid. - Protocol casing inconsistencies could also lead to fragile routing behavior when profile data is consumed. Changes: 1) Relax backend eligibility during endpoint resolution - In Query_Tool_Handler::refresh_target_registry(), endpoint discovery now accepts any server status except OFFLINE_HARD. - ONLINE is still preferred deterministically via ORDER BY, but non-ONLINE candidates remain eligible for MCP execution. 2) Normalize protocol values to lowercase - In MCP_Threads_Handler::load_target_auth_map(), target protocol is normalized to lowercase when loaded from runtime profiles. - In Query_Tool_Handler::refresh_target_registry(), protocol is normalized again defensively before routing decisions. 3) Improve diagnostics for target executability - Added explicit warnings when: - a target resolves to a backend but has empty db_username, - a target has no eligible backend for its hostgroup/protocol. - These logs make it clear why a target is being marked non-executable. Expected behavioral impact: - MCP query tools should execute for valid target_id entries even when monitor status is not strictly ONLINE. - Query-rules blocking/rewriting/OK actions should be exercised again. - stats_mcp_query_rules hits and stats_mcp_query_digest counters should resume incrementing once queries run. Validation performed: - Recompiled modified objects successfully: - lib/obj/Query_Tool_Handler.oo - lib/obj/MCP_Thread.oo	6 days ago
Rene Cannao	6a788e48c4	mcp: make /mcp/query self-healing when targets/backends appear after startup Problem addressed: - MCP query endpoint could stay unusable with 'Tool Handler not initialized' after restart/reload flows. - This was triggered when Query_Tool_Handler could not build an executable pool at init time (for example profiles loaded before ONLINE servers), leaving runtime commands like 'LOAD MCP QUERY RULES FROM MEMORY' blocked behind a NULL query tool handler. - Users had to manually toggle MCP enablement to recover, which is the opposite of expected self-healing behavior. What this commit changes: 1) Query_Tool_Handler pool init is now resilient and idempotent - init_connection_pool() now starts with close() so reinitialization fully resets stale mysql/pgsql pool state before rebuilding from runtime profile + server tables. - If no executable targets are available, init_connection_pool() now returns success with a warning instead of hard failure. This allows the query tool handler (and /mcp/query endpoint) to stay initialized even before backends are ready. 2) Lazy auto-rebuild on first query usage - get_connection() and get_pgsql_connection() now: - refresh target registry before resolution, - attempt to use an existing pooled connection, - if unavailable, trigger a full pool rebuild (init_connection_pool()) and retry once. - This provides automatic recovery when hostgroups/servers/profiles are loaded or changed after MCP startup, without requiring manual MCP disable/enable. 3) Admin runtime rule load path attempts MCP recovery - load_mcp_query_rules_to_runtime() now detects NULL query tool handler and calls load_mcp_server() once before failing. - This turns a hard, immediate admin error into a self-recovery attempt consistent with MCP runtime semantics. Behavioral impact: - /mcp/query endpoint remains online even when there are temporarily zero executable targets. - As soon as compatible runtime targets/backends exist, run_sql_readonly/explain_sql can recover automatically on demand. - LOAD MCP QUERY RULES TO/FROM RUNTIME no longer fails immediately on first NULL handler condition; it retries after MCP server recovery. Validation performed: - Recompiled modified objects successfully: - lib/obj/Query_Tool_Handler.oo - lib/obj/ProxySQL_Admin.oo - Full TAP runtime test execution is not possible in this sandbox due blocked local TCP socket creation; validation should be run in the normal test environment where ProxySQL/MySQL/PGSQL are reachable.	6 days ago
Rene Cannao	998bd82387	MCP TAP startup: fix tool-handler initialization order, improve MCP PROFILES observability, and seed monitor users This change fixes recurring MCP TAP failures where `/mcp/query` returned: Tool Handler not initialized for endpoint: query and where backend monitor auth failures flooded logs. Problem summary - MCP server startup can occur before runtime target/auth profiles and backend server mappings are loaded. - If that happens, Query_Tool_Handler initialization sees no executable targets and remains NULL. - MCP endpoint resources bind the handler pointer at creation time, so a NULL query handler at startup breaks `/mcp/query` until server restart. Code changes 1) Add explicit admin command logging for MCP PROFILES commands - Added `Received <command>` logging in the MCP PROFILES command block, matching behavior of other admin command handlers. - File: `lib/Admin_Handler.cpp` 2) Trigger MCP server refresh after `LOAD MCP PROFILES TO RUNTIME` - After copying profiles into runtime and rebuilding target/auth map, call `ProxySQL_Admin::load_mcp_server()`. - This allows MCP to self-heal when profiles become available after initial startup. - File: `lib/Admin_Handler.cpp` 3) Restart MCP server when query handler is missing - Extended `ProxySQL_Admin::load_mcp_server()` restart checks to include: - running server + `query_tool_handler == NULL` - Restart reason now includes tool handler initialization mismatch. - File: `lib/ProxySQL_Admin.cpp` 4) Fix TAP configurator load order to avoid early MCP startup - Reordered `test/tap/tests/mcp_rules_testing/configure_mcp.sh` runtime sequence: - `LOAD MYSQL SERVERS TO RUNTIME` - `LOAD PGSQL SERVERS TO RUNTIME` (best effort) - `LOAD MCP PROFILES TO RUNTIME` - `LOAD MCP VARIABLES TO RUNTIME` (last) - This ensures MCP starts only after routing/auth context is present. 5) Seed monitor credentials in AI local infra pre-hook - Added backend user/role creation for default monitor credentials `monitor/monitor`: - MySQL: create user + monitor-relevant grants - PostgreSQL: create role + `pg_monitor` + DB connect grants - Reduces monitor auth noise in local AI TAP dockerized setup. - File: `test/tap/groups/ai/pre-proxysql.bash` 6) Mark new TAP phase scripts executable - `test_phase10_eval_explain.sh` - `test_phase11_pgsql_target.sh` Expected outcome - MCP query endpoint no longer stays stuck with an uninitialized tool handler after TAP configuration. - MCP query-rules admin commands stop failing due to missing Query_Tool_Handler. - MCP profile command flow is visible in logs for easier debugging. - Local AI TAP infra no longer emits continuous monitor authentication failures for default monitor credentials.	6 days ago
Rene Cannao	af0411bd46	MCP: add target-aware rules/stats tests, explain_sql rule coverage, and AI local docker TAP infra This commit completes end-to-end MCP query-rules validation for multi-target routing and introduces a self-contained TAP infra for the `ai` group that can run both in Jenkins and manually. Main MCP/runtime changes: - Extended MCP query-rule matching context to include both `target_id` and resolved backend `username`. - Added `target_id` column to `mcp_query_rules` and `runtime_mcp_query_rules` table definitions. - Extended `stats_mcp_query_rules` to include `username` and `target_id` alongside `rule_id` and `hits`. - Updated load/save/runtime refresh paths to persist and rehydrate the expanded MCP rule schema. - Wired the rule engine into `explain_sql` so MCP rules apply consistently across `run_sql_readonly` and `explain_sql`. - Included startup-order fix in `src/main.cpp` to initialize MCP/GenAI thread handlers early, preventing startup crashes in PROXYSQLGENAI builds. Test coverage changes: - Updated existing MCP TAP phases to assert target-aware and username-aware behavior: - `test_phase4_stats.sh` - `test_phase6_eval_block.sh` - Added new MCP TAP phases: - `test_phase10_eval_explain.sh` (rule engine coverage for `explain_sql`) - `test_phase11_pgsql_target.sh` (pgsql target routing/rule/stats coverage; graceful skip if no pgsql target is configured) - Updated `test_mcp_query_rules-t.sh` runner to execute new phases. AI group isolated infra (manual + CI compatible): - Added `test/tap/groups/ai/docker-compose.yml` with MySQL 9.0 and PostgreSQL 16 backends. - Added lifecycle scripts: - `docker-compose-init.bash` - `docker-compose-destroy.bash` - `pre-proxysql.bash` - `post-proxysql.bash` - Extended `test/tap/groups/ai/env.sh` with local default ports/credentials/target IDs used by MCP TAP tests. - Added `test/tap/groups/ai/README.md` documenting manual execution flow outside Jenkins. Outcome: - MCP tests now validate routing-aware rule enforcement and stats attribution for both MySQL and PostgreSQL targets. - The `ai` TAP group can be run with an isolated local backend stack without relying on external Jenkins infra repositories.	6 days ago
René Cannaò	28b1114cd9	Merge pull request #5353 from sysown/v3.0_pgsql-prepared-statement-refcount-race-5352 Fix PostgreSQL prepared statement purge crash: use memory_order_acquire instead of memory_order_relaxed	6 days ago
René Cannaò	acd955b9d6	Merge pull request #5382 from sysown/v3.0-webui_2602 feat: add friend declaration for MetricsCollector to access MyHostGroups	6 days ago
Rene Cannao	b46fb575ea	TAP MCP client: port PR5372 coverage to MCP profiles/target_id routing model This commit ports the TAP MCP client test coverage from PR #5372 to the new MCP profiles architecture, replacing legacy direct backend variable assumptions with target/profile-based routing. What was updated: - Reworked test setup to use `mcp_target_profiles` + `mcp_auth_profiles` rather than legacy `mcp-mysql_*` config variables. - Updated MCP request payloads and helper flows to route via `target_id`. - Aligned assertions with profile-backed runtime behavior and removed obsolete expectations tied to static mysql host/user/password globals. Why this was needed: - The previous tests were written before MCP target/profile routing existed. - Without this update, TAP coverage would exercise deprecated paths and miss regressions in the current MCP execution model. Result: - Test suite now validates the same functional intent as PR #5372 but against the current profile-driven MCP routing implementation.	6 days ago
Rene Cannao	67cb1b72b5	MCP TAP: pass target_id explicitly in query-rules test payloads This commit updates MCP TAP query-rules tests to always include `target_id` in tool call payloads where query execution is expected. Details: - Added explicit `target_id` arguments to MCP `/mcp/query` tool invocations in rule evaluation tests. - Removed implicit dependence on server-selected defaults during test execution. Why this change is important: - Query-rules behavior is now route-aware and can depend on logical target selection. - Explicit routing in tests prevents non-determinism and ensures assertions are tied to the intended backend target. Result: - More deterministic TAP behavior and clearer validation of rule evaluation under target-based routing.	6 days ago
Rene Cannao	4a8b224038	MCP TAP/docs: migrate tests and documentation from legacy MCP mysql vars to profile-based routing This commit migrates MCP TAP assets and related docs away from deprecated `mcp-mysql_*` style configuration toward the profile-based model centered on auth profiles, target profiles, and `target_id` routing. Changes included: - Updated TAP configuration helpers and test inputs to build and load MCP auth/target profiles. - Replaced legacy variable-driven setup assumptions with runtime profile loading (`LOAD MCP PROFILES ...`). - Refreshed documentation snippets and examples to describe the profile-based configuration flow. Motivation: - Keep test and documentation surfaces aligned with the current MCP architecture. - Eliminate ambiguity between old POC paths and supported profile-driven routing. Outcome: - TAP and docs now consistently describe and exercise MCP routing through profile tables and `target_id` selection.	6 days ago
Rene Cannao	013864b36f	MCP: introduce profile-based target/auth routing and unified LOAD/SAVE MCP PROFILES commands This commit introduces the core MCP routing architecture that decouples client-facing target selection from backend connection details. Architecture introduced: - New MCP auth profile table to store backend credentials/policy (`mcp_auth_profiles`). - New MCP target profile table to map opaque `target_id` -> protocol/hostgroup/auth profile (`mcp_target_profiles`). - Runtime mirrors for both profile tables. Behavioral changes: - MCP query tools resolve execution context by `target_id` and route internally by protocol + hostgroup. - Client no longer needs backend-specific connection details; only logical target identifiers. - Unified admin commands for profile lifecycle management: - `LOAD MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` - `SAVE MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` Design goals: - Support multi-backend MCP execution (MySQL + PostgreSQL) behind one endpoint. - Provide server-side credential management and cleaner operational model. - Replace legacy one-backend POC configuration with scalable target/auth abstractions. Result: - MCP routing is now dynamic, protocol-agnostic at the client layer, and fully managed through profile tables and unified load/save commands.	6 days ago
René Cannaò	5de836a4c1	Merge pull request #5373 from sysown/v3.0_improve_mysql_monitoring_5256 Improve MySQL Monitor connection management	1 week ago
René Cannaò	3b578db4bf	Merge pull request #5374 from sysown/v3.0-test0213 Fix Valgrind-reported uninitialized memory issues	1 week ago
René Cannaò	83209da487	Merge pull request #5356 from sysown/v3.0-5355 Fix #5355: Add null pointer check in RequestEnd() to prevent use-after-free crash	1 week ago
Rene Cannao	37e72ea3f9	Fix PROXY protocol detection in MySQL_Data_Stream::read_from_net When pkts_recv==0, the code reads 4 bytes expecting a MySQL packet header, but PROXY protocol starts with 'PROXY ' which gets misinterpreted as a MySQL header with a huge packet length (0x504f5250 = ~1.3GB). Add check for 'PROX' prefix before parsing as MySQL header. This prevents passing an incorrect length parameter to recv() that exceeds the buffer size. Fixes #5376	1 week ago
Rene Cannao	11a43bf768	Fix uninitialized max_allowed_pkt in MySQL_Connection constructor Initialize options.max_allowed_pkt to 0 to prevent Valgrind errors when logging handshake debug information.	1 week ago
Rene Cannao	ca4f858b29	Fix uninitialized memory read in pgsql tokenizer Change malloc() to calloc() in get_result_buffer() to zero-initialize the query digest buffer before use. This prevents stage_2_parsing() from reading uninitialized memory when processing query digests. Fixes #5375	1 week ago
Rene Cannao	85952373d9	Fix uninitialized mondb pointer in MySQL_Monitor_State_Data constructor Add mondb = NULL initialization to prevent Valgrind errors when monitor threads access the pointer before it's properly set.	1 week ago
Rahim Kanji	d42d89ae36	Fix TAP test when statement count after worker threads is below cache limit. Calculate dynamically how many statements to create to ensure we exceed CACHE_LIMIT (1024) and trigger a purge.	1 week ago
Rahim Kanji	8ca6323eef	Merge remote-tracking branch 'Master/v3.0' into v3.0_pgsql-prepared-statement-refcount-race-5352	1 week ago
Rene Cannao	200fdecb8d	Apply AI agent review fixes to PR #5374 Critical Fixes: - Fix NULL pointer dereferences in MySQL_Logger.cpp: * Add null checks for myds->myconn->parent in log_audit_entry() * Fix cleanup code in log_request() and log_audit_entry() * Change sprintf to snprintf for buffer safety - Remove redundant pthread_rwlock_init in debug.cpp: * filters_rwlock already initialized with PTHREAD_RWLOCK_INITIALIZER * Re-initialization caused undefined behavior per POSIX Major Improvements: - Add bounded wait helper wait_for_glo_mth() in proxysql_utils.h: * Replaces indefinite while loops with 10s timeout * Applied to 26 locations across 4 files Build System: - Require C++17 in Makefile (remove C++11 fallback): * Code uses C++17 structured bindings extensively * Cleanest solution vs adding fallback guards to 27+ locations Compilation fixes: - Add unistd.h include for usleep - Remove conflicting GloMyMon extern declaration Files changed: - Makefile - include/proxysql_utils.h - lib/MySQL_Logger.cpp - lib/debug.cpp - lib/MySQL_Monitor.cpp - lib/ProxySQL_Admin.cpp - lib/ClickHouse_Server.cpp - src/SQLite3_Server.cpp	1 week ago
Rahim Kanji	1cccfb3220	Fix connection cleanup in failure paths	1 week ago
Rene Cannao	941e6b7320	Fix filters_rwlock initialization race condition Add static initialization PTHREAD_RWLOCK_INITIALIZER to filters_rwlock in debug.cpp to prevent race condition where proxy_debug() is called before init_debug_struct() initializes the rwlock. This fixes Valgrind errors in filter_debug_entry() where threads were accessing an uninitialized rwlock.	1 week ago
Rene Cannao	3c85c593cf	Add GloMTH initialization wait to all monitor and server threads Fix race condition where threads were accessing GloMTH before it was fully initialized, causing Valgrind errors in get_variable_int() and related functions. Added wait loop pattern (while GloMTH==NULL { usleep(50000); }) followed by usleep(100000) to ensure GloMTH is fully initialized before use: In lib/MySQL_Monitor.cpp: - monitor_connect(), monitor_ping(), monitor_read_only() - monitor_group_replication(), monitor_galera(), monitor_replication_lag() - monitor_AWS_Aurora_thread_HG(), monitor_aws_aurora() - init_mysql_thread_struct() In src/SQLite3_Server.cpp: - SQLite3_Server_session_handler In lib/ProxySQL_Admin.cpp: - admin_session_handler In lib/ClickHouse_Server.cpp: - ClickHouse_Server_session_handler The pattern ensures threads wait for GloMTH initialization during startup, then check for NULL during shutdown to exit cleanly.	1 week ago

1 2 3 4 5 ...

8525 Commits (v3.0_pgsql_advanced_logging) All Branches Search

8525 Commits (v3.0_pgsql_advanced_logging)

All Branches