13 Commits (v3.0_pgsql_sslkeylog_5281)

Author	SHA1	Message	Date
Wazir Ahmed	daa0b2037e	fix: Use consistent value representation in MCP query rules ... `mcp_query_rules` and `runtime_mcp_query_rules` uses different representations for the same fields, causing the web UI to always report config differences even after loading to runtime. - Use `-1` sentinel for nullable int fields (`flagOUT`, `timeout_ms`, `log`) so `NULL` values from main are preserved when written to runtime, following the `MySQL QP_rule_t` pattern. - Serialize `re_modifiers` bitmask back to `VARCHAR` (`1→"CASELESS"`) to match the format used in main table - Update `evaluate_mcp_query_rules()` to handle tri-state log field Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	2 months ago
Rene Cannao	9685cdaa4b	mcp discovery: enforce target-scoped run model and add protocol-aware static harvesting (mysql+pgsql) ... Refactor the MCP query/discovery stack to remove prototype-era MySQL-only assumptions and make discovery/catalog semantics explicitly target-scoped. This commit is intentionally not backward compatible with legacy single-target catalog metadata. Key implementation details: - Discovery_Schema: - runs table now stores target_id, protocol, and server_version - resolve_run_id() now requires target_id and resolves schemas within target scope - create_run() now records target/protocol/server version - added legacy schema detection + destructive catalog table rebuild when old runs layout is found - fixed resultset lifetime/escaping issues in target-aware run resolution - New PostgreSQL static harvester: - added PgSQL_Static_Harvester class and build wiring - harvests schemas/objects/columns/indexes/fks/view definitions into Discovery_Schema - maps pg metadata to object_id-based insert APIs used by Discovery_Schema - Query_Tool_Handler: - constructor simplified to catalog_path only (no mcp-mysql_* runtime ctor deps) - discovery.run_static now requires target_id and dispatches protocol-aware harvester at runtime - catalog.*, agent.run_start, and llm.* tool contracts now require target_id when resolving run_id - all run_id resolution call sites switched to resolve_run_id(target_id, ...) - list_schemas catalog query now scoped by target_id via runs join - digest logging path now resolves run_id in target context - ProxySQL_MCP_Server: - updated Query_Tool_Handler construction to new signature - Docs updated: - added explicit target_id scoping guidance for discovery/catalog/agent/llm workflows - clarified protocol-aware routing and noted legacy examples still present in large script docs Build validation: - Recompiled changed objects with PROXYSQLGENAI=1: - Query_Tool_Handler.oo - Discovery_Schema.oo - PgSQL_Static_Harvester.oo - ProxySQL_MCP_Server.oo	2 months ago
Rene Cannao	af0411bd46	MCP: add target-aware rules/stats tests, explain_sql rule coverage, and AI local docker TAP infra ... This commit completes end-to-end MCP query-rules validation for multi-target routing and introduces a self-contained TAP infra for the `ai` group that can run both in Jenkins and manually. Main MCP/runtime changes: - Extended MCP query-rule matching context to include both `target_id` and resolved backend `username`. - Added `target_id` column to `mcp_query_rules` and `runtime_mcp_query_rules` table definitions. - Extended `stats_mcp_query_rules` to include `username` and `target_id` alongside `rule_id` and `hits`. - Updated load/save/runtime refresh paths to persist and rehydrate the expanded MCP rule schema. - Wired the rule engine into `explain_sql` so MCP rules apply consistently across `run_sql_readonly` and `explain_sql`. - Included startup-order fix in `src/main.cpp` to initialize MCP/GenAI thread handlers early, preventing startup crashes in PROXYSQLGENAI builds. Test coverage changes: - Updated existing MCP TAP phases to assert target-aware and username-aware behavior: - `test_phase4_stats.sh` - `test_phase6_eval_block.sh` - Added new MCP TAP phases: - `test_phase10_eval_explain.sh` (rule engine coverage for `explain_sql`) - `test_phase11_pgsql_target.sh` (pgsql target routing/rule/stats coverage; graceful skip if no pgsql target is configured) - Updated `test_mcp_query_rules-t.sh` runner to execute new phases. AI group isolated infra (manual + CI compatible): - Added `test/tap/groups/ai/docker-compose.yml` with MySQL 9.0 and PostgreSQL 16 backends. - Added lifecycle scripts: - `docker-compose-init.bash` - `docker-compose-destroy.bash` - `pre-proxysql.bash` - `post-proxysql.bash` - Extended `test/tap/groups/ai/env.sh` with local default ports/credentials/target IDs used by MCP TAP tests. - Added `test/tap/groups/ai/README.md` documenting manual execution flow outside Jenkins. Outcome: - MCP tests now validate routing-aware rule enforcement and stats attribution for both MySQL and PostgreSQL targets. - The `ai` TAP group can be run with an isolated local backend stack without relying on external Jenkins infra repositories.	2 months ago
Rene Cannao	48bc7dd7bf	Merge v4.0 GenAI features into v3.0 with conditional compilation ... This commit merges the experimental v4.0 GenAI/MCP features into the stable v3.0 branch using conditional compilation. All v4.0 features are disabled by default and only enabled when PROXYSQLGENAI=1 is set at compile time. Changes: Build System: - Modified main Makefile to pass PROXYSQLGENAI flag to sub-makefiles - Modified deps/Makefile to conditionally build sqlite-vec and sqlite-rembed - Modified lib/Makefile to add PSQLGA flag and include GenAI object files - Modified src/Makefile to add PSQLGA flag and conditional linking Headers (wrapped with #ifdef PROXYSQLGENAI): - All 20 new GenAI header files in include/ - Modified cpp.h, proxysql_glovars.hpp, proxysql_admin.h - Modified ProxySQL_Admin_Tables_Definitions.h for GenAI/MCP tables Source Files: - All 22 new GenAI source files in lib/ wrapped with #ifdef PROXYSQLGENAI - Modified src/main.cpp for conditional global variables and init/shutdown - Modified Admin_Handler.cpp for conditional command handlers - Modified Admin_Bootstrap.cpp for conditional table registration - Modified Admin_FlushVariables.cpp for conditional variable flushing - Modified ProxySQL_Admin.cpp for conditional admin methods - Modified ProxySQL_Admin_Stats.cpp for conditional MCP stats functions - Modified proxy_sqlite3_symbols.cpp to always compile (needed by core) - Modified MySQL_Session.cpp for conditional GenAI function calls Test Files: - Renamed test_mcp_query_rules-t to test_mcp_query_rules-t.sh - Renamed test_mcp_rag_metrics-t to test_mcp_rag_metrics-t.sh - Modified anomaly_detection-t.cpp for conditional test execution Usage: # Build without GenAI (v3.0 mode - default) make clean && make build_deps -j$(nproc) && make build_lib -j$(nproc) && make build_src -j$(nproc) # Build with GenAI (v4.0 mode) make clean && PROXYSQLGENAI=1 make build_deps -j$(nproc) && PROXYSQLGENAI=1 make build_lib -j$(nproc) && PROXYSQLGENAI=1 make build_src -j$(nproc)	3 months ago
Javier Jaramago Fernández	a5ef787c7e	feat: Improve logging for MCP and RAG tools ... - Add new column 'endpoint' for 'stats_mcp_query_tools_counters'. - Add new table 'rag_search_log' to log 'log_rag_search_fts' operations.	3 months ago
Rene Cannao	f01fc79584	feat: Add runtime_mcp_query_rules table and fix stats_mcp_query_rules schema ... - Add ADMIN_SQLITE_TABLE_RUNTIME_MCP_QUERY_RULES schema (17 columns, same as mcp_query_rules) - Fix STATS_SQLITE_TABLE_MCP_QUERY_RULES to only have rule_id and hits columns - Add runtime_mcp_query_rules detection and refresh in ProxySQL_Admin - Implement save_mcp_query_rules_from_runtime(bool _runtime) for both config and runtime tables - Update get_mcp_query_rules() to return 17 columns (no hits) - get_stats_mcp_query_rules() returns 2 columns (rule_id, hits) Mirrors the MySQL query rules pattern: - mcp_query_rules: config table (17 cols) - runtime_mcp_query_rules: runtime state (17 cols) - stats_mcp_query_rules: hit counters (2 cols)	3 months ago
Rene Cannao	5b502c0864	feat: Add question learning capability to demo agent ... Add ability for the demo agent to learn new questions and add them to the catalog, making it smarter over time. Changes: - Added get_last_agent_run_id() function to Discovery_Schema: - Queries agent_runs table for the most recent agent_run_id for a run_id - Returns 0 if no agent runs exist for the schema - Updated llm.question_template_add handler: - Made agent_run_id optional (defaults to 0 when not provided) - When agent_run_id <= 0, auto-fetches last agent_run_id for the schema - Returns helpful error if no agent run exists for the schema - Returns agent_run_id in response for visibility - Updated llm.question_template_add tool schema: - Moved agent_run_id from required to optional parameters - Updated description to explain auto-fetch behavior - Updated demo_agent_claude.sh prompt: - Added llm.question_template_add to available tools - Added Step 4: "Learn from Success" to workflow - Added explicit instruction to ALWAYS LEARN new questions - Added example showing learning workflow - Expanded from 4 steps to 5 steps to include learning Now the demo agent can: 1. Search for existing questions 2. Reuse SQL if a good match exists 3. Generate new SQL if no good match 4. LEARN new questions by adding them to the catalog 5. Present results This enables continuous learning - the more users interact with it, the smarter it becomes.	3 months ago
Rene Cannao	ee13e4bf13	feat: Add include_objects parameter to llm_search for complete object retrieval ... Enhance the llm_search MCP tool to return complete question template data and optionally include full object schemas, reducing the need for additional MCP calls when answering questions. Changes: - Added related_objects column to llm_question_templates table - Updated add_question_template() to accept and store related_objects JSON array - Enhanced fts_search_llm() with include_objects parameter: - LEFT JOIN with llm_question_templates to return example_sql, related_objects, template_json, and confidence - When include_objects=true, fetches full object schemas (columns, indexes) for all related objects in a single batch operation - Added error checking for SQL execution failures - Fixed fts_search_llm() get_object() call to pass schema_name and object_name separately instead of combined object_key - Updated Query_Tool_Handler: - Added is_boolean() handling to json_int() helper to properly convert JSON boolean true/false to int 1/0 - Updated llm.search handler to extract and pass include_objects parameter - Updated llm.question_template_add to extract and pass related_objects - Updated tool schemas to document new parameters This change allows agents to get all necessary schema information in a single llm_search call instead of making multiple catalog_get_object calls, significantly reducing MCP call overhead.	3 months ago
Rene Cannao	5668c86809	fix: Implement FTS indexing for LLM artifacts and fix reserved keyword issue ... - Rename llm_search_log column from \"limit\" to \"lmt\" to avoid SQL reserved keyword - Add FTS inserts to all LLM artifact upsert functions: - add_question_template(): index question templates for search - add_llm_note(): index notes for search - upsert_llm_summary(): index object summaries for search - upsert_llm_domain(): index domains for search - upsert_llm_metric(): index metrics for search - Remove content='' from fts_llm table to store content directly - Add <functional> header for std::hash usage This fixes the bug where llm_search always returned empty results because the FTS index was never populated.	3 months ago
Rene Cannao	2250b762a3	feat: Add query_tool_calls table to log MCP tool invocations ... Add query_tool_calls table to Discovery Schema to track all MCP tool invocations via the /mcp/query/ endpoint. Logs: - tool_name: Name of the tool that was called - schema: Schema name (nullable, empty if not applicable) - run_id: Run ID from discovery (nullable, 0 if not applicable) - start_time: Start monotonic time in microseconds - execution_time: Execution duration in microseconds - error: Error message (null if success) Modified files: - Discovery_Schema.cpp: Added table creation and log_query_tool_call function - Discovery_Schema.h: Added function declaration - Query_Tool_Handler.cpp: Added logging after each tool execution	3 months ago
Rene Cannao	a816a756d4	feat: Add MCP query tool usage counters to stats schema ... Add stats_mcp_query_tools_counters and stats_mcp_query_tools_counters_reset tables to track MCP query tool usage statistics. - Added get_tool_usage_stats_resultset() method to Query_Tool_Handler - Defined table schemas in ProxySQL_Admin_Tables_Definitions.h - Registered tables in Admin_Bootstrap.cpp - Added pattern matching in ProxySQL_Admin.cpp - Added stats___mcp_query_tools_counters() in ProxySQL_Admin_Stats.cpp - Fixed friend declaration for track_tool_invocation() - Fixed Discovery_Schema.cpp log_llm_search() to use prepare_v2/finalize	3 months ago
Rene Cannao	623675b369	feat: Add schema name resolver and deprecate direct DB tools ... - Add resolve_run_id() to map schema names to latest run_id - Update all catalog and LLM tools to accept schema names - Deprecate describe_table, table_profile, column_profile - Deprecate get_constraints, suggest_joins, find_reference_candidates - Keep sample_rows, sample_distinct for data preview	3 months ago
Rene Cannao	6f23d5bcd0	feat: Implement two-phase schema discovery architecture ... Phase 1 (Static/Deterministic): - Add Discovery_Schema: SQLite catalog with deterministic and LLM tables - Add Static_Harvester: MySQL INFORMATION_SCHEMA metadata extraction - Harvest schemas, objects, columns, indexes, foreign keys, view definitions - Compute derived hints: is_time, is_id_like, has_pk, has_fks, has_time - Build quick profiles and FTS5 indexes Phase 2 (LLM Agent): - Add 19 new MCP tools for two-phase discovery - discovery.run_static: Trigger ProxySQL's static harvest - Catalog tools: init, search, get_object, list_objects, get_relationships - Agent tools: run_start, run_finish, event_append - LLM tools: summary_upsert, relationship_upsert, domain_upsert, etc. Files: - include/Discovery_Schema.h, lib/Discovery_Schema.cpp - include/Static_Harvester.h, lib/Static_Harvester.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp (updated) - lib/Makefile (updated) - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/ - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py	3 months ago