15 Commits (ade0130e67e88b9052d0dc4e76d48d98d3328485)

Author	SHA1	Message	Date
Rene Cannao	af0411bd46	MCP: add target-aware rules/stats tests, explain_sql rule coverage, and AI local docker TAP infra ... This commit completes end-to-end MCP query-rules validation for multi-target routing and introduces a self-contained TAP infra for the `ai` group that can run both in Jenkins and manually. Main MCP/runtime changes: - Extended MCP query-rule matching context to include both `target_id` and resolved backend `username`. - Added `target_id` column to `mcp_query_rules` and `runtime_mcp_query_rules` table definitions. - Extended `stats_mcp_query_rules` to include `username` and `target_id` alongside `rule_id` and `hits`. - Updated load/save/runtime refresh paths to persist and rehydrate the expanded MCP rule schema. - Wired the rule engine into `explain_sql` so MCP rules apply consistently across `run_sql_readonly` and `explain_sql`. - Included startup-order fix in `src/main.cpp` to initialize MCP/GenAI thread handlers early, preventing startup crashes in PROXYSQLGENAI builds. Test coverage changes: - Updated existing MCP TAP phases to assert target-aware and username-aware behavior: - `test_phase4_stats.sh` - `test_phase6_eval_block.sh` - Added new MCP TAP phases: - `test_phase10_eval_explain.sh` (rule engine coverage for `explain_sql`) - `test_phase11_pgsql_target.sh` (pgsql target routing/rule/stats coverage; graceful skip if no pgsql target is configured) - Updated `test_mcp_query_rules-t.sh` runner to execute new phases. AI group isolated infra (manual + CI compatible): - Added `test/tap/groups/ai/docker-compose.yml` with MySQL 9.0 and PostgreSQL 16 backends. - Added lifecycle scripts: - `docker-compose-init.bash` - `docker-compose-destroy.bash` - `pre-proxysql.bash` - `post-proxysql.bash` - Extended `test/tap/groups/ai/env.sh` with local default ports/credentials/target IDs used by MCP TAP tests. - Added `test/tap/groups/ai/README.md` documenting manual execution flow outside Jenkins. Outcome: - MCP tests now validate routing-aware rule enforcement and stats attribution for both MySQL and PostgreSQL targets. - The `ai` TAP group can be run with an isolated local backend stack without relying on external Jenkins infra repositories.	6 days ago
Rene Cannao	4a8b224038	MCP TAP/docs: migrate tests and documentation from legacy MCP mysql vars to profile-based routing ... This commit migrates MCP TAP assets and related docs away from deprecated `mcp-mysql_*` style configuration toward the profile-based model centered on auth profiles, target profiles, and `target_id` routing. Changes included: - Updated TAP configuration helpers and test inputs to build and load MCP auth/target profiles. - Replaced legacy variable-driven setup assumptions with runtime profile loading (`LOAD MCP PROFILES ...`). - Refreshed documentation snippets and examples to describe the profile-based configuration flow. Motivation: - Keep test and documentation surfaces aligned with the current MCP architecture. - Eliminate ambiguity between old POC paths and supported profile-driven routing. Outcome: - TAP and docs now consistently describe and exercise MCP routing through profile tables and `target_id` selection.	6 days ago
Rene Cannao	013864b36f	MCP: introduce profile-based target/auth routing and unified LOAD/SAVE MCP PROFILES commands ... This commit introduces the core MCP routing architecture that decouples client-facing target selection from backend connection details. Architecture introduced: - New MCP auth profile table to store backend credentials/policy (`mcp_auth_profiles`). - New MCP target profile table to map opaque `target_id` -> protocol/hostgroup/auth profile (`mcp_target_profiles`). - Runtime mirrors for both profile tables. Behavioral changes: - MCP query tools resolve execution context by `target_id` and route internally by protocol + hostgroup. - Client no longer needs backend-specific connection details; only logical target identifiers. - Unified admin commands for profile lifecycle management: - `LOAD MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` - `SAVE MCP PROFILES TO/FROM MEMORY/RUNTIME/DISK` Design goals: - Support multi-backend MCP execution (MySQL + PostgreSQL) behind one endpoint. - Provide server-side credential management and cleaner operational model. - Replace legacy one-backend POC configuration with scalable target/auth abstractions. Result: - MCP routing is now dynamic, protocol-agnostic at the client layer, and fully managed through profile tables and unified load/save commands.	6 days ago
Rene Cannao	b4f521c634	Merge v3.1-MCP2 into v3.1-vec ... Conflict resolution summary: **Features preserved from v3.1-vec:** - mcp_use_ssl variable (HTTP/HTTPS mode support) - RAG tool handler support - FTS (Full Text Search) functionality in MySQL_Tool_Handler **Changes accepted from v3.1-MCP2:** - Schema isolation for catalog (removed mcp_catalog_path/fts_path variables) - Schema parameter added to catalog functions - Comprehensive query rewriting improvements in Query_Tool_Handler - Format string fixes in vector_db_performance-t.cpp **Key decisions:** - catalog_path/fts_path: Removed per v3.1-MCP2 commit 35b0b224 (intentional) - MySQL_Catalog.cpp: Accepted v3.1-MCP2 version (schema isolation improvements) - Query_Tool_Handler.cpp: Accepted v3.1-MCP2 version (significant improvements) - Makefile: Kept v3.1-vec's EXCLUDE_TRACKING_VARIABLES (correct spelling) **Build notes:** - MySQL_Tool_Handler now requires schema parameter for catalog operations - Catalog path is hardcoded to datadir/mcp_catalog.db - FTS functionality preserved but may need updates for schema isolation	1 month ago
Rene Cannao	02918d18b8	Fix PR #25 Review: All AI code reviewer feedback addressed ... This commit addresses all recommendations from CodeRabbit, Gemini Code Assist, and Copilot for PR #25 (FTS security and code quality improvements). Critical Security Fixes: - MCP_Thread.cpp: Rollback fts_path on reset failure to keep config consistent - MySQL_FTS.cpp: Add escape_mysql_identifier() for MySQL query identifier escaping - MySQL_FTS.cpp: Add unique hash-based fallback to sanitize_name() for empty strings - MySQL_FTS.cpp: Add where_clause validation to block dangerous SQL patterns Memory Safety Fixes: - MySQL_FTS.cpp: Fix indexes_result memory leak on early return in search() - MySQL_FTS.h: Delete copy/move operations to prevent accidental resource duplication Thread Safety Documentation: - MySQL_Tool_Handler.cpp: Add comment explaining FTS lock design rationale Test Script Improvements: - test_mcp_fts.sh: Add curl timeouts (5s connect, 30s max) - test_mcp_fts.sh: Remove unused delete_response variable - test_mcp_fts_detailed.sh: Make cleanup tolerant of non-existent indexes Build Fixes: - Makefile: Fix EXCLUDE_TRACKING_VARAIABLES typo to EXCLUDE_TRACKING_VARIABLES - vector_db_performance-t.cpp: Fix printf format specifiers to %lld with cast Schema Fixes: - Query_Tool_Handler.cpp: Change fts_index_table columns schema from string to array Code Cleanup: - MySQL_Tool_Handler.cpp: Remove all remaining debug fprintf statements (34 lines) - Documentation: Change "Full Text" to "Full-Text" (hyphenated) Total: ~50 fixes across 10 files	1 month ago
Rene Cannao	a10c09bcc9	Fix PR #21 review: Security, memory safety, thread safety, and code cleanup ... Security fixes: - Add escape_identifier() helper for proper SQLite identifier escaping - Replace sanitize_name() with allowlist validation (ASCII letters, digits, underscore only) - Fix MATCH clause FTS5 operator injection by wrapping query in double quotes - Apply escape_identifier() to all DDL statements (CREATE, DROP, triggers) Memory safety fixes: - Replace VLA with std::vector in MySQL_FTS::init(), add delete on error path - Fix memory leak: free error string before return in list_indexes() - Fix reindex_json["error"] potential exception using .value() with default Thread safety fixes: - reinit_fts(): Add mutex lock around pointer swap - reset_fts_path(): Move blocking init() outside lock, only swap pointer under lock Code cleanup: - Remove 7 debug fprintf statements from Query_Tool_Handler.cpp - Remove unused #include <memory> from MySQL_FTS.h Test script security fixes: - Use MYSQL_PWD environment variable instead of -p"..." for password - Add escape_sql() function and apply to INSERT statement - Fix CURL_OPTS quoting: ${CURL_OPTS:+"${CURL_OPTS}"} - Remove unused FTS_INDEX_NAME and SEARCH_QUERIES variables Documentation fixes: - Fix bare URL to markdown link format - Add code block language identifiers (text, bash)	1 month ago
Rahim Kanji	8e2230c3e2	Add FTS_User_Guide.md	1 month ago
Rene Cannao	aced263367	docs: Update MCP documentation to reflect current implementation ... This commit completes a comprehensive review and update of all MCP-related documentation: - Remove deprecated FastAPI POC files - Update Architecture.md to reflect implemented multi-endpoint architecture - Update VARIABLES.md with new AI endpoint authentication - Update Tool_Discovery_Guide.md with discovery and LLM tools - Update FTS_Implementation_Plan.md to reflect implemented status - Update Vector_Embeddings_Implementation_Plan.md to show planned status - Update Database_Discovery_Agent.md to clarify conceptual design status - Update scripts/mcp/README.md with current multi-handler architecture - Update STDIO_BRIDGE_README.md with complete tool list All documentation now accurately reflects the current ProxySQL MCP implementation with 6 dedicated tool handlers and two-phase discovery.	1 month ago
Rene Cannao	35b0b224ff	refactor: Remove mcp-catalog_path variable and hardcode catalog path ... Remove the mcp-catalog_path configuration variable and hardcode the catalog database path to datadir/mcp_catalog.db for stability. Rationale: The catalog database is session state, not user configuration. Runtime swapping of the catalog could cause tables to be missed and the catalog to fail even if it was succeeding a second earlier. Changes: - Removed catalog_path from mcp_thread_variables_names array - Removed mcp_catalog_path from MCP_Thread variables struct - Removed getter/setter logic for catalog_path - Hardcoded catalog path to GloVars.datadir/mcp_catalog.db in: - ProxySQL_MCP_Server.cpp (Query_Tool_Handler initialization) - Admin_FlushVariables.cpp (MySQL_Tool_Handler reinitialization) - Updated VARIABLES.md to document the hardcoded path - Updated configure_mcp.sh to remove catalog_path configuration - Updated MCP README to remove catalog_path references	1 month ago
Rene Cannao	2ef44e7c3e	Add MCP implementation plans for FTS and Vector Embeddings ... Comprehensive implementation documentation for two new search capabilities: FTS (Full Text Search): - 6 tools for lexical search using SQLite FTS5 - Separate mcp_fts.db database - Keyword matching and phrase search - Tools: fts_index_table, fts_search, fts_list_indexes, fts_delete_index, fts_reindex, fts_rebuild_all Vector Embeddings: - 6 tools for semantic search using sqlite-vec - Separate mcp_embeddings.db database - Vector similarity search with sqlite-rembed integration - Placeholder for future GenAI module - Tools: embed_index_table, embed_search, embed_list_indexes, embed_delete_index, embed_reindex, embed_rebuild_all Both systems: - Follow MySQL_Catalog patterns for SQLite management - Integrate with existing MCP Query endpoint - Work alongside Catalog for AI agent memory - 13-step implementation plans with detailed code examples	1 month ago
Rene Cannao	07dc887af2	Add MCP Tool Discovery Guide ... Comprehensive guide for discovering and using MCP Query endpoint tools: - Tool discovery via tools/list method - Complete list of all Query endpoint tools with parameters - cURL and Python examples for tool discovery and execution - Complete database exploration example - Test script usage guide	1 month ago
Rene Cannao	5846cd8b40	Add Database Discovery Agent architecture documentation ... Comprehensive architecture for AI-powered database discovery agent: - Mixture-of-experts approach (Structural, Statistical, Semantic, Query) - Orchestrator pattern for coordinated exploration - Four-phase discovery process (Blind → Patterns → Hypotheses → Synthesis) - Domain-agnostic design for any database complexity or type - Catalog as shared memory for collaborative expert findings - Multiple real-world examples (law firm, scientific research, ecommerce)	1 month ago
Rene Cannao	ced10dd054	Implement per-endpoint authentication for MCP endpoints ... This commit implements Phase 2 of the MCP multi-endpoint architecture: per-endpoint Bearer token authentication. ## Changes ### lib/MCP_Endpoint.cpp - Implemented `authenticate_request()` method with: - Per-endpoint token validation (mcp-{endpoint}_endpoint_auth) - Bearer token support via Authorization header - Query parameter fallback (?token=xxx) for simple testing - No authentication when token is not configured (backward compatible) - Proper 401 Unauthorized response on auth failure - Token whitespace trimming - Debug logging for troubleshooting ### doc/MCP/Architecture.md - Updated Per-Endpoint Authentication section with complete implementation - Marked Phase 3 authentication task as completed (✅) - Added authentication implementation code example ## Authentication Flow 1. Client sends request with Bearer token: - Header: `Authorization: Bearer <token>` - Or query param: `?token=<token>` 2. Server validates against endpoint-specific variable: - `/mcp/config` → `mcp-config_endpoint_auth` - `/mcp/observe` → `mcp-observe_endpoint_auth` - `/mcp/query` → `mcp-query_endpoint_auth` - `/mcp/admin` → `mcp-admin_endpoint_auth` - `/mcp/cache` → `mcp-cache_endpoint_auth` 3. Returns 401 Unauthorized if: - Auth is required but not provided - Token doesn't match expected value 4. Allows request if: - No auth token configured (backward compatible) - Token matches expected value ## Testing ```bash # Set auth token for /mcp/query endpoint mysql -h 127.0.0.1 -P 6032 -u admin -padmin \ -e "SET mcp-query_endpoint_auth='my-secret-token'; LOAD MCP VARIABLES TO RUNTIME;" # Test with Bearer token curl -k -X POST https://127.0.0.1:6071/mcp/query \ -H "Content-Type: application/json" \ -H "Authorization: Bearer my-secret-token" \ -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' # Test with query parameter curl -k -X POST "https://127.0.0.1:6071/mcp/query?token=my-secret-token" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' ``` ## Status ✅ Authentication fully implemented and functional ⚠️ Testing with running ProxySQL instance still needed Co-authored-by: Claude <claude@anthropic.com>	1 month ago
Rene Cannao	c86a048d9c	Implement MCP multi-endpoint architecture with dedicated tool handlers ... This commit implements Option 1 (Multiple Tool Handlers) for the MCP module, where each of the 5 endpoints has its own dedicated tool handler with specific tools. ## Architecture Changes - Created MCP_Tool_Handler base class interface for all tool handlers - Each endpoint now has its own dedicated tool handler: - /mcp/config → Config_Tool_Handler (configuration management) - /mcp/query → Query_Tool_Handler (database exploration) - /mcp/admin → Admin_Tool_Handler (administrative operations) - /mcp/cache → Cache_Tool_Handler (cache management) - /mcp/observe → Observe_Tool_Handler (monitoring & metrics) ## New Files Base Interface: - include/MCP_Tool_Handler.h - Base class for all tool handlers Tool Handlers: - include/Config_Tool_Handler.h, lib/Config_Tool_Handler.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp - include/Admin_Tool_Handler.h, lib/Admin_Tool_Handler.cpp - include/Cache_Tool_Handler.h, lib/Cache_Tool_Handler.cpp - include/Observe_Tool_Handler.h, lib/Observe_Tool_Handler.cpp Documentation: - doc/MCP/Architecture.md - Comprehensive architecture documentation ## Modified Files - include/MCP_Thread.h, lib/MCP_Thread.cpp - Added 5 tool handler pointers - include/MCP_Endpoint.h, lib/MCP_Endpoint.cpp - Use tool_handler base class - lib/ProxySQL_MCP_Server.cpp - Create and pass handlers to endpoints - lib/Makefile - Added new source files ## Implementation Status - Config_Tool_Handler: Functional (get_config, set_config, list_variables, get_status) - Query_Tool_Handler: Functional (wraps MySQL_Tool_Handler, all 18 tools) - Admin_Tool_Handler: Stub implementations (TODO: implement) - Cache_Tool_Handler: Stub implementations (TODO: implement) - Observe_Tool_Handler: Stub implementations (TODO: implement) See GitHub Issue #8 for detailed TODO list. Co-authored-by: Claude <claude@anthropic.com>	1 month ago
Rene Cannao	a5f712e7d9	Add MCP variables documentation ... Added comprehensive documentation for all 14 MCP module configuration variables: Server Configuration: - mcp-enabled (boolean, default: false) - mcp-port (integer, default: 6071) - mcp-timeout_ms (integer, default: 30000) Endpoint Authentication (5 variables): - mcp-config_endpoint_auth (string, default: "") - mcp-observe_endpoint_auth (string, default: "") - mcp-query_endpoint_auth (string, default: "") - mcp-admin_endpoint_auth (string, default: "") - mcp-cache_endpoint_auth (string, default: "") MySQL Tool Handler Configuration (5 variables): - mcp-mysql_hosts (string, default: "127.0.0.1") - mcp-mysql_ports (string, default: "3306") - mcp-mysql_user (string, default: "") - mcp-mysql_password (string, default: "") - mcp-mysql_schema (string, default: "") Catalog Configuration: - mcp-catalog_path (string, default: "mcp_catalog.db") Includes documentation for management commands, variable persistence, status variables, and security considerations.	1 month ago