proxysql

Commit Graph

Author	SHA1	Message	Date
Javier Jaramago Fernández	51e91378c4	test: Add/Improve TAP tests for MCP query_rules/digests	4 weeks ago
Rene Cannao	f2536f01d2	Merge v3.1-vec into v3.1-MCP2_QR Resolved merge conflict in lib/ProxySQL_Admin_Stats.cpp: - Kept v3.1-MCP2_QR version with proper NULL handling - Uses stmt_unique_ptr (RAII smart pointer) - Checks for NULL before calling atoll() on numeric fields - Discarded v3.1-vec version which lacked NULL checks The merged code correctly handles NULL values for: - run_id - count_star - first_seen - last_seen - sum_time - min_time - max_time	4 weeks ago
Rene Cannao	5d4318b547	fix: Address coderabbitai review concerns for PR #27 - Fix Discovery_Schema.cpp fingerprint output JSON consistency - Quote placeholders (e.g., "?" instead of ?) for valid JSON - Fix test_mcp_query_rules_block.sh exec_admin_silent function - Add -B and -N flags for batch mode with no headers - Remove unused YELLOW variable from test_phase1_crud.sh - Fix test_phase2_load_save.sh runtime DELETE to be non-fatal - Remove unused INITIAL_COUNT from test_phase3_runtime.sh - Fix test_phase3_runtime.sh to verify exact ID ordering - Fix test_phase4_stats.sh read-only test assertion - Make INSERT/DELETE non-fatal and fix assertion logic - Remove unused DIGEST_TEXT_CHECK from test_phase5_digest.sh - Fix test_phase8_eval_timeout.sh non-timeout response handling - Return failure for non-timeout responses - Remove stray exit 1 from test_phase8_eval_timeout.sh	4 weeks ago
Javier Jaramago Fernández	52142c4648	fix: Multiple issues with MCP query_(rules/digests) - Fixed invalid memory accesses for tables: + mcp_query_rules + stats_mcp_query_rules + stats_mcp_query_digests - Fixed inactive 'mcp_query_rules' being loaded to runtime. - Fixed hash computation in 'compute_mcp_digest'. - Fixed invalid escaping during 'stats_mcp_query_digests' gen. - Fixed digest generation for MCP arguments: + SQL queries are now preserved using 'mysql_query_digest_and_first_comment'. + TODO: Options for the tokenizer are right now hardcoded. - Added initial testing and testing plan for MCP query_(rules/digests). + TODO: Test finished on phase8. Timeouts destroy the MCP connection, leaving it unusable for subsequent queries this should be fixed for continuing testing. - TODO: There are several limitations to fix in 'validate_readonly_query'. This reflect in some query hacks in the testing. + 'SELECT' starting with comments (--) gets flagged as non-read. + 'SELECT' must have a 'SELECT .* FROM' structure. While common, simple testing queries many times lack this form.	1 month ago
Rene Cannao	b4f521c634	Merge v3.1-MCP2 into v3.1-vec Conflict resolution summary: Features preserved from v3.1-vec: - mcp_use_ssl variable (HTTP/HTTPS mode support) - RAG tool handler support - FTS (Full Text Search) functionality in MySQL_Tool_Handler Changes accepted from v3.1-MCP2: - Schema isolation for catalog (removed mcp_catalog_path/fts_path variables) - Schema parameter added to catalog functions - Comprehensive query rewriting improvements in Query_Tool_Handler - Format string fixes in vector_db_performance-t.cpp Key decisions: - catalog_path/fts_path: Removed per v3.1-MCP2 commit `35b0b224` (intentional) - MySQL_Catalog.cpp: Accepted v3.1-MCP2 version (schema isolation improvements) - Query_Tool_Handler.cpp: Accepted v3.1-MCP2 version (significant improvements) - Makefile: Kept v3.1-vec's EXCLUDE_TRACKING_VARIABLES (correct spelling) Build notes: - MySQL_Tool_Handler now requires schema parameter for catalog operations - Catalog path is hardcoded to datadir/mcp_catalog.db - FTS functionality preserved but may need updates for schema isolation	1 month ago
Rene Cannao	02918d18b8	Fix PR #25 Review: All AI code reviewer feedback addressed This commit addresses all recommendations from CodeRabbit, Gemini Code Assist, and Copilot for PR #25 (FTS security and code quality improvements). Critical Security Fixes: - MCP_Thread.cpp: Rollback fts_path on reset failure to keep config consistent - MySQL_FTS.cpp: Add escape_mysql_identifier() for MySQL query identifier escaping - MySQL_FTS.cpp: Add unique hash-based fallback to sanitize_name() for empty strings - MySQL_FTS.cpp: Add where_clause validation to block dangerous SQL patterns Memory Safety Fixes: - MySQL_FTS.cpp: Fix indexes_result memory leak on early return in search() - MySQL_FTS.h: Delete copy/move operations to prevent accidental resource duplication Thread Safety Documentation: - MySQL_Tool_Handler.cpp: Add comment explaining FTS lock design rationale Test Script Improvements: - test_mcp_fts.sh: Add curl timeouts (5s connect, 30s max) - test_mcp_fts.sh: Remove unused delete_response variable - test_mcp_fts_detailed.sh: Make cleanup tolerant of non-existent indexes Build Fixes: - Makefile: Fix EXCLUDE_TRACKING_VARAIABLES typo to EXCLUDE_TRACKING_VARIABLES - vector_db_performance-t.cpp: Fix printf format specifiers to %lld with cast Schema Fixes: - Query_Tool_Handler.cpp: Change fts_index_table columns schema from string to array Code Cleanup: - MySQL_Tool_Handler.cpp: Remove all remaining debug fprintf statements (34 lines) - Documentation: Change "Full Text" to "Full-Text" (hyphenated) Total: ~50 fixes across 10 files	1 month ago
Rene Cannao	a10c09bcc9	Fix PR #21 review: Security, memory safety, thread safety, and code cleanup Security fixes: - Add escape_identifier() helper for proper SQLite identifier escaping - Replace sanitize_name() with allowlist validation (ASCII letters, digits, underscore only) - Fix MATCH clause FTS5 operator injection by wrapping query in double quotes - Apply escape_identifier() to all DDL statements (CREATE, DROP, triggers) Memory safety fixes: - Replace VLA with std::vector in MySQL_FTS::init(), add delete on error path - Fix memory leak: free error string before return in list_indexes() - Fix reindex_json["error"] potential exception using .value() with default Thread safety fixes: - reinit_fts(): Add mutex lock around pointer swap - reset_fts_path(): Move blocking init() outside lock, only swap pointer under lock Code cleanup: - Remove 7 debug fprintf statements from Query_Tool_Handler.cpp - Remove unused #include <memory> from MySQL_FTS.h Test script security fixes: - Use MYSQL_PWD environment variable instead of -p"..." for password - Add escape_sql() function and apply to INSERT statement - Fix CURL_OPTS quoting: ${CURL_OPTS:+"${CURL_OPTS}"} - Remove unused FTS_INDEX_NAME and SEARCH_QUERIES variables Documentation fixes: - Fix bare URL to markdown link format - Add code block language identifiers (text, bash)	1 month ago
René Cannaò	a831670a79	Merge pull request #21 from rahim-kanji/v3.1_fts-support Add full-text search (FTS) tools to MCP query server	1 month ago
René Cannaò	75a62f334d	Merge branch 'v4.0' into v3.1-vec Signed-off-by: René Cannaò <rene@proxysql.com>	1 month ago
Rahim Kanji	18dd249438	Merge remote-tracking branch 'v3.1-vec' into v3.1_fts-support	1 month ago
Rahim Kanji	0d56918744	Add full-text search (FTS) tools to MCP query server Implement BM25-ranked full-text search capability for MySQL/MariaDB tables using SQLite-based external FTS index. Changes: - Add MySQL_FTS class for managing SQLite FTS indexes - Add FTS tools: fts_index_table, fts_search, fts_reindex, fts_delete_index, fts_list_indexes, fts_rebuild_all - Add thread-safe FTS lifecycle management with fts_lock mutex - Add reset_fts_path() for runtime FTS database path configuration - Add comprehensive FTS test scripts (test_mcp_fts.sh, test_mcp_fts_detailed.sh)	1 month ago
Rene Cannao	3daaa5c592	feat: Implement RAG (Retrieval-Augmented Generation) subsystem Adds a complete RAG subsystem to ProxySQL with: - RAG_Tool_Handler implementing all MCP tools for retrieval operations - Database schema with FTS and vector support - FTS, vector, and hybrid search capabilities - Fetch and refetch tools for document/chunk retrieval - Admin tools for monitoring - Configuration variables for RAG parameters - Comprehensive documentation and test scripts Implements v0 deliverables from RAG blueprint: - SQLite schema initialization - Source registry management - MCP tools: search_fts, search_vector, search_hybrid, get_chunks, get_docs, fetch_from_source, admin.stats - Unit/integration tests and examples	1 month ago
Rahim Kanji	5b8bb1952e	Merge remote-tracking branch 'wqv3.1-vec' into v3.1_mcp-http-ssl-toggle	1 month ago
Rene Cannao	aced263367	docs: Update MCP documentation to reflect current implementation This commit completes a comprehensive review and update of all MCP-related documentation: - Remove deprecated FastAPI POC files - Update Architecture.md to reflect implemented multi-endpoint architecture - Update VARIABLES.md with new AI endpoint authentication - Update Tool_Discovery_Guide.md with discovery and LLM tools - Update FTS_Implementation_Plan.md to reflect implemented status - Update Vector_Embeddings_Implementation_Plan.md to show planned status - Update Database_Discovery_Agent.md to clarify conceptual design status - Update scripts/mcp/README.md with current multi-handler architecture - Update STDIO_BRIDGE_README.md with complete tool list All documentation now accurately reflects the current ProxySQL MCP implementation with 6 dedicated tool handlers and two-phase discovery.	1 month ago
Rahim Kanji	f7397f633c	Fix catalog search to use FTS5 and enhance test suite The catalog_fts FTS5 virtual table was being created but the search() function was using slow LIKE queries instead of FTS5 MATCH operator. Changes to lib/MySQL_Catalog.cpp: - Use FTS5 MATCH with INNER JOIN to catalog_fts when query provided - Add BM25 relevance ranking (ORDER BY bm25(f) ASC) - Significant performance improvement: O(log n) vs O(n) Changes to scripts/mcp/test_catalog.sh: - Add 8 new FTS5-specific tests (CAT013-CAT020): - Multi-term search (AND logic) - Phrase search with quotes - Boolean operators (OR, NOT) - Prefix search with wildcards - Kind and tags filter combinations - Relevance ranking verification - Add SSL/HTTP support with auto-detection - New options: --ssl, --no-ssl, MCP_USE_SSL env var - Fix endpoint path: /query -> /mcp/querywq	1 month ago
Rene Cannao	f9c5a00f8a	chore: Delete temporary discovery output files Remove temporary discovery output files that were previously generated but are now managed elsewhere.	1 month ago
Rene Cannao	5b502c0864	feat: Add question learning capability to demo agent Add ability for the demo agent to learn new questions and add them to the catalog, making it smarter over time. Changes: - Added get_last_agent_run_id() function to Discovery_Schema: - Queries agent_runs table for the most recent agent_run_id for a run_id - Returns 0 if no agent runs exist for the schema - Updated llm.question_template_add handler: - Made agent_run_id optional (defaults to 0 when not provided) - When agent_run_id <= 0, auto-fetches last agent_run_id for the schema - Returns helpful error if no agent run exists for the schema - Returns agent_run_id in response for visibility - Updated llm.question_template_add tool schema: - Moved agent_run_id from required to optional parameters - Updated description to explain auto-fetch behavior - Updated demo_agent_claude.sh prompt: - Added llm.question_template_add to available tools - Added Step 4: "Learn from Success" to workflow - Added explicit instruction to ALWAYS LEARN new questions - Added example showing learning workflow - Expanded from 4 steps to 5 steps to include learning Now the demo agent can: 1. Search for existing questions 2. Reuse SQL if a good match exists 3. Generate new SQL if no good match 4. LEARN new questions by adding them to the catalog 5. Present results This enables continuous learning - the more users interact with it, the smarter it becomes.	1 month ago
Rene Cannao	ba6cfdc8bd	feat: Update demo agent prompt to always pass schema parameter Update the demo_agent_claude.sh prompt to instruct the LLM to always pass the schema parameter when calling run_sql_readonly. Changes: - System prompt now uses unquoted heredoc (ENDPROMPT) to expand $SCHEMA variable - Updated tool description to show schema parameter as required - Updated workflow steps to show schema="$SCHEMA" in llm_search and run_sql_readonly calls - Added explicit instruction: "ALWAYS provide schema="$SCHEMA" to run_sql_readonly" - Updated example interaction to show schema parameter being passed - Changed "Ready to help" line from hardcoded "Chinook" to dynamic $SCHEMA This ensures queries run against the correct database schema, fixing the issue where queries would execute against the default schema instead of the discovered schema.	1 month ago
Rene Cannao	a0e72aed03	feat: Add related_objects support to two-phase discovery Update two-phase discovery prompts and script to populate related_objects when creating question templates. Changes: - Updated two_phase_discovery_prompt.md: - Added related_objects parameter to llm.question_template_add tool description - Added instruction to extract table/view names from example_sql - Added example showing proper related_objects format - Updated two_phase_user_prompt.md: - Added example showing how to extract and pass related_objects - Updated two_phase_discovery.py: - Made --schema parameter required (not optional) - Updated usage examples to show required --schema flag - Removed empty string fallback for schema_filter This ensures that question templates created during discovery include the related_objects field, enabling efficient object schema retrieval when templates are searched via llm.search with include_objects=true.	1 month ago
Rene Cannao	7faf993295	feat: Update demo agent script to leverage include_objects and add --help Update the demo_agent_claude.sh script to use the new include_objects parameter and improve usability. Changes: - Updated system prompt to leverage include_objects=true: - llm_search now returns question templates AND object schemas in one call - Removed catalog_get_object from primary workflow (no longer needed) - Reduced workflow from 5 steps to 4 steps - Added explicit instruction to ALWAYS use include_objects=true - Added --help/-h flag with comprehensive usage information - Made schema name mandatory (removed default "Chinook" fallback) - Script now exits with error if no schema is provided - Updated example interaction to show new streamlined workflow This significantly reduces the number of MCP calls needed to answer questions, as object schemas are now included in the initial llm_search response.	1 month ago
Rene Cannao	73d3431c92	fix: Use heredocs for system prompt to preserve special characters Rewrite demo_agent_claude.sh to use heredocs with quoted delimiters for system and append prompts. This prevents bash from interpreting backticks, emojis, and other special characters as commands. Changes: - Use cat > file << 'ENDPROMPT' syntax for prompts - Write prompts to temp files first - Read from files when calling claude CLI - Removed problematic inline variable assignments	1 month ago
Rene Cannao	1b42cfbd27	feat: Add empty query support to llm_search for listing all artifacts Changes: - fts_search_llm(): Empty query now returns all artifacts (list mode) - Update llm.search tool: query parameter is now optional - Tool description mentions empty query lists all artifacts - Add body field to llm_search results - Update demo script: Add special case for "What questions can I ask?" This enables agents to retrieve all pre-defined question templates when users ask what questions are available, instead of inferring questions from schema.	1 month ago
Rene Cannao	be675d4165	wip: Add interactive MCP query agent demo script using Claude Code Add demo_agent_claude.sh script for demonstrating the MCP query agent workflow with Claude Code interactive sessions. The script: - Configures MCP connection via proxysql_mcp_stdio_bridge.py - Sets up system prompt with agent workflow instructions - Demonstrates step-by-step query answering process Note: Script highlights current implementation gaps that need to be addressed before full functionality.	1 month ago
Rene Cannao	35b0b224ff	refactor: Remove mcp-catalog_path variable and hardcode catalog path Remove the mcp-catalog_path configuration variable and hardcode the catalog database path to datadir/mcp_catalog.db for stability. Rationale: The catalog database is session state, not user configuration. Runtime swapping of the catalog could cause tables to be missed and the catalog to fail even if it was succeeding a second earlier. Changes: - Removed catalog_path from mcp_thread_variables_names array - Removed mcp_catalog_path from MCP_Thread variables struct - Removed getter/setter logic for catalog_path - Hardcoded catalog path to GloVars.datadir/mcp_catalog.db in: - ProxySQL_MCP_Server.cpp (Query_Tool_Handler initialization) - Admin_FlushVariables.cpp (MySQL_Tool_Handler reinitialization) - Updated VARIABLES.md to document the hardcoded path - Updated configure_mcp.sh to remove catalog_path configuration - Updated MCP README to remove catalog_path references	1 month ago
Rahim Kanji	ddc4e65706	Add plain HTTP support for MCP server and fix SSL/port restart issues * Add full support for both HTTP and HTTPS modes in MCP server via the mcp_use_ssl configuration variable, enabling plain HTTP for development and HTTPS for production with proper certificate validation * Server now automatically restarts when SSL mode or port configuration changes, fixing silent configuration failures where changes appeared to succeed but didn't take effect until manual restart. Features: - Explicit support for HTTP mode (mcp_use_ssl=false) without SSL certificates - Explicit support for HTTPS mode (mcp_use_ssl=true) with certificate validation - Configurable via configure_mcp.sh with --no-ssl or --use-ssl flags - Settable via admin interface: SET mcp-use_ssl=true/false - Automatic restart detection for SSL mode changes (HTTP ↔ HTTPS) - Automatic restart detection for port changes (mcp_port)	1 month ago
Rene Cannao	1b7335acfe	Fix two-phase discovery documentation and scripts - Add mcp_config.example.json for Claude Code MCP configuration - Fix MCP bridge path in example config (../../proxysql_mcp_stdio_bridge.py) - Update Two_Phase_Discovery_Implementation.md with correct Phase 1/Phase 2 usage - Fix Two_Phase_Discovery_Implementation.md DELETE FROM fts_objects to scope to run_id - Update README.md with two-phase discovery section and multi-agent legacy note - Create static_harvest.sh bash wrapper for Phase 1 - Create two_phase_discovery.py orchestration script with prompts - Add --run-id parameter to skip auto-fetch - Fix RUN_ID placeholder mismatch (<USE_THE_PROVIDED_RUN_ID>) - Fix catalog path default to mcp_catalog.db - Add test_catalog.sh to verify catalog tools work - Fix Discovery_Schema.cpp FTS5 syntax (missing space) - Remove invalid CREATE INDEX on FTS virtual tables - Add MCP tool call logging to track tool usage - Fix Static_Harvester::get_harvest_stats() to accept run_id parameter - Fix DELETE FROM fts_objects to only delete for specific run_id - Update system prompts to say DO NOT call discovery.run_static - Update user prompts to say Phase 1 is already complete - Add --mcp-only flag to restrict Claude Code to MCP tools only - Make FTS table failures non-fatal (check if table exists first) - Add comprehensive documentation for both discovery approaches	1 month ago
Rene Cannao	6f23d5bcd0	feat: Implement two-phase schema discovery architecture Phase 1 (Static/Deterministic): - Add Discovery_Schema: SQLite catalog with deterministic and LLM tables - Add Static_Harvester: MySQL INFORMATION_SCHEMA metadata extraction - Harvest schemas, objects, columns, indexes, foreign keys, view definitions - Compute derived hints: is_time, is_id_like, has_pk, has_fks, has_time - Build quick profiles and FTS5 indexes Phase 2 (LLM Agent): - Add 19 new MCP tools for two-phase discovery - discovery.run_static: Trigger ProxySQL's static harvest - Catalog tools: init, search, get_object, list_objects, get_relationships - Agent tools: run_start, run_finish, event_append - LLM tools: summary_upsert, relationship_upsert, domain_upsert, etc. Files: - include/Discovery_Schema.h, lib/Discovery_Schema.cpp - include/Static_Harvester.h, lib/Static_Harvester.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp (updated) - lib/Makefile (updated) - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/ - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py	1 month ago
Rene Cannao	7de3f0c510	feat: Add schema separation to MCP catalog and discovery scope constraint This commit addresses two issues: 1. MCP Catalog Schema Separation: - Add 'schema' column to catalog table for proper isolation - Update all catalog methods (upsert, get, search, list, remove) to accept schema parameter - Update MCP tool handlers and JSON-RPC parameter parsing - Unique constraint changed from (kind, key) to (schema, kind, key) - FTS table updated to include schema column 2. Discovery Prompt Scope Constraint: - Add explicit SCOPE CONSTRAINT section to multi_agent_discovery_prompt.md - Agents now respect Target Schema and skip list_schemas when specified - Prevents analyzing all schemas when only one is targeted Files modified: - include/MySQL_Catalog.h: Add schema parameter to all catalog methods - include/MySQL_Tool_Handler.h: Update wrapper method signatures - lib/MySQL_Catalog.cpp: Implement schema filtering in all operations - lib/MySQL_Tool_Handler.cpp: Update wrapper implementations - lib/Query_Tool_Handler.cpp: Extract schema from JSON-RPC params, update tool descriptions - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/multi_agent_discovery_prompt.md: Add scope constraint	1 month ago
Rene Cannao	25cd0b71f6	chore: Add comprehensive gitignore for discovery output files Added patterns to ignore all discovery output files: Root .gitignore: - discovery_.md (main discovery reports at repo root) - database_discovery_report.md scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/.gitignore: - _QUESTION_CATALOG.md (individual agent question catalogs) - _round1_.md, _round2_.md, _round3_.md, _round4_.md (round outputs) - _COORDINATOR_SUMMARY.md (coordinator summaries) - _HYPOTHESIS_TESTING.md (hypothesis testing results) - _INDEX.md (index files) - _QUICK_REFERENCE.md (quick reference docs) - META_ANALYSIS_.md (META agent outputs) - SECURITY_AGENT_.txt (security agent text outputs) - query_agent_.md, security_agent_.md, semantic_.md, statistical_.md These files are generated when agents incorrectly use Write tool instead of catalog_upsert (the issue fixed in v1.2). They should not be committed to the repository as they are test outputs and are stored in the MCP catalog.	1 month ago
Rene Cannao	6fd58a6fd4	docs: Update README for v1.3 improvements Updated README to reflect new capabilities: 1. Statistical Analysis section: - Added Statistical Significance Testing subsection - Listed 5 required statistical tests (normality, correlation, chi-square, outliers, group comparisons) - Mentioned p-values and effect sizes 2. Query Analysis section: - Added Performance Baseline Measurement subsection - Listed 6 required query types with timing - Mentioned efficiency scoring (EXPLAIN vs actual) 3. Question Catalogs section: - Updated to reflect 15+ cross-domain questions (enhanced in v1.3) - Added 5 cross-domain categories with question counts 4. Cross-Domain Questions section: - Expanded from 3 examples to 15 specific questions - Organized by 5 categories with question counts - Matched new v1.3 requirements	1 month ago
Rene Cannao	3895fe5ad3	feat: Add Priority 1 improvements from META agent analysis (v1.3) Implemented three critical improvements identified by META agent: 1. Performance Baseline Measurement (QUERY agent): - Added 5 required query types with timing measurements - Compare actual execution time vs EXPLAIN cost - Efficiency score calculation (1-10 scale) - Output format with performance scores per table 2. Statistical Significance Testing (STATISTICAL agent): - Added 5 required statistical tests: * Normality tests (Shapiro-Wilk, Anderson-Darling) * Correlation analysis (Pearson, Spearman) with CI * Chi-square tests for categorical associations * Outlier detection (Modified Z-score, IQR) * Group comparisons (t-test, Mann-Whitney U) - All tests report p-values and effect sizes - Statistical confidence scoring (1-10) 3. Enhanced Cross-Domain Question Synthesis: - Increased minimum from general suggestion to 15 questions - Added 5 cross-domain categories with examples - Detailed question template with multi-phase plans - Priority distribution (URGENT/HIGH/MEDIUM) - Quality criteria for cross-domain questions Expected Impact: +25% overall quality, +30% confidence in findings	1 month ago
Rene Cannao	24d2bb2c84	fix: Enforce MCP catalog usage and prohibit Write tool for agent findings Add prominent warnings and explicit instructions to prevent agents from creating individual markdown files instead of using the MCP catalog. Changes: - Add "CRITICAL: MCP CATALOG USAGE" section at the top of the prompt - Explicitly prohibit Write tool for individual agent discoveries - Specify that ONLY Round 4 final report should be written to file - Add detailed catalog_upsert usage examples - Update all round descriptions with CRITICAL catalog usage instructions - Add "IMPORTANT - Catalog Usage Rules" section Key Instructions Added: - "DO NOT use the Write tool to create separate markdown files" - "ALL agent findings MUST be written to MCP catalog using catalog_upsert" - "ONLY Round 4 Final Synthesis writes to local file using Write tool" - "DO NOT use Write tool for individual agent discoveries in Rounds 1-3" This ensures agents store findings in the MCP catalog for cross-agent collaboration, with only the final consolidated report written to disk.	1 month ago
Rene Cannao	7ade08f572	chore: Remove accidentally committed discovery output file	1 month ago
Rene Cannao	da0b5a5cf2	fix: Correct log message from 4-agent to 6-agent discovery	1 month ago
Rene Cannao	39b9ce6d58	feat: Add Question Catalog generation to all agents Each agent now generates a comprehensive catalog of questions they can answer about the database, with executable answer plans using MCP tools. Question Catalog Features: - 90+ questions across all agents (minimum 15-20 per agent) - Executable answer plans with specific MCP tools for each question - Answer templates with structured output formats - Complexity ratings (LOW/MEDIUM/HIGH) and time estimates - Cross-domain questions requiring multiple agents New Output Locations: - kind="question_catalog", key="structural_questions" - kind="question_catalog", key="statistical_questions" - kind="question_catalog", key="semantic_questions" - kind="question_catalog", key="query_questions" - kind="question_catalog", key="security_questions" - kind="question_catalog", key="cross_domain_questions" Benefits: 1. Fast Answers: Pre-validated plans skip analysis phase 2. Consistent Quality: All answers follow proven templates 3. Tool Reuse: Efficient MCP tool usage patterns 4. Comprehensive Coverage: 90+ questions cover most user needs Example Question Catalog Entry: - Question: "What sensitive data exists in table X?" - Answer Plan: sample_rows + column_profile on table X - Answer Template: Structured list with sensitivity classification - Complexity: MEDIUM - Estimated Time: 30 seconds This creates a reusable knowledge base for future LLM interactions, enabling quick, accurate responses to common database questions.	1 month ago
Rene Cannao	130981d1be	feat: Add SECURITY and META agents to multi-agent discovery Expand the 4-agent system to 6 agents (5 analysis + 1 meta) with enhanced security analysis and self-improving prompt optimization. New Agents: - SECURITY: Identifies sensitive data (PII, credentials, financial), assesses access patterns, identifies vulnerabilities, and provides compliance assessment (GDPR, PCI-DSS) - META: Analyzes report quality by section, identifies gaps, suggests specific prompt improvements for future runs Protocol Changes: - Expanded from 4 rounds to 5 rounds - Round 5 is Meta Analysis (META agent only) - META agent does not participate in rounds 1-4 New Report Sections: - 5. SECURITY ANALYSIS with data classification (PUBLIC/INTERNAL/ CONFIDENTIAL/RESTRICTED) - E. Security data classification appendix New Output: - Separate META ANALYSIS document with: - Section quality ratings (depth, completeness) - Specific prompt improvement suggestions - Gap identification - Evolution history tracking This enables continuous prompt optimization through multiple discovery iterations, with each run informing improvements for the next.	1 month ago
Rene Cannao	82d7f0c87f	chore: Ignore discovery output files and remove accidentally committed file	1 month ago
Rene Cannao	4df56f1c4a	fix: Increase default timeout to 1 hour and improve error handling - Change default timeout from 300s to 3600s (1 hour) for multi-agent discovery - Add specific timeout error detection (exit code 124) with helpful message - Add empty output file detection with diagnostic suggestions - Improve error messages to guide users on how to debug issues - Show example commands for increasing timeout when timeout occurs The multi-agent discovery process involves 4 rounds of agent collaboration and can take significantly longer than the previous single-agent approach.	1 month ago
Rene Cannao	aed042ba0d	feat: Replace single-agent with multi-agent database discovery Replace the old single-agent discovery approach with a new 4-agent collaborative system that provides more comprehensive analysis. Changes: - Add prompts/ directory with multi-agent system prompts - Replace README.md with multi-agent discovery documentation - Replace headless_db_discovery.py to use 4-agent prompt - Replace headless_db_discovery.sh to use 4-agent prompt - Remove HEADLESS_DISCOVERY_README.md (replaced by README.md) The new system uses 4 collaborating agents (STRUCTURAL, STATISTICAL, SEMANTIC, QUERY) that communicate via MCP catalog across 4 rounds: 1. Blind Exploration (parallel independent discovery) 2. Pattern Recognition (cross-agent collaboration) 3. Hypothesis Testing (validation with evidence) 4. Final Synthesis (comprehensive report) This provides deeper analysis with 15+ hypothesis validations, health scores, and prioritized recommendations.	1 month ago
René Cannaò	61ad3c4306	Merge pull request #13 from ProxySQL/v3.1-MCP1_genAI Comprehensive AI Features Implementation: NL2SQL, Anomaly Detection, and Vector Storage	1 month ago
Rene Cannao	c5a7fc31f7	Add external LLM setup guide and live testing script - test_external_live.sh: Interactive script for testing with custom LLM - EXTERNAL_LLM_SETUP.md: Complete guide for external model configuration Covers: - Custom LLM endpoint configuration for NL2SQL - llama-server configuration for embeddings - Architecture overview - Configuration variables - Testing procedures - Troubleshooting tips	1 month ago
Rene Cannao	3b7033f44d	Add vector features verification script Simple script to verify that all vector features are properly implemented: - NL2SQL vector cache methods - Anomaly threat pattern management - sqlite-vec integration - GenAI module integration - Documentation completeness	1 month ago
Rene Cannao	f5c18fd8d7	scripts: Add threat pattern documentation script Add helper script showing sample threat patterns that can be added to the Anomaly Detection system for testing embedding similarity. Includes 10 sample patterns: 1. OR 1=1 tautology (severity 9) 2. UNION SELECT data extraction (severity 8) 3. Comment injection (severity 7) 4. Sleep-based DoS (severity 6) 5. Benchmark-based DoS (severity 6) 6. INTO OUTFILE exfiltration (severity 9) 7. DROP TABLE destruction (severity 10) 8. Schema probing (severity 3) 9. CONCAT injection (severity 8) 10. Hex encoding bypass (severity 7)	1 month ago
Rene Cannao	3f44229e28	feat: Add MCP AI Tool Handler for NL2SQL with test script Phase 5: MCP Tool Implementation for NL2SQL This commit implements the AI Tool Handler for the MCP (Model Context Protocol) server, exposing NL2SQL functionality as an MCP tool. New Files: - include/AI_Tool_Handler.h: Header for AI_Tool_Handler class - Provides ai_nl2sql_convert tool via MCP protocol - Wraps NL2SQL_Converter and Anomaly_Detector - Inherits from MCP_Tool_Handler base class - lib/AI_Tool_Handler.cpp: Implementation - Implements ai_nl2sql_convert tool execution - Accepts parameters: natural_language (required), schema, context_tables, max_latency_ms, allow_cache - Returns JSON response with sql_query, confidence, explanation, cached, cache_id - scripts/mcp/test_nl2sql_tools.sh: Test script for NL2SQL MCP tool - Tests ai_nl2sql_convert via JSON-RPC over HTTPS - 10 test cases covering SELECT, WHERE, JOIN, aggregation, etc. - Includes error handling test for empty queries - Supports --verbose, --quiet options Modified Files: - include/MCP_Thread.h: Add AI_Tool_Handler forward declaration and pointer - lib/Makefile: Add AI_Tool_Handler.oo to _OBJ_CXX list - lib/ProxySQL_MCP_Server.cpp: Initialize and register AI tool handler - Creates AI_Tool_Handler with GloAI components - Registers /mcp/ai endpoint - Adds cleanup in destructor MCP Tool Details: - Endpoint: /mcp/ai - Tool: ai_nl2sql_convert - Parameters: - natural_language (string, required): Natural language query - schema (string, optional): Database schema name - context_tables (string, optional): Comma-separated table list - max_latency_ms (integer, optional): Max acceptable latency - allow_cache (boolean, optional): Check semantic cache (default: true) Testing: Run the test script with: ./scripts/mcp/test_nl2sql_tools.sh [--verbose] [--quiet] See scripts/mcp/test_nl2sql_tools.sh --help for usage. Related: Phase 1-4 (Documentation, Unit Tests, Integration Tests, E2E Tests) Related: Phase 6-8 (User Docs, Developer Docs, Test Docs)	1 month ago
Rene Cannao	aee9c3117b	test: Add E2E test script for NL2SQL - test_nl2sql_e2e.sh: End-to-end testing with --mock and --live modes - Tests complete workflow from natural language to executed SQL - Includes test schema setup, LLM configuration, and 8 test cases - Supports both mocked LLM responses (fast) and live LLM testing	1 month ago
Rene Cannao	6dd2613d63	Move discovery docs to examples directory Relocate DATABASE_DISCOVERY_REPORT.md and DATABASE_QUESTION_CAPABILITIES.md to scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/examples/ for better organization.	1 month ago
Rene Cannao	fdee58a26d	Add comprehensive database discovery outputs and enhance headless discovery - Add DATABASE_DISCOVERY_REPORT.md: Complete multi-agent database discovery findings covering structure, statistics, business domain, and query analysis - Add DATABASE_QUESTION_CAPABILITIES.md: Showcase of 14 question categories answerable via the discovery system with examples - Enhance headless_db_discovery.py: Improve JSON parsing and error handling - Enhance headless_db_discovery.sh: Add better argument handling and validation	1 month ago
Rene Cannao	b627f836f5	Refactor: Reorganize headless discovery scripts to dedicated directory Move headless database discovery scripts from scripts/ to scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/ for better organization. Also update README to: - Focus only on ProxySQL Query MCP (remove generic database examples) - Use relative paths (./) instead of absolute paths - Simplify configuration documentation Files moved: - scripts/HEADLESS_DISCOVERY_README.md - scripts/headless_db_discovery.py - scripts/headless_db_discovery.sh	1 month ago
Rene Cannao	d73ce0c41e	Add headless database discovery scripts Implement scripts for running Claude Code in non-interactive mode to perform comprehensive database discovery on any database. Files added: - headless_db_discovery.sh: Bash script implementation - headless_db_discovery.py: Python script implementation (recommended) - HEADLESS_DISCOVERY_README.md: Comprehensive documentation Features: - Works with any database accessible via MCP - Database-agnostic discovery prompt - Comprehensive analysis: structure, data, semantics, performance - Markdown report output with ERD, data quality score, recommendations - CI/CD integration ready - Supports custom MCP server configuration - Configurable timeout, output, verbosity Usage: python scripts/headless_db_discovery.py --database mydb	1 month ago
Rene Cannao	f852900365	Fix: Correct MCP catalog JSON parsing to handle special characters The catalog_search() and catalog_list() methods in MySQL_Catalog.cpp were manually building JSON strings by concatenating raw TEXT from SQLite without proper escaping. This caused parse errors when stored JSON contained quotes, backslashes, or newlines. Changes: - MySQL_Catalog.cpp: Use nlohmann::json to build proper nested JSON in search() and list() methods instead of manual concatenation - MySQL_Tool_Handler.cpp: Add try-catch for JSON parsing in catalog_get() - test_catalog.sh: Fix MCP URL path, add jq extraction for MCP protocol responses, add 3 special character tests (CAT013-CAT015) Test Results: All 15 catalog tests pass, including new tests that verify special characters (quotes, backslashes) are preserved.	1 month ago

1 2 3

102 Commits (7c0ff770f6cefe98789993e15f1a7b95251e2e0d)