proxysql

Commit Graph

Author	SHA1	Message	Date
René Cannaò	9727871a16	Merge pull request #5656 from sysown/dependabot/pip/scripts/mcp/DiscoveryAgent/Rich/python-dotenv-1.2.2 build(deps): bump python-dotenv from 1.2.1 to 1.2.2 in /scripts/mcp/DiscoveryAgent/Rich	2 months ago
dependabot[bot]	b6379e448e	build(deps): bump python-dotenv in /scripts/mcp/DiscoveryAgent/Rich Bumps [python-dotenv](https://github.com/theskumar/python-dotenv) from 1.2.1 to 1.2.2. - [Release notes](https://github.com/theskumar/python-dotenv/releases) - [Changelog](https://github.com/theskumar/python-dotenv/blob/main/CHANGELOG.md) - [Commits](https://github.com/theskumar/python-dotenv/compare/v1.2.1...v1.2.2) --- updated-dependencies: - dependency-name: python-dotenv dependency-version: 1.2.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>	2 months ago
dependabot[bot]	9d2979d96c	build(deps): bump python-dotenv Bumps [python-dotenv](https://github.com/theskumar/python-dotenv) from 1.0.1 to 1.2.2. - [Release notes](https://github.com/theskumar/python-dotenv/releases) - [Changelog](https://github.com/theskumar/python-dotenv/blob/main/CHANGELOG.md) - [Commits](https://github.com/theskumar/python-dotenv/compare/v1.0.1...v1.2.2) --- updated-dependencies: - dependency-name: python-dotenv dependency-version: 1.2.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>	2 months ago
Rene Cannao	9ffc3f8d71	mcp tests: add phase-B TAP coverage and optional real Claude CLI E2E runner Add two complementary TAP tests for phase-B validation and an optional real Claude CLI E2E helper, so we can validate both 'without Claude credentials' and 'with real Claude CLI' workflows. What was added: 1) CI-safe deterministic phase-B TAP - New: test/tap/tests/test_mcp_llm_discovery_phaseb-t.sh - Validates MCP phase-B primitives end-to-end without external LLM API: - list_targets - discovery.run_static (target_id-scoped setup) - catalog.list_objects - agent.run_start / agent.run_finish - llm.summary_upsert / llm.summary_get - llm.domain_upsert / llm.domain_set_members - llm.metric_upsert - llm.question_template_add - llm.search - Uses unique per-run markers to assert persisted artifacts are retrievable 2) Claude headless flow TAP smoke - New: test/tap/tests/test_mcp_claude_headless_flow-t.sh - Always validates integration path without external dependencies: - static_harvest.sh wrapper executes and yields run_id - two_phase_discovery.py --dry-run executes with target_id/run_id context - Optional real Claude execution: - enabled via TAP_RUN_REAL_CLAUDE=1 - skipped by default to keep CI deterministic 3) Manual real-CLI E2E runner - New: scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/run_real_claude_e2e.sh - Runs full two-step flow manually when credentials/CLI are available: - phase-A static harvest (or --skip-phase-a + --run-id) - phase-B real Claude run via two_phase_discovery.py 4) Documentation updates - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/README.md: - documents run_real_claude_e2e.sh usage - test/tap/groups/ai/README.md: - adds manual run instructions for: - phase-A static harvest test - phase-B deterministic TAP test - Claude headless smoke (with optional real mode) Validation run: - bash -n: - test_mcp_llm_discovery_phaseb-t.sh - test_mcp_claude_headless_flow-t.sh - run_real_claude_e2e.sh - static_harvest.sh - python3 -m py_compile: - two_phase_discovery.py	4 months ago
Rene Cannao	ade0130e67	discoveryagent: update Claude Code headless flow for target_id-scoped MCP tools Enhance the existing Claude Code custom-prompt discovery examples to match the new MCP contract introduced by multi-target routing and target-scoped catalog semantics. Changes included: - static_harvest.sh - added required --target-id argument (with MCP_TARGET_ID env fallback) - always sends target_id in discovery.run_static arguments - updated usage/help/examples to include target_id - two_phase_discovery.py - added required --target-id argument - static-harvest bootstrap call now passes target_id - improved run_id error guidance with target_id-aware commands - injects <TARGET_ID> placeholder into user prompt template - improved runtime logging to display target_id - Prompt templates - two_phase_discovery_prompt.md: - catalog/agent/llm tool signatures updated to include target_id - Stage 0 and workflow text updated to use provided target_id + run_id - removed contradictory instruction that asked to call discovery.run_static in Phase 2 - two_phase_user_prompt.md: - added target_id input section - all example calls updated to pass target_id - start instructions now require target_id + run_id - README and utility example - README quick-start curl and script examples now include target_id - test_catalog.sh now accepts/prints target_id and passes it to catalog/llm calls Validation: - bash -n passed for updated shell scripts - python3 -m py_compile passed for two_phase_discovery.py	4 months ago
Rene Cannao	aced263367	docs: Update MCP documentation to reflect current implementation This commit completes a comprehensive review and update of all MCP-related documentation: - Remove deprecated FastAPI POC files - Update Architecture.md to reflect implemented multi-endpoint architecture - Update VARIABLES.md with new AI endpoint authentication - Update Tool_Discovery_Guide.md with discovery and LLM tools - Update FTS_Implementation_Plan.md to reflect implemented status - Update Vector_Embeddings_Implementation_Plan.md to show planned status - Update Database_Discovery_Agent.md to clarify conceptual design status - Update scripts/mcp/README.md with current multi-handler architecture - Update STDIO_BRIDGE_README.md with complete tool list All documentation now accurately reflects the current ProxySQL MCP implementation with 6 dedicated tool handlers and two-phase discovery.	5 months ago
Rene Cannao	f9c5a00f8a	chore: Delete temporary discovery output files Remove temporary discovery output files that were previously generated but are now managed elsewhere.	5 months ago
Rene Cannao	a0e72aed03	feat: Add related_objects support to two-phase discovery Update two-phase discovery prompts and script to populate related_objects when creating question templates. Changes: - Updated two_phase_discovery_prompt.md: - Added related_objects parameter to llm.question_template_add tool description - Added instruction to extract table/view names from example_sql - Added example showing proper related_objects format - Updated two_phase_user_prompt.md: - Added example showing how to extract and pass related_objects - Updated two_phase_discovery.py: - Made --schema parameter required (not optional) - Updated usage examples to show required --schema flag - Removed empty string fallback for schema_filter This ensures that question templates created during discovery include the related_objects field, enabling efficient object schema retrieval when templates are searched via llm.search with include_objects=true.	5 months ago
Rene Cannao	1b7335acfe	Fix two-phase discovery documentation and scripts - Add mcp_config.example.json for Claude Code MCP configuration - Fix MCP bridge path in example config (../../proxysql_mcp_stdio_bridge.py) - Update Two_Phase_Discovery_Implementation.md with correct Phase 1/Phase 2 usage - Fix Two_Phase_Discovery_Implementation.md DELETE FROM fts_objects to scope to run_id - Update README.md with two-phase discovery section and multi-agent legacy note - Create static_harvest.sh bash wrapper for Phase 1 - Create two_phase_discovery.py orchestration script with prompts - Add --run-id parameter to skip auto-fetch - Fix RUN_ID placeholder mismatch (<USE_THE_PROVIDED_RUN_ID>) - Fix catalog path default to mcp_catalog.db - Add test_catalog.sh to verify catalog tools work - Fix Discovery_Schema.cpp FTS5 syntax (missing space) - Remove invalid CREATE INDEX on FTS virtual tables - Add MCP tool call logging to track tool usage - Fix Static_Harvester::get_harvest_stats() to accept run_id parameter - Fix DELETE FROM fts_objects to only delete for specific run_id - Update system prompts to say DO NOT call discovery.run_static - Update user prompts to say Phase 1 is already complete - Add --mcp-only flag to restrict Claude Code to MCP tools only - Make FTS table failures non-fatal (check if table exists first) - Add comprehensive documentation for both discovery approaches	5 months ago
Rene Cannao	6f23d5bcd0	feat: Implement two-phase schema discovery architecture Phase 1 (Static/Deterministic): - Add Discovery_Schema: SQLite catalog with deterministic and LLM tables - Add Static_Harvester: MySQL INFORMATION_SCHEMA metadata extraction - Harvest schemas, objects, columns, indexes, foreign keys, view definitions - Compute derived hints: is_time, is_id_like, has_pk, has_fks, has_time - Build quick profiles and FTS5 indexes Phase 2 (LLM Agent): - Add 19 new MCP tools for two-phase discovery - discovery.run_static: Trigger ProxySQL's static harvest - Catalog tools: init, search, get_object, list_objects, get_relationships - Agent tools: run_start, run_finish, event_append - LLM tools: summary_upsert, relationship_upsert, domain_upsert, etc. Files: - include/Discovery_Schema.h, lib/Discovery_Schema.cpp - include/Static_Harvester.h, lib/Static_Harvester.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp (updated) - lib/Makefile (updated) - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/ - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py	5 months ago
Rene Cannao	7de3f0c510	feat: Add schema separation to MCP catalog and discovery scope constraint This commit addresses two issues: 1. MCP Catalog Schema Separation: - Add 'schema' column to catalog table for proper isolation - Update all catalog methods (upsert, get, search, list, remove) to accept schema parameter - Update MCP tool handlers and JSON-RPC parameter parsing - Unique constraint changed from (kind, key) to (schema, kind, key) - FTS table updated to include schema column 2. Discovery Prompt Scope Constraint: - Add explicit SCOPE CONSTRAINT section to multi_agent_discovery_prompt.md - Agents now respect Target Schema and skip list_schemas when specified - Prevents analyzing all schemas when only one is targeted Files modified: - include/MySQL_Catalog.h: Add schema parameter to all catalog methods - include/MySQL_Tool_Handler.h: Update wrapper method signatures - lib/MySQL_Catalog.cpp: Implement schema filtering in all operations - lib/MySQL_Tool_Handler.cpp: Update wrapper implementations - lib/Query_Tool_Handler.cpp: Extract schema from JSON-RPC params, update tool descriptions - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/multi_agent_discovery_prompt.md: Add scope constraint	5 months ago
Rene Cannao	25cd0b71f6	chore: Add comprehensive gitignore for discovery output files Added patterns to ignore all discovery output files: Root .gitignore: - discovery_.md (main discovery reports at repo root) - database_discovery_report.md scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/.gitignore: - _QUESTION_CATALOG.md (individual agent question catalogs) - _round1_.md, _round2_.md, _round3_.md, _round4_.md (round outputs) - _COORDINATOR_SUMMARY.md (coordinator summaries) - _HYPOTHESIS_TESTING.md (hypothesis testing results) - _INDEX.md (index files) - _QUICK_REFERENCE.md (quick reference docs) - META_ANALYSIS_.md (META agent outputs) - SECURITY_AGENT_.txt (security agent text outputs) - query_agent_.md, security_agent_.md, semantic_.md, statistical_.md These files are generated when agents incorrectly use Write tool instead of catalog_upsert (the issue fixed in v1.2). They should not be committed to the repository as they are test outputs and are stored in the MCP catalog.	5 months ago
Rene Cannao	6fd58a6fd4	docs: Update README for v1.3 improvements Updated README to reflect new capabilities: 1. Statistical Analysis section: - Added Statistical Significance Testing subsection - Listed 5 required statistical tests (normality, correlation, chi-square, outliers, group comparisons) - Mentioned p-values and effect sizes 2. Query Analysis section: - Added Performance Baseline Measurement subsection - Listed 6 required query types with timing - Mentioned efficiency scoring (EXPLAIN vs actual) 3. Question Catalogs section: - Updated to reflect 15+ cross-domain questions (enhanced in v1.3) - Added 5 cross-domain categories with question counts 4. Cross-Domain Questions section: - Expanded from 3 examples to 15 specific questions - Organized by 5 categories with question counts - Matched new v1.3 requirements	5 months ago
Rene Cannao	3895fe5ad3	feat: Add Priority 1 improvements from META agent analysis (v1.3) Implemented three critical improvements identified by META agent: 1. Performance Baseline Measurement (QUERY agent): - Added 5 required query types with timing measurements - Compare actual execution time vs EXPLAIN cost - Efficiency score calculation (1-10 scale) - Output format with performance scores per table 2. Statistical Significance Testing (STATISTICAL agent): - Added 5 required statistical tests: * Normality tests (Shapiro-Wilk, Anderson-Darling) * Correlation analysis (Pearson, Spearman) with CI * Chi-square tests for categorical associations * Outlier detection (Modified Z-score, IQR) * Group comparisons (t-test, Mann-Whitney U) - All tests report p-values and effect sizes - Statistical confidence scoring (1-10) 3. Enhanced Cross-Domain Question Synthesis: - Increased minimum from general suggestion to 15 questions - Added 5 cross-domain categories with examples - Detailed question template with multi-phase plans - Priority distribution (URGENT/HIGH/MEDIUM) - Quality criteria for cross-domain questions Expected Impact: +25% overall quality, +30% confidence in findings	5 months ago
Rene Cannao	24d2bb2c84	fix: Enforce MCP catalog usage and prohibit Write tool for agent findings Add prominent warnings and explicit instructions to prevent agents from creating individual markdown files instead of using the MCP catalog. Changes: - Add "CRITICAL: MCP CATALOG USAGE" section at the top of the prompt - Explicitly prohibit Write tool for individual agent discoveries - Specify that ONLY Round 4 final report should be written to file - Add detailed catalog_upsert usage examples - Update all round descriptions with CRITICAL catalog usage instructions - Add "IMPORTANT - Catalog Usage Rules" section Key Instructions Added: - "DO NOT use the Write tool to create separate markdown files" - "ALL agent findings MUST be written to MCP catalog using catalog_upsert" - "ONLY Round 4 Final Synthesis writes to local file using Write tool" - "DO NOT use Write tool for individual agent discoveries in Rounds 1-3" This ensures agents store findings in the MCP catalog for cross-agent collaboration, with only the final consolidated report written to disk.	5 months ago
Rene Cannao	7ade08f572	chore: Remove accidentally committed discovery output file	5 months ago
Rene Cannao	da0b5a5cf2	fix: Correct log message from 4-agent to 6-agent discovery	5 months ago
Rene Cannao	39b9ce6d58	feat: Add Question Catalog generation to all agents Each agent now generates a comprehensive catalog of questions they can answer about the database, with executable answer plans using MCP tools. Question Catalog Features: - 90+ questions across all agents (minimum 15-20 per agent) - Executable answer plans with specific MCP tools for each question - Answer templates with structured output formats - Complexity ratings (LOW/MEDIUM/HIGH) and time estimates - Cross-domain questions requiring multiple agents New Output Locations: - kind="question_catalog", key="structural_questions" - kind="question_catalog", key="statistical_questions" - kind="question_catalog", key="semantic_questions" - kind="question_catalog", key="query_questions" - kind="question_catalog", key="security_questions" - kind="question_catalog", key="cross_domain_questions" Benefits: 1. Fast Answers: Pre-validated plans skip analysis phase 2. Consistent Quality: All answers follow proven templates 3. Tool Reuse: Efficient MCP tool usage patterns 4. Comprehensive Coverage: 90+ questions cover most user needs Example Question Catalog Entry: - Question: "What sensitive data exists in table X?" - Answer Plan: sample_rows + column_profile on table X - Answer Template: Structured list with sensitivity classification - Complexity: MEDIUM - Estimated Time: 30 seconds This creates a reusable knowledge base for future LLM interactions, enabling quick, accurate responses to common database questions.	5 months ago
Rene Cannao	130981d1be	feat: Add SECURITY and META agents to multi-agent discovery Expand the 4-agent system to 6 agents (5 analysis + 1 meta) with enhanced security analysis and self-improving prompt optimization. New Agents: - SECURITY: Identifies sensitive data (PII, credentials, financial), assesses access patterns, identifies vulnerabilities, and provides compliance assessment (GDPR, PCI-DSS) - META: Analyzes report quality by section, identifies gaps, suggests specific prompt improvements for future runs Protocol Changes: - Expanded from 4 rounds to 5 rounds - Round 5 is Meta Analysis (META agent only) - META agent does not participate in rounds 1-4 New Report Sections: - 5. SECURITY ANALYSIS with data classification (PUBLIC/INTERNAL/ CONFIDENTIAL/RESTRICTED) - E. Security data classification appendix New Output: - Separate META ANALYSIS document with: - Section quality ratings (depth, completeness) - Specific prompt improvement suggestions - Gap identification - Evolution history tracking This enables continuous prompt optimization through multiple discovery iterations, with each run informing improvements for the next.	5 months ago
Rene Cannao	82d7f0c87f	chore: Ignore discovery output files and remove accidentally committed file	5 months ago
Rene Cannao	4df56f1c4a	fix: Increase default timeout to 1 hour and improve error handling - Change default timeout from 300s to 3600s (1 hour) for multi-agent discovery - Add specific timeout error detection (exit code 124) with helpful message - Add empty output file detection with diagnostic suggestions - Improve error messages to guide users on how to debug issues - Show example commands for increasing timeout when timeout occurs The multi-agent discovery process involves 4 rounds of agent collaboration and can take significantly longer than the previous single-agent approach.	5 months ago
Rene Cannao	aed042ba0d	feat: Replace single-agent with multi-agent database discovery Replace the old single-agent discovery approach with a new 4-agent collaborative system that provides more comprehensive analysis. Changes: - Add prompts/ directory with multi-agent system prompts - Replace README.md with multi-agent discovery documentation - Replace headless_db_discovery.py to use 4-agent prompt - Replace headless_db_discovery.sh to use 4-agent prompt - Remove HEADLESS_DISCOVERY_README.md (replaced by README.md) The new system uses 4 collaborating agents (STRUCTURAL, STATISTICAL, SEMANTIC, QUERY) that communicate via MCP catalog across 4 rounds: 1. Blind Exploration (parallel independent discovery) 2. Pattern Recognition (cross-agent collaboration) 3. Hypothesis Testing (validation with evidence) 4. Final Synthesis (comprehensive report) This provides deeper analysis with 15+ hypothesis validations, health scores, and prioritized recommendations.	5 months ago
Rene Cannao	6dd2613d63	Move discovery docs to examples directory Relocate DATABASE_DISCOVERY_REPORT.md and DATABASE_QUESTION_CAPABILITIES.md to scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/examples/ for better organization.	5 months ago
Rene Cannao	fdee58a26d	Add comprehensive database discovery outputs and enhance headless discovery - Add DATABASE_DISCOVERY_REPORT.md: Complete multi-agent database discovery findings covering structure, statistics, business domain, and query analysis - Add DATABASE_QUESTION_CAPABILITIES.md: Showcase of 14 question categories answerable via the discovery system with examples - Enhance headless_db_discovery.py: Improve JSON parsing and error handling - Enhance headless_db_discovery.sh: Add better argument handling and validation	5 months ago
Rene Cannao	b627f836f5	Refactor: Reorganize headless discovery scripts to dedicated directory Move headless database discovery scripts from scripts/ to scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/ for better organization. Also update README to: - Focus only on ProxySQL Query MCP (remove generic database examples) - Use relative paths (./) instead of absolute paths - Simplify configuration documentation Files moved: - scripts/HEADLESS_DISCOVERY_README.md - scripts/headless_db_discovery.py - scripts/headless_db_discovery.sh	6 months ago
Rene Cannao	1d046148d4	Fix: Address code review feedback from coderabbitai and gemini-code-assist Critical fixes: - Remove stray backslash at start of discover_cli.py (causes syntax error) - Fix escaped newlines (\\n) to real newlines (\n) in discover_cli.py Documentation fixes: - Update README.md: remove incorrect requirements_cli.txt reference - Update README.md: use generic path placeholder instead of /home/rene/... - Update STDIO_BRIDGE_README.md: mark PROXYSQL_MCP_ENDPOINT as optional with default Dependency updates: - Update package versions: httpx 0.27.0->0.28.1, pydantic 2.8.2->2.12.5, python-dotenv 1.0.1->1.2.1, rich 13.7.1->14.2.0 Per review: https://github.com/ProxySQL/proxysql-vec/pull/10	6 months ago
Rene Cannao	9d6a2173bf	Enhance Rich CLI with configurable LLM chat path and better tracing LLM improvements: - Add configurable chat path (LLM_CHAT_PATH or --llm-chat-path) to support non-standard endpoints like Z.ai's /api/coding/paas/v4 - Add optional JSON mode (LLM_JSON_MODE or --llm-json-mode) for models that support native JSON output - Enhanced tracing: log HTTP status and response body snippet on every request - Safer JSON parsing: treat empty content as error with helpful message - Better error messages with diagnostic hints Code cleanup: - Remove intent command (simplify CLI) - Remove user_intent reading and passing - Simplify stopping logic (just run max_iterations) - Clean up formatting and remove unused code	6 months ago
Rene Cannao	f2ca750c05	Add MCP Database Discovery Agent (initial commit) Add a database discovery agent prototype that uses LLMs to explore databases through the MCP Query endpoint. Includes: - Rich CLI (discover_cli.py): Working async CLI with Rich TUI, proper MCP tools/call JSON-RPC method, and full tracing support - FastAPI_deprecated_POC: Early prototype with incorrect MCP protocol, kept for reference only The Rich CLI version implements a multi-expert agent architecture: - Planner: Chooses next tasks based on state - Structural Expert: Analyzes table structure and relationships - Statistical Expert: Profiles tables and columns - Semantic Expert: Infers domain meaning - Query Expert: Validates access patterns	6 months ago

28 Commits (6cdd4e697eb86982e3dfd2ceab009ead5da0efcb)