proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	35b0b224ff	refactor: Remove mcp-catalog_path variable and hardcode catalog path Remove the mcp-catalog_path configuration variable and hardcode the catalog database path to datadir/mcp_catalog.db for stability. Rationale: The catalog database is session state, not user configuration. Runtime swapping of the catalog could cause tables to be missed and the catalog to fail even if it was succeeding a second earlier. Changes: - Removed catalog_path from mcp_thread_variables_names array - Removed mcp_catalog_path from MCP_Thread variables struct - Removed getter/setter logic for catalog_path - Hardcoded catalog path to GloVars.datadir/mcp_catalog.db in: - ProxySQL_MCP_Server.cpp (Query_Tool_Handler initialization) - Admin_FlushVariables.cpp (MySQL_Tool_Handler reinitialization) - Updated VARIABLES.md to document the hardcoded path - Updated configure_mcp.sh to remove catalog_path configuration - Updated MCP README to remove catalog_path references	3 months ago
Rene Cannao	a816a756d4	feat: Add MCP query tool usage counters to stats schema Add stats_mcp_query_tools_counters and stats_mcp_query_tools_counters_reset tables to track MCP query tool usage statistics. - Added get_tool_usage_stats_resultset() method to Query_Tool_Handler - Defined table schemas in ProxySQL_Admin_Tables_Definitions.h - Registered tables in Admin_Bootstrap.cpp - Added pattern matching in ProxySQL_Admin.cpp - Added stats___mcp_query_tools_counters() in ProxySQL_Admin_Stats.cpp - Fixed friend declaration for track_tool_invocation() - Fixed Discovery_Schema.cpp log_llm_search() to use prepare_v2/finalize	3 months ago
Rene Cannao	393967f511	fix: Use row->cnt instead of row->fields_count	3 months ago
Rene Cannao	df0527c044	refactor: list_schemas to use catalog instead of live database - Query schemas from catalog's schemas table - Maintains same output format for compatibility - Removes dependency on live MySQL connection	3 months ago
Rene Cannao	527a748d16	refactor: Remove describe_table tool completely Tool was deprecated; users should use catalog.get_object instead.	3 months ago
Rene Cannao	623675b369	feat: Add schema name resolver and deprecate direct DB tools - Add resolve_run_id() to map schema names to latest run_id - Update all catalog and LLM tools to accept schema names - Deprecate describe_table, table_profile, column_profile - Deprecate get_constraints, suggest_joins, find_reference_candidates - Keep sample_rows, sample_distinct for data preview	3 months ago
Rene Cannao	757cdaff15	fix: Improve error logging and fix llm.domain_set_members 1. Fix error logging to catch ALL tool failures, not just those with both success and result fields. Previously, error responses like {"success": false, "error": "..."} without a result field were silently ignored. 2. Fix llm.domain_set_members to accept both array and JSON string formats for the members parameter. Some clients send it as a JSON string, others as a native array. 3. Add detailed error logging for llm.domain_set_members failures, including what was actually received.	3 months ago
Rene Cannao	d962caea7e	feat: Improve MCP error logging with request payloads Exception handlers now log the full request payload that caused the error, making debugging much easier. Changes: - Move req_body/req_path declarations outside try block so catch handlers can access them - Log request payload in all exception handlers (parse errors, std::exception, and catch-all) - Log tool arguments when tool execution fails Previously, exceptions would only log the error message without context, making it impossible to reproduce the issue. Now the full payload is logged.	3 months ago
Rene Cannao	53ecda7730	fix: Add comprehensive error handling and logging for MCP tools - Add try-catch around handle_jsonrpc_request to catch unexpected exceptions - Add detailed logging for tool execution success/failure - Add proper SQLite error checking in create_agent_run with error messages - Fix json_int/json_double to handle both numbers and numeric strings The json_int function was throwing exceptions when receiving numeric strings (e.g., "14" instead of 14) from clients, causing 500 errors. Now it handles both formats gracefully. Also added logging so tool failures are visible in logs instead of being silent 500 errors.	3 months ago
Rene Cannao	1b7335acfe	Fix two-phase discovery documentation and scripts - Add mcp_config.example.json for Claude Code MCP configuration - Fix MCP bridge path in example config (../../proxysql_mcp_stdio_bridge.py) - Update Two_Phase_Discovery_Implementation.md with correct Phase 1/Phase 2 usage - Fix Two_Phase_Discovery_Implementation.md DELETE FROM fts_objects to scope to run_id - Update README.md with two-phase discovery section and multi-agent legacy note - Create static_harvest.sh bash wrapper for Phase 1 - Create two_phase_discovery.py orchestration script with prompts - Add --run-id parameter to skip auto-fetch - Fix RUN_ID placeholder mismatch (<USE_THE_PROVIDED_RUN_ID>) - Fix catalog path default to mcp_catalog.db - Add test_catalog.sh to verify catalog tools work - Fix Discovery_Schema.cpp FTS5 syntax (missing space) - Remove invalid CREATE INDEX on FTS virtual tables - Add MCP tool call logging to track tool usage - Fix Static_Harvester::get_harvest_stats() to accept run_id parameter - Fix DELETE FROM fts_objects to only delete for specific run_id - Update system prompts to say DO NOT call discovery.run_static - Update user prompts to say Phase 1 is already complete - Add --mcp-only flag to restrict Claude Code to MCP tools only - Make FTS table failures non-fatal (check if table exists first) - Add comprehensive documentation for both discovery approaches	3 months ago
Rene Cannao	f9270e6c8b	fix: Correct two_phase_discovery.py usage example in docs Add missing --mcp-config argument which is required by the script. The dry-run example now correctly shows all required parameters.	3 months ago
Rene Cannao	6f23d5bcd0	feat: Implement two-phase schema discovery architecture Phase 1 (Static/Deterministic): - Add Discovery_Schema: SQLite catalog with deterministic and LLM tables - Add Static_Harvester: MySQL INFORMATION_SCHEMA metadata extraction - Harvest schemas, objects, columns, indexes, foreign keys, view definitions - Compute derived hints: is_time, is_id_like, has_pk, has_fks, has_time - Build quick profiles and FTS5 indexes Phase 2 (LLM Agent): - Add 19 new MCP tools for two-phase discovery - discovery.run_static: Trigger ProxySQL's static harvest - Catalog tools: init, search, get_object, list_objects, get_relationships - Agent tools: run_start, run_finish, event_append - LLM tools: summary_upsert, relationship_upsert, domain_upsert, etc. Files: - include/Discovery_Schema.h, lib/Discovery_Schema.cpp - include/Static_Harvester.h, lib/Static_Harvester.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp (updated) - lib/Makefile (updated) - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/ - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py	3 months ago
Rene Cannao	7de3f0c510	feat: Add schema separation to MCP catalog and discovery scope constraint This commit addresses two issues: 1. MCP Catalog Schema Separation: - Add 'schema' column to catalog table for proper isolation - Update all catalog methods (upsert, get, search, list, remove) to accept schema parameter - Update MCP tool handlers and JSON-RPC parameter parsing - Unique constraint changed from (kind, key) to (schema, kind, key) - FTS table updated to include schema column 2. Discovery Prompt Scope Constraint: - Add explicit SCOPE CONSTRAINT section to multi_agent_discovery_prompt.md - Agents now respect Target Schema and skip list_schemas when specified - Prevents analyzing all schemas when only one is targeted Files modified: - include/MySQL_Catalog.h: Add schema parameter to all catalog methods - include/MySQL_Tool_Handler.h: Update wrapper method signatures - lib/MySQL_Catalog.cpp: Implement schema filtering in all operations - lib/MySQL_Tool_Handler.cpp: Update wrapper implementations - lib/Query_Tool_Handler.cpp: Extract schema from JSON-RPC params, update tool descriptions - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/multi_agent_discovery_prompt.md: Add scope constraint	3 months ago
Rene Cannao	25cd0b71f6	chore: Add comprehensive gitignore for discovery output files Added patterns to ignore all discovery output files: Root .gitignore: - discovery_.md (main discovery reports at repo root) - database_discovery_report.md scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/.gitignore: - _QUESTION_CATALOG.md (individual agent question catalogs) - _round1_.md, _round2_.md, _round3_.md, _round4_.md (round outputs) - _COORDINATOR_SUMMARY.md (coordinator summaries) - _HYPOTHESIS_TESTING.md (hypothesis testing results) - _INDEX.md (index files) - _QUICK_REFERENCE.md (quick reference docs) - META_ANALYSIS_.md (META agent outputs) - SECURITY_AGENT_.txt (security agent text outputs) - query_agent_.md, security_agent_.md, semantic_.md, statistical_.md These files are generated when agents incorrectly use Write tool instead of catalog_upsert (the issue fixed in v1.2). They should not be committed to the repository as they are test outputs and are stored in the MCP catalog.	3 months ago
Rene Cannao	6fd58a6fd4	docs: Update README for v1.3 improvements Updated README to reflect new capabilities: 1. Statistical Analysis section: - Added Statistical Significance Testing subsection - Listed 5 required statistical tests (normality, correlation, chi-square, outliers, group comparisons) - Mentioned p-values and effect sizes 2. Query Analysis section: - Added Performance Baseline Measurement subsection - Listed 6 required query types with timing - Mentioned efficiency scoring (EXPLAIN vs actual) 3. Question Catalogs section: - Updated to reflect 15+ cross-domain questions (enhanced in v1.3) - Added 5 cross-domain categories with question counts 4. Cross-Domain Questions section: - Expanded from 3 examples to 15 specific questions - Organized by 5 categories with question counts - Matched new v1.3 requirements	3 months ago
Rene Cannao	3895fe5ad3	feat: Add Priority 1 improvements from META agent analysis (v1.3) Implemented three critical improvements identified by META agent: 1. Performance Baseline Measurement (QUERY agent): - Added 5 required query types with timing measurements - Compare actual execution time vs EXPLAIN cost - Efficiency score calculation (1-10 scale) - Output format with performance scores per table 2. Statistical Significance Testing (STATISTICAL agent): - Added 5 required statistical tests: * Normality tests (Shapiro-Wilk, Anderson-Darling) * Correlation analysis (Pearson, Spearman) with CI * Chi-square tests for categorical associations * Outlier detection (Modified Z-score, IQR) * Group comparisons (t-test, Mann-Whitney U) - All tests report p-values and effect sizes - Statistical confidence scoring (1-10) 3. Enhanced Cross-Domain Question Synthesis: - Increased minimum from general suggestion to 15 questions - Added 5 cross-domain categories with examples - Detailed question template with multi-phase plans - Priority distribution (URGENT/HIGH/MEDIUM) - Quality criteria for cross-domain questions Expected Impact: +25% overall quality, +30% confidence in findings	3 months ago
Rene Cannao	24d2bb2c84	fix: Enforce MCP catalog usage and prohibit Write tool for agent findings Add prominent warnings and explicit instructions to prevent agents from creating individual markdown files instead of using the MCP catalog. Changes: - Add "CRITICAL: MCP CATALOG USAGE" section at the top of the prompt - Explicitly prohibit Write tool for individual agent discoveries - Specify that ONLY Round 4 final report should be written to file - Add detailed catalog_upsert usage examples - Update all round descriptions with CRITICAL catalog usage instructions - Add "IMPORTANT - Catalog Usage Rules" section Key Instructions Added: - "DO NOT use the Write tool to create separate markdown files" - "ALL agent findings MUST be written to MCP catalog using catalog_upsert" - "ONLY Round 4 Final Synthesis writes to local file using Write tool" - "DO NOT use Write tool for individual agent discoveries in Rounds 1-3" This ensures agents store findings in the MCP catalog for cross-agent collaboration, with only the final consolidated report written to disk.	3 months ago
Rene Cannao	7ade08f572	chore: Remove accidentally committed discovery output file	3 months ago
Rene Cannao	da0b5a5cf2	fix: Correct log message from 4-agent to 6-agent discovery	3 months ago
Rene Cannao	39b9ce6d58	feat: Add Question Catalog generation to all agents Each agent now generates a comprehensive catalog of questions they can answer about the database, with executable answer plans using MCP tools. Question Catalog Features: - 90+ questions across all agents (minimum 15-20 per agent) - Executable answer plans with specific MCP tools for each question - Answer templates with structured output formats - Complexity ratings (LOW/MEDIUM/HIGH) and time estimates - Cross-domain questions requiring multiple agents New Output Locations: - kind="question_catalog", key="structural_questions" - kind="question_catalog", key="statistical_questions" - kind="question_catalog", key="semantic_questions" - kind="question_catalog", key="query_questions" - kind="question_catalog", key="security_questions" - kind="question_catalog", key="cross_domain_questions" Benefits: 1. Fast Answers: Pre-validated plans skip analysis phase 2. Consistent Quality: All answers follow proven templates 3. Tool Reuse: Efficient MCP tool usage patterns 4. Comprehensive Coverage: 90+ questions cover most user needs Example Question Catalog Entry: - Question: "What sensitive data exists in table X?" - Answer Plan: sample_rows + column_profile on table X - Answer Template: Structured list with sensitivity classification - Complexity: MEDIUM - Estimated Time: 30 seconds This creates a reusable knowledge base for future LLM interactions, enabling quick, accurate responses to common database questions.	3 months ago
Rene Cannao	130981d1be	feat: Add SECURITY and META agents to multi-agent discovery Expand the 4-agent system to 6 agents (5 analysis + 1 meta) with enhanced security analysis and self-improving prompt optimization. New Agents: - SECURITY: Identifies sensitive data (PII, credentials, financial), assesses access patterns, identifies vulnerabilities, and provides compliance assessment (GDPR, PCI-DSS) - META: Analyzes report quality by section, identifies gaps, suggests specific prompt improvements for future runs Protocol Changes: - Expanded from 4 rounds to 5 rounds - Round 5 is Meta Analysis (META agent only) - META agent does not participate in rounds 1-4 New Report Sections: - 5. SECURITY ANALYSIS with data classification (PUBLIC/INTERNAL/ CONFIDENTIAL/RESTRICTED) - E. Security data classification appendix New Output: - Separate META ANALYSIS document with: - Section quality ratings (depth, completeness) - Specific prompt improvement suggestions - Gap identification - Evolution history tracking This enables continuous prompt optimization through multiple discovery iterations, with each run informing improvements for the next.	3 months ago
Rene Cannao	82d7f0c87f	chore: Ignore discovery output files and remove accidentally committed file	3 months ago
Rene Cannao	4df56f1c4a	fix: Increase default timeout to 1 hour and improve error handling - Change default timeout from 300s to 3600s (1 hour) for multi-agent discovery - Add specific timeout error detection (exit code 124) with helpful message - Add empty output file detection with diagnostic suggestions - Improve error messages to guide users on how to debug issues - Show example commands for increasing timeout when timeout occurs The multi-agent discovery process involves 4 rounds of agent collaboration and can take significantly longer than the previous single-agent approach.	3 months ago
Rene Cannao	aed042ba0d	feat: Replace single-agent with multi-agent database discovery Replace the old single-agent discovery approach with a new 4-agent collaborative system that provides more comprehensive analysis. Changes: - Add prompts/ directory with multi-agent system prompts - Replace README.md with multi-agent discovery documentation - Replace headless_db_discovery.py to use 4-agent prompt - Replace headless_db_discovery.sh to use 4-agent prompt - Remove HEADLESS_DISCOVERY_README.md (replaced by README.md) The new system uses 4 collaborating agents (STRUCTURAL, STATISTICAL, SEMANTIC, QUERY) that communicate via MCP catalog across 4 rounds: 1. Blind Exploration (parallel independent discovery) 2. Pattern Recognition (cross-agent collaboration) 3. Hypothesis Testing (validation with evidence) 4. Final Synthesis (comprehensive report) This provides deeper analysis with 15+ hypothesis validations, health scores, and prioritized recommendations.	3 months ago
René Cannaò	91ea6f5e49	Merge pull request #10 from ProxySQL/v3.1-MCP1_discovery Add MCP stdio bridge and database discovery tools	3 months ago
René Cannaò	61ad3c4306	Merge pull request #13 from ProxySQL/v3.1-MCP1_genAI Comprehensive AI Features Implementation: NL2SQL, Anomaly Detection, and Vector Storage	3 months ago
Rene Cannao	1193a55e7b	docs: Remove Version History section from LLM Bridge README	3 months ago
Rene Cannao	5afb71ca90	docs: Rename NL2SQL documentation to LLM Bridge - Rename doc/NL2SQL/ to doc/LLM_Bridge/ - Update README.md for generic LLM bridge functionality - Replace NL2SQL terminology with LLM Bridge - Update variable references from ai_nl2sql_* to genai_llm_* - Add migration guide from NL2SQL to LLM Bridge - Document new use cases (summarization, code generation, etc.)	3 months ago
Rene Cannao	a3f0bade4e	feat: Convert NL2SQL to generic LLM bridge - Rename NL2SQL_Converter to LLM_Bridge for generic prompt processing - Update MySQL protocol handler from /* NL2SQL: / to / LLM: / - Remove SQL-specific fields (sql_query, confidence, tables_used) - Add GENAI_OP_LLM operation type to GenAI module - Rename all genai_nl2sql_ variables to genai_llm_* - Update AI_Features_Manager to use LLM_Bridge - Deprecate ai_nl2sql_convert MCP tool with error message - LLM bridge now handles any prompt type via MySQL protocol This enables generic LLM access (summarization, code generation, translation, analysis) while preserving infrastructure for future NL2SQL implementation via Web UI + external agents.	3 months ago
Rene Cannao	3fe8a48f70	Fix genai variable handling and add API key masking - Add has_variable() method to GenAI_Threads_Handler for variable validation - Add genai- prefix check in is_valid_global_variable() - Auto-initialize NL2SQL converter when genai-nl2sql_enabled is set to true at runtime - Make init_nl2sql() public to allow runtime initialization - Mask API keys in logs (show only first 2 chars, rest as 'x')	3 months ago
Rene Cannao	1eb42c57d0	fix: Add GenAI variables to runtime_global_variables population Add flush_genai_variables___runtime_to_database() call to the central location where all modules populate runtime_global_variables table. This was missing, causing genai-* variables to not appear in runtime_global_variables.	3 months ago
Rene Cannao	6ffb59b856	fix: Use db parameter instead of hardcoded admindb in GenAI database_to_runtime The flush_genai_variables___database_to_runtime() function was using hardcoded 'admindb' instead of the 'db' parameter passed to the function. This caused the function to always query from admindb, ignoring the actual database parameter. This fixes the issue where runtime_global_variables table was not being populated on startup because the query was always hitting the same database regardless of the parameter.	3 months ago
Rene Cannao	4018a0ad3b	fix: Follow MCP pattern for GenAI variables runtime table population Update flush_genai_variables___database_to_runtime() to match the MCP pattern exactly: - Add 'lock' parameter (default true) for flexibility - Use ProxySQL_Admin's wrlock()/wrunlock() instead of GloGATH's - Use consistent variable naming (var_name = name + 6 for 'genai-' prefix) - Follow exact same locking pattern as MCP variables This fixes the issue where runtime_global_variables table was not being populated on startup because the locking pattern was incorrect.	3 months ago
Rene Cannao	1ea67900ab	fix: Populate runtime_global_variables for GenAI variables on startup The flush_genai_variables___database_to_runtime() function was only setting internal state in GloGATH but not populating the runtime_global_variables table. This caused variables to appear in global_variables but not in runtime_global_variables after startup. Fix: Add call to flush_genai_variables___runtime_to_database() with runtime=true to populate the runtime table, following the same pattern used by MCP variables.	3 months ago
Rene Cannao	51fd51e3f5	fix: Add missing GenAI_Thread.h include and fix variables reference Added missing #include for GenAI_Thread.h in AI_Features_Manager.cpp to resolve the compilation error in debug mode. Also fixed the remaining reference to variables.ai_features_enabled which should now use GloGATH->variables.genai_enabled. This fixes the "make debug" build failure.	3 months ago
Rene Cannao	349320a67f	docs: Fix NL2SQL documentation with genai variables and async architecture Updated documentation to reflect the architectural changes: - Changed all ai_* variables to genai-* variables - Changed LOAD MYSQL VARIABLES to LOAD GENAI VARIABLES - Clarified that NL2SQL queries run on MySQL module, not Admin - Updated ARCHITECTURE.md to document async/non-blocking architecture - Added LOAD GENAI VARIABLES TO RUNTIME after variable changes - Changed stats_ai_nl2sql_cache to SHOW STATUS LIKE 'genai-nl2sql%' Key corrections in README.md: - Variables now use genai-* prefix (genai-nl2sql_enabled, etc.) - Commands use LOAD GENAI VARIABLES instead of LOAD MYSQL VARIABLES - Added note that NL2SQL queries are on MySQL module, not Admin interface Key corrections in ARCHITECTURE.md: - Updated system flow diagram to show async socketpair path - Added detailed async flow explanation - Documented that MySQL threads are NOT blocked during LLM calls Related to: https://github.com/ProxySQL/proxysql-vec/pull/13	3 months ago
Rene Cannao	a7dac5ef3d	feat: Make NL2SQL use async GenAI path instead of blocking calls This is a critical architectural fix - NL2SQL was making blocking calls to LLMs which would block the entire MySQL thread. Now NL2SQL uses the same async socketpair pattern as the GENAI embed/rerank operations. Changes: - Added nl2sql operation type to process_json_query() in GenAI module - Updated NL2SQL handler to construct JSON query and use async GENAI path - Added extern declaration for GloAI in GenAI_Thread.cpp - Falls back to synchronous path only on systems without epoll Architecture: - Before: NL2SQL: query → blocking nl2sql->convert() → blocks MySQL thread - After: NL2SQL: query → JSON GENAI request → async socketpair → non-blocking JSON protocol for NL2SQL: GENAI: {"type": "nl2sql", "query": "Show customers", "schema": "mydb"} The NL2SQL result is delivered asynchronously through the existing GENAI response handler, making the system fully non-blocking. Related to: https://github.com/ProxySQL/proxysql-vec/pull/13	3 months ago
Rene Cannao	527bfed297	fix: Migrate AI variables to GenAI module for proper architecture This commit fixes a serious design flaw where AI configuration variables were not integrated with the ProxySQL admin interface. All ai_* variables have been migrated to the GenAI module as genai-* variables. Changes: - Added 21 new genai_* variables to GenAI_Thread.h structure - Implemented get/set functions for all new variables in GenAI_Thread.cpp - Removed internal variables struct from AI_Features_Manager - AI_Features_Manager now reads from GloGATH instead of internal state - Updated documentation to reference genai-* variables - Fixed debug.cpp assertion for PROXY_DEBUG_NL2SQL and PROXY_DEBUG_ANOMALY Variable mapping: - ai_nl2sql_enabled → genai-nl2sql_enabled - ai_anomaly_detection_enabled → genai-anomaly_enabled - ai_features_enabled → genai-enabled - All other ai_* variables follow the same pattern The flush functions automatically handle all variables in the genai_thread_variables_names array, so database persistence works correctly without additional changes. Related to: https://github.com/ProxySQL/proxysql-vec/pull/13	3 months ago
Rene Cannao	3f25fb6c4e	Enhance anomaly detector unit tests with additional edge case coverage - Add edge case tests for SQL injection pattern detection: * Empty and NULL query handling * Very long query processing - Add edge case tests for query normalization: * Empty query handling * Unicode character support * Nested comment processing * Multiple line comment handling - Add edge case tests for risk scoring: * Negative match handling * Large match count capping * Boundary condition validation - Fix test assertions for more robust validation - All 39 tests now pass successfully These improvements address potential feedback from code reviewers by providing more comprehensive test coverage for edge cases and boundary conditions.	3 months ago
Rene Cannao	2888ee3f45	Fix gemini-code-assist recommendations and implement comprehensive anomaly detection tests - Fix retry logic to use is_retryable_error function for proper HTTP error handling - Add exception handling to get_json_int function with try-catch around std::stoi - Improve validate_numeric_range to use strtol instead of atoi for better error reporting - Fix Chinese characters in documentation (non-zero -> non-zero) - Replace placeholder tests with actual comprehensive tests for anomaly detection functionality - Create new standalone unit test anomaly_detector_unit-t.cpp with 29 tests covering: * SQL injection pattern detection (12 tests) * Query normalization (8 tests) * Risk scoring calculations (5 tests) * Configuration validation (4 tests) - All tests pass successfully, providing meaningful validation of core anomaly detection logic Thanks to gemini-code-assist for the thorough code review and recommendations.	3 months ago
René Cannaò	7665b3b3c9	Merge pull request #12 from ProxySQL/v3.1-MCP1_discovery_cc1 Multi-Agent Database Discovery System using Claude Code	3 months ago
Rene Cannao	ae4200dbc0	Enhance AI features with improved validation, memory safety, error handling, and performance monitoring - Rename validate_provider_name to validate_provider_format for clarity - Add null checks and error handling for all strdup() operations - Enhance error messages with more context and HTTP status codes - Implement performance monitoring with timing metrics for LLM calls and cache operations - Add comprehensive test coverage for edge cases, retry scenarios, and performance - Extend status variables to track performance metrics - Update MySQL session to report timing information to AI manager	3 months ago
Rene Cannao	3032dffed4	test: Add NL2SQL internal functionality unit tests Add comprehensive TAP unit tests for NL2SQL internal functions: - Error code conversion (5 tests): Validate nl2sql_error_code_to_string() covers all 13 defined error codes plus UNKNOWN_ERROR - SQL validation patterns (17 tests): Test validate_and_score_sql() * Valid SELECT queries (4 tests) * Non-SELECT queries (4 tests) * Injection pattern detection (4 tests) * Edge cases (4 tests): empty, lone keyword, semicolons, complex queries - Request ID generation (12 tests): Test UUID-like ID generation * Format validation (20 assertions for 10 IDs) * Uniqueness (100 IDs checked for duplicates) * Hexadecimal character validation - Prompt building (8 tests): Test build_prompt() * Basic prompt structure (3 tests) * Schema context inclusion (3 tests) * Section ordering (1 test) * Special character handling (2 tests) Note: Tests are self-contained with standalone implementations matching the logic in NL2SQL_Converter.cpp.	3 months ago
Rene Cannao	8a6b7480b6	docs: Update NL2SQL documentation for v0.2.0 features This commit updates the NL2SQL documentation to reflect the new features added in v0.2.0: README.md changes: - Added Request Configuration section with retry parameters - Added Error Handling section with error code table - Added Request Correlation section with log format examples - Updated Results section with error columns - Updated Troubleshooting with retry behavior documentation - Added v0.2.0 to Version History API.md changes: - Updated NL2SQLRequest struct with request_id and retry config fields - Updated NL2SQLResult struct with error details fields - Added NL2SQLErrorCode enum documentation - Updated Result Format with new columns - Expanded Error Codes section with structured error codes TESTING.md changes: - Added Validation Tests to test suite overview - Documented ai_validation-t.cpp test categories - Added instructions for running validation tests - Documented all 61 test cases across 5 categories	3 months ago
Rene Cannao	49092e9c8d	test: Add unit tests for AI configuration validation This commit adds comprehensive unit tests for the AI configuration validation functions used in AI_Features_Manager. Changes: - Add test/tap/tests/ai_validation-t.cpp with 61 unit tests - Test URL format validation (validate_url_format) - Test API key format validation (validate_api_key_format) - Test numeric range validation (validate_numeric_range) - Test provider name validation (validate_provider_name) - Test edge cases and boundary conditions The test file is self-contained with its own copies of the validation functions to avoid complex linking dependencies on libproxysql. Test Categories: - URL validation: 15 tests (http://, https:// protocols) - API key validation: 14 tests (OpenAI, Anthropic formats) - Numeric range: 13 tests (min/max boundaries) - Provider name: 8 tests (openai, anthropic) - Edge cases: 11 tests (NULL handling, long values) All 61 tests pass successfully. Part of: Phase 4 of NL2SQL improvement plan	3 months ago
Rene Cannao	8f38b8a577	feat: Add exponential backoff retry for transient LLM failures This commit adds configurable retry logic with exponential backoff for NL2SQL LLM API calls. Changes: - Add retry configuration to NL2SQLRequest (max_retries, retry_backoff_ms, retry_multiplier, retry_max_backoff_ms) - Add is_retryable_error() to identify retryable HTTP/CURL errors - Add sleep_with_jitter() for exponential backoff with 10% jitter - Add call_generic_openai_with_retry() wrapper - Add call_generic_anthropic_with_retry() wrapper - Update NL2SQL_Converter::convert() to use retry wrappers Default retry behavior: - 3 retries with 1000ms initial backoff - 2.0x multiplier, 30000ms max backoff - Retries on empty responses (transient failures) Part of: Phase 3 of NL2SQL improvement plan	3 months ago
Rene Cannao	d0dc36ac0b	feat: Add structured logging with timing and request IDs Add comprehensive structured logging for NL2SQL LLM API calls with request correlation, timing metrics, and detailed error context. Changes: - Add request_id field to NL2SQLRequest with UUID-like auto-generation - Add structured logging macros: * LOG_LLM_REQUEST: Logs URL, model, prompt length with request ID * LOG_LLM_RESPONSE: Logs HTTP status, duration_ms, response preview * LOG_LLM_ERROR: Logs error phase, message, and status code - Update call_generic_openai() signature to accept req_id parameter - Update call_generic_anthropic() signature to accept req_id parameter - Add timing metrics to both LLM call functions using clock_gettime() - Replace existing debug logging with structured logging macros - Update convert() to pass request_id to LLM calls Request IDs are generated as UUID-like strings (e.g., "12345678-9abc-def0-1234-567890abcdef") and are included in all log messages for correlation. This allows tracking a single NL2SQL request through all log lines from request to response. Timing is measured using CLOCK_MONOTONIC for accurate duration tracking of LLM API calls, reported in milliseconds. This provides much better debugging capability when troubleshooting NL2SQL issues, as administrators can now: - Correlate all log lines for a single request - See exact timing of LLM API calls - Identify which phase of processing failed - Track request/response metrics Fixes #2 - Add Structured Logging	3 months ago
Rene Cannao	45e592b623	feat: Add structured error messages with context to NL2SQL Add comprehensive error details to help users debug NL2SQL conversion issues. Changes: - Add error_code, error_details, http_status_code, provider_used fields to NL2SQLResult - Add NL2SQLErrorCode enum with structured error codes: * SUCCESS, ERR_API_KEY_MISSING, ERR_API_KEY_INVALID, ERR_TIMEOUT * ERR_CONNECTION_FAILED, ERR_RATE_LIMITED, ERR_SERVER_ERROR * ERR_EMPTY_RESPONSE, ERR_INVALID_RESPONSE, ERR_SQL_INJECTION_DETECTED * ERR_VALIDATION_FAILED, ERR_UNKNOWN_PROVIDER, ERR_REQUEST_TOO_LARGE - Add nl2sql_error_code_to_string() function for error code conversion - Add format_error_context() helper to create detailed error messages including: * Query (truncated if too long) * Schema name * Provider attempted * Endpoint URL * Specific error message - Add set_error_details() helper to populate error fields - Update error handling in convert() to use new error details - Track provider_used in successful conversions This provides much better debugging information when NL2SQL conversions fail, making it easier to identify misconfigurations and connectivity issues. Fixes #1 - Improve Error Messages	3 months ago
Rene Cannao	40b2608c2d	feat: Add configuration validation to AI_Features_Manager Add comprehensive validation for AI features configuration variables to prevent invalid states and improve error messages. Changes: - Add validate_url_format(): Checks for http:// or https:// prefix and host part - Add validate_api_key_format(): Validates API key format, checks for whitespace, minimum length, and incomplete key patterns (sk- with <20 chars, sk-ant- with <25 chars) - Add validate_numeric_range(): Validates numeric values are within min/max range - Add validate_provider_name(): Ensures provider is 'openai' or 'anthropic' - Update set_variable() to call validation functions before setting values Validated variables: - ai_nl2sql_provider: Must be 'openai' or 'anthropic' - ai_nl2sql_provider_url: Must have http:// or https:// prefix - ai_nl2sql_provider_key: No whitespace, minimum 10 chars - ai_nl2sql_cache_similarity_threshold: Range [0, 100] - ai_nl2sql_timeout_ms: Range [1000, 300000] (1 second to 5 minutes) - ai_nl2sql_max_cloud_requests_per_hour: Range [1, 10000] - ai_anomaly_similarity_threshold: Range [0, 100] - ai_anomaly_risk_threshold: Range [0, 100] - ai_anomaly_rate_limit: Range [1, 10000] - ai_vector_dimension: Range [128, 4096] This prevents misconfigurations and provides clear error messages to users when invalid values are provided. Fixes compilation issue by moving validation helper functions before set_variable() to resolve forward declaration errors.	3 months ago
Rene Cannao	36b11223b2	feat: Improve SQL validation with multi-factor scoring Add comprehensive SQL validation with confidence scoring based on: - SQL keyword detection (17 keywords covering DDL/DML/transactions) - Structural validation (balanced parentheses and quotes) - SQL injection pattern detection - Length and quality checks Confidence scoring: - Base 0.4 for valid SQL keyword - +0.15 for balanced parentheses - +0.15 for balanced quotes - +0.1 for minimum length - +0.1 for FROM clause in SELECT statements - +0.1 for no injection patterns - -0.3 penalty for injection patterns detected Low confidence (< 0.5) results are logged with detailed info. Cache storage threshold updated to 0.5 confidence (from implicit valid_sql). This improves detection of malformed or potentially malicious SQL while providing granular confidence scores for downstream use.	3 months ago

1 2 3 4 5 ...

8099 Commits (35b0b224ff1a8987c501a9e05926a9d89c3e3f85) All Branches Search

8099 Commits (35b0b224ff1a8987c501a9e05926a9d89c3e3f85)

All Branches