proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	cc3cc25532	fix: Remove unused reset parameter from stats___mcp_query_rules() Change function signature from stats___mcp_query_rules(bool reset) to stats___mcp_query_rules() to match MySQL query rules pattern. The reset parameter was never used in the function body and MySQL's stats___mysql_query_rules() has no reset parameter.	3 months ago
Rene Cannao	8c9aecce9b	feat: Add LOAD MCP QUERY RULES FROM DISK / TO MEMORY commands - Add LOAD MCP QUERY RULES FROM DISK command - Add LOAD MCP QUERY RULES TO MEMORY command - Both commands copy rules from disk.mcp_query_rules to main.mcp_query_rules This completes the full set of MCP query rules LOAD/SAVE commands, matching the MySQL query rules pattern.	3 months ago
Rene Cannao	7e6f9f0ab3	fix: Add MCP query rules LOAD/SAVE command handlers - Add separate MCP QUERY RULES command block in Admin_Handler - Fix string length comparison (21 chars for "SAVE/LOAD MCP QUERY RULES ") - Add handlers for: - LOAD MCP QUERY RULES TO RUNTIME - SAVE MCP QUERY RULES TO DISK - SAVE MCP QUERY RULES TO MEMORY / FROM RUNTIME - Register mcp_query_rules in disk database (tables_defs_config) Previously MCP commands were incorrectly nested inside MYSQL/PGSQL block and could not be reached. Now they have their own conditional block.	3 months ago
Rene Cannao	f01fc79584	feat: Add runtime_mcp_query_rules table and fix stats_mcp_query_rules schema - Add ADMIN_SQLITE_TABLE_RUNTIME_MCP_QUERY_RULES schema (17 columns, same as mcp_query_rules) - Fix STATS_SQLITE_TABLE_MCP_QUERY_RULES to only have rule_id and hits columns - Add runtime_mcp_query_rules detection and refresh in ProxySQL_Admin - Implement save_mcp_query_rules_from_runtime(bool _runtime) for both config and runtime tables - Update get_mcp_query_rules() to return 17 columns (no hits) - get_stats_mcp_query_rules() returns 2 columns (rule_id, hits) Mirrors the MySQL query rules pattern: - mcp_query_rules: config table (17 cols) - runtime_mcp_query_rules: runtime state (17 cols) - stats_mcp_query_rules: hit counters (2 cols)	3 months ago
Rene Cannao	f449c4236f	fix: Improve question learning fallback and error logging Two bug fixes for the question learning feature: 1. Fallback to most recent agent_run across all schemas - get_last_agent_run_id() now falls back to the most recent agent_run_id across ALL runs if none exists for the specific run_id - This allows adding questions even when the current schema's discovery didn't include an agent run - Adds logging to show when fallback is used 2. Fix error message extraction for query_tool_calls logging - Fixed bug where error messages weren't being extracted correctly - The old code checked for result["error"]["message"] but create_error_response only has result["error"] (no nested "message" field) - Now correctly extracts result["error"] as a string when present - This ensures failed tool calls are properly logged with error messages This fixes the issue where llm.question_template_add would fail with "No agent run found" even when agent runs exist for other schemas.	3 months ago
Rene Cannao	5b502c0864	feat: Add question learning capability to demo agent Add ability for the demo agent to learn new questions and add them to the catalog, making it smarter over time. Changes: - Added get_last_agent_run_id() function to Discovery_Schema: - Queries agent_runs table for the most recent agent_run_id for a run_id - Returns 0 if no agent runs exist for the schema - Updated llm.question_template_add handler: - Made agent_run_id optional (defaults to 0 when not provided) - When agent_run_id <= 0, auto-fetches last agent_run_id for the schema - Returns helpful error if no agent run exists for the schema - Returns agent_run_id in response for visibility - Updated llm.question_template_add tool schema: - Moved agent_run_id from required to optional parameters - Updated description to explain auto-fetch behavior - Updated demo_agent_claude.sh prompt: - Added llm.question_template_add to available tools - Added Step 4: "Learn from Success" to workflow - Added explicit instruction to ALWAYS LEARN new questions - Added example showing learning workflow - Expanded from 4 steps to 5 steps to include learning Now the demo agent can: 1. Search for existing questions 2. Reuse SQL if a good match exists 3. Generate new SQL if no good match 4. LEARN new questions by adding them to the catalog 5. Present results This enables continuous learning - the more users interact with it, the smarter it becomes.	3 months ago
Rene Cannao	ee74384c79	fix: Prevent llm.search from returning huge object lists in list mode When llm.search is called with an empty query (list mode) to retrieve all available questions, include_objects=true was returning full object schemas for all related objects, resulting in massive responses that could fill the LLM's context and cause rejections. Fix: include_objects now only works when query is non-empty (search mode). When query is empty (list mode), only question templates are returned without object details, regardless of include_objects setting. This makes semantic sense: - Empty query = "list all questions" → just titles/bodies (compact) - Non-empty query = "search for specific questions" → full details including object schemas (for answering the question) Changes: - Modified fts_search_llm() to check !query.empty() before fetching objects - Updated tool schema description to clarify this behavior	3 months ago
Rene Cannao	7e522aa2c0	feat: Add schema parameter to run_sql_readonly with per-connection tracking Add optional schema parameter to run_sql_readonly tool that allows queries to be executed against a specific schema, independent of the default schema configured in mcp-mysql_schema. Changes: - Added current_schema field to MySQLConnection structure to track the currently selected schema for each connection in the pool - Added find_connection() helper to find connection wrapper by mysql pointer - Added execute_query_with_schema() function that: - Uses mysql_select_db() instead of 'USE schema' SQL statement - Only calls mysql_select_db() if the requested schema differs from the current schema (optimization to avoid unnecessary switches) - Updates current_schema after successful schema switch - Updated run_sql_readonly handler: - Extracts optional 'schema' parameter - Calls execute_query_with_schema() instead of execute_query() - Returns error response when query fails (instead of success) - Updated tool schema to document the new 'schema' parameter This fixes the issue where queries would run against the default schema (configured in mcp-mysql_schema) instead of the schema being queried, causing "Table doesn't exist" errors when the default schema differs from the discovered schema.	3 months ago
Rene Cannao	ee13e4bf13	feat: Add include_objects parameter to llm_search for complete object retrieval Enhance the llm_search MCP tool to return complete question template data and optionally include full object schemas, reducing the need for additional MCP calls when answering questions. Changes: - Added related_objects column to llm_question_templates table - Updated add_question_template() to accept and store related_objects JSON array - Enhanced fts_search_llm() with include_objects parameter: - LEFT JOIN with llm_question_templates to return example_sql, related_objects, template_json, and confidence - When include_objects=true, fetches full object schemas (columns, indexes) for all related objects in a single batch operation - Added error checking for SQL execution failures - Fixed fts_search_llm() get_object() call to pass schema_name and object_name separately instead of combined object_key - Updated Query_Tool_Handler: - Added is_boolean() handling to json_int() helper to properly convert JSON boolean true/false to int 1/0 - Updated llm.search handler to extract and pass include_objects parameter - Updated llm.question_template_add to extract and pass related_objects - Updated tool schemas to document new parameters This change allows agents to get all necessary schema information in a single llm_search call instead of making multiple catalog_get_object calls, significantly reducing MCP call overhead.	3 months ago
Rene Cannao	1b42cfbd27	feat: Add empty query support to llm_search for listing all artifacts Changes: - fts_search_llm(): Empty query now returns all artifacts (list mode) - Update llm.search tool: query parameter is now optional - Tool description mentions empty query lists all artifacts - Add body field to llm_search results - Update demo script: Add special case for "What questions can I ask?" This enables agents to retrieve all pre-defined question templates when users ask what questions are available, instead of inferring questions from schema.	3 months ago
Rene Cannao	5668c86809	fix: Implement FTS indexing for LLM artifacts and fix reserved keyword issue - Rename llm_search_log column from \"limit\" to \"lmt\" to avoid SQL reserved keyword - Add FTS inserts to all LLM artifact upsert functions: - add_question_template(): index question templates for search - add_llm_note(): index notes for search - upsert_llm_summary(): index object summaries for search - upsert_llm_domain(): index domains for search - upsert_llm_metric(): index metrics for search - Remove content='' from fts_llm table to store content directly - Add <functional> header for std::hash usage This fixes the bug where llm_search always returned empty results because the FTS index was never populated.	3 months ago
Rene Cannao	2250b762a3	feat: Add query_tool_calls table to log MCP tool invocations Add query_tool_calls table to Discovery Schema to track all MCP tool invocations via the /mcp/query/ endpoint. Logs: - tool_name: Name of the tool that was called - schema: Schema name (nullable, empty if not applicable) - run_id: Run ID from discovery (nullable, 0 if not applicable) - start_time: Start monotonic time in microseconds - execution_time: Execution duration in microseconds - error: Error message (null if success) Modified files: - Discovery_Schema.cpp: Added table creation and log_query_tool_call function - Discovery_Schema.h: Added function declaration - Query_Tool_Handler.cpp: Added logging after each tool execution	3 months ago
Rene Cannao	8a395b9b47	style: Add spaces around commas in SQL CREATE TABLE statements Format column definitions in CREATE TABLE IF NOT EXISTS statements to have a space before and after each comma (e.g., " , "). This allows ProxySQL Admin to properly display multi-line table schemas. Modified files: - Discovery_Schema.cpp - MySQL_Catalog.cpp - AI_Features_Manager.cpp	3 months ago
Rene Cannao	7c93280174	fix: Escape SQL reserved keyword 'limit' in llm_search_log table The column name 'limit' conflicts with SQL reserved keyword. Escaped as "\"limit\"" to fix table creation.	3 months ago
Rene Cannao	77643859e3	feat: Add timing columns to stats_mcp_query_tools_counters Extend the stats_mcp_query_tools_counters table with timing statistics (first_seen, last_seen, sum_time, min_time, max_time) following the same pattern as stats_mysql_query_digest. All timing values are in microseconds using monotonic_time(). New schema: - tool VARCHAR - schema VARCHAR - count INT - first_seen INTEGER (microseconds) - last_seen INTEGER (microseconds) - sum_time INTEGER (microseconds - total execution time) - min_time INTEGER (microseconds - minimum execution time) - max_time INTEGER (microseconds - maximum execution time)	3 months ago
Rene Cannao	fb66af7c1b	feat: Expose MCP catalog database in ProxySQL Admin interface The MCP catalog database is now accessible as the 'mcp_catalog' schema from the ProxySQL Admin interface, enabling direct SQL queries against discovered schemas and LLM memories.	3 months ago
Rene Cannao	35b0b224ff	refactor: Remove mcp-catalog_path variable and hardcode catalog path Remove the mcp-catalog_path configuration variable and hardcode the catalog database path to datadir/mcp_catalog.db for stability. Rationale: The catalog database is session state, not user configuration. Runtime swapping of the catalog could cause tables to be missed and the catalog to fail even if it was succeeding a second earlier. Changes: - Removed catalog_path from mcp_thread_variables_names array - Removed mcp_catalog_path from MCP_Thread variables struct - Removed getter/setter logic for catalog_path - Hardcoded catalog path to GloVars.datadir/mcp_catalog.db in: - ProxySQL_MCP_Server.cpp (Query_Tool_Handler initialization) - Admin_FlushVariables.cpp (MySQL_Tool_Handler reinitialization) - Updated VARIABLES.md to document the hardcoded path - Updated configure_mcp.sh to remove catalog_path configuration - Updated MCP README to remove catalog_path references	3 months ago
Rene Cannao	a816a756d4	feat: Add MCP query tool usage counters to stats schema Add stats_mcp_query_tools_counters and stats_mcp_query_tools_counters_reset tables to track MCP query tool usage statistics. - Added get_tool_usage_stats_resultset() method to Query_Tool_Handler - Defined table schemas in ProxySQL_Admin_Tables_Definitions.h - Registered tables in Admin_Bootstrap.cpp - Added pattern matching in ProxySQL_Admin.cpp - Added stats___mcp_query_tools_counters() in ProxySQL_Admin_Stats.cpp - Fixed friend declaration for track_tool_invocation() - Fixed Discovery_Schema.cpp log_llm_search() to use prepare_v2/finalize	3 months ago
Rene Cannao	393967f511	fix: Use row->cnt instead of row->fields_count	3 months ago
Rene Cannao	df0527c044	refactor: list_schemas to use catalog instead of live database - Query schemas from catalog's schemas table - Maintains same output format for compatibility - Removes dependency on live MySQL connection	3 months ago
Rene Cannao	527a748d16	refactor: Remove describe_table tool completely Tool was deprecated; users should use catalog.get_object instead.	3 months ago
Rene Cannao	623675b369	feat: Add schema name resolver and deprecate direct DB tools - Add resolve_run_id() to map schema names to latest run_id - Update all catalog and LLM tools to accept schema names - Deprecate describe_table, table_profile, column_profile - Deprecate get_constraints, suggest_joins, find_reference_candidates - Keep sample_rows, sample_distinct for data preview	3 months ago
Rene Cannao	757cdaff15	fix: Improve error logging and fix llm.domain_set_members 1. Fix error logging to catch ALL tool failures, not just those with both success and result fields. Previously, error responses like {"success": false, "error": "..."} without a result field were silently ignored. 2. Fix llm.domain_set_members to accept both array and JSON string formats for the members parameter. Some clients send it as a JSON string, others as a native array. 3. Add detailed error logging for llm.domain_set_members failures, including what was actually received.	3 months ago
Rene Cannao	d962caea7e	feat: Improve MCP error logging with request payloads Exception handlers now log the full request payload that caused the error, making debugging much easier. Changes: - Move req_body/req_path declarations outside try block so catch handlers can access them - Log request payload in all exception handlers (parse errors, std::exception, and catch-all) - Log tool arguments when tool execution fails Previously, exceptions would only log the error message without context, making it impossible to reproduce the issue. Now the full payload is logged.	3 months ago
Rene Cannao	53ecda7730	fix: Add comprehensive error handling and logging for MCP tools - Add try-catch around handle_jsonrpc_request to catch unexpected exceptions - Add detailed logging for tool execution success/failure - Add proper SQLite error checking in create_agent_run with error messages - Fix json_int/json_double to handle both numbers and numeric strings The json_int function was throwing exceptions when receiving numeric strings (e.g., "14" instead of 14) from clients, causing 500 errors. Now it handles both formats gracefully. Also added logging so tool failures are visible in logs instead of being silent 500 errors.	3 months ago
Rene Cannao	1b7335acfe	Fix two-phase discovery documentation and scripts - Add mcp_config.example.json for Claude Code MCP configuration - Fix MCP bridge path in example config (../../proxysql_mcp_stdio_bridge.py) - Update Two_Phase_Discovery_Implementation.md with correct Phase 1/Phase 2 usage - Fix Two_Phase_Discovery_Implementation.md DELETE FROM fts_objects to scope to run_id - Update README.md with two-phase discovery section and multi-agent legacy note - Create static_harvest.sh bash wrapper for Phase 1 - Create two_phase_discovery.py orchestration script with prompts - Add --run-id parameter to skip auto-fetch - Fix RUN_ID placeholder mismatch (<USE_THE_PROVIDED_RUN_ID>) - Fix catalog path default to mcp_catalog.db - Add test_catalog.sh to verify catalog tools work - Fix Discovery_Schema.cpp FTS5 syntax (missing space) - Remove invalid CREATE INDEX on FTS virtual tables - Add MCP tool call logging to track tool usage - Fix Static_Harvester::get_harvest_stats() to accept run_id parameter - Fix DELETE FROM fts_objects to only delete for specific run_id - Update system prompts to say DO NOT call discovery.run_static - Update user prompts to say Phase 1 is already complete - Add --mcp-only flag to restrict Claude Code to MCP tools only - Make FTS table failures non-fatal (check if table exists first) - Add comprehensive documentation for both discovery approaches	3 months ago
Rene Cannao	6f23d5bcd0	feat: Implement two-phase schema discovery architecture Phase 1 (Static/Deterministic): - Add Discovery_Schema: SQLite catalog with deterministic and LLM tables - Add Static_Harvester: MySQL INFORMATION_SCHEMA metadata extraction - Harvest schemas, objects, columns, indexes, foreign keys, view definitions - Compute derived hints: is_time, is_id_like, has_pk, has_fks, has_time - Build quick profiles and FTS5 indexes Phase 2 (LLM Agent): - Add 19 new MCP tools for two-phase discovery - discovery.run_static: Trigger ProxySQL's static harvest - Catalog tools: init, search, get_object, list_objects, get_relationships - Agent tools: run_start, run_finish, event_append - LLM tools: summary_upsert, relationship_upsert, domain_upsert, etc. Files: - include/Discovery_Schema.h, lib/Discovery_Schema.cpp - include/Static_Harvester.h, lib/Static_Harvester.cpp - include/Query_Tool_Handler.h, lib/Query_Tool_Handler.cpp (updated) - lib/Makefile (updated) - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/ - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/two_phase_discovery.py	3 months ago
Rene Cannao	7de3f0c510	feat: Add schema separation to MCP catalog and discovery scope constraint This commit addresses two issues: 1. MCP Catalog Schema Separation: - Add 'schema' column to catalog table for proper isolation - Update all catalog methods (upsert, get, search, list, remove) to accept schema parameter - Update MCP tool handlers and JSON-RPC parameter parsing - Unique constraint changed from (kind, key) to (schema, kind, key) - FTS table updated to include schema column 2. Discovery Prompt Scope Constraint: - Add explicit SCOPE CONSTRAINT section to multi_agent_discovery_prompt.md - Agents now respect Target Schema and skip list_schemas when specified - Prevents analyzing all schemas when only one is targeted Files modified: - include/MySQL_Catalog.h: Add schema parameter to all catalog methods - include/MySQL_Tool_Handler.h: Update wrapper method signatures - lib/MySQL_Catalog.cpp: Implement schema filtering in all operations - lib/MySQL_Tool_Handler.cpp: Update wrapper implementations - lib/Query_Tool_Handler.cpp: Extract schema from JSON-RPC params, update tool descriptions - scripts/mcp/DiscoveryAgent/ClaudeCode_Headless/prompts/multi_agent_discovery_prompt.md: Add scope constraint	3 months ago
Rene Cannao	a3f0bade4e	feat: Convert NL2SQL to generic LLM bridge - Rename NL2SQL_Converter to LLM_Bridge for generic prompt processing - Update MySQL protocol handler from /* NL2SQL: / to / LLM: / - Remove SQL-specific fields (sql_query, confidence, tables_used) - Add GENAI_OP_LLM operation type to GenAI module - Rename all genai_nl2sql_ variables to genai_llm_* - Update AI_Features_Manager to use LLM_Bridge - Deprecate ai_nl2sql_convert MCP tool with error message - LLM bridge now handles any prompt type via MySQL protocol This enables generic LLM access (summarization, code generation, translation, analysis) while preserving infrastructure for future NL2SQL implementation via Web UI + external agents.	3 months ago
Rene Cannao	3fe8a48f70	Fix genai variable handling and add API key masking - Add has_variable() method to GenAI_Threads_Handler for variable validation - Add genai- prefix check in is_valid_global_variable() - Auto-initialize NL2SQL converter when genai-nl2sql_enabled is set to true at runtime - Make init_nl2sql() public to allow runtime initialization - Mask API keys in logs (show only first 2 chars, rest as 'x')	3 months ago
Rene Cannao	1eb42c57d0	fix: Add GenAI variables to runtime_global_variables population Add flush_genai_variables___runtime_to_database() call to the central location where all modules populate runtime_global_variables table. This was missing, causing genai-* variables to not appear in runtime_global_variables.	3 months ago
Rene Cannao	6ffb59b856	fix: Use db parameter instead of hardcoded admindb in GenAI database_to_runtime The flush_genai_variables___database_to_runtime() function was using hardcoded 'admindb' instead of the 'db' parameter passed to the function. This caused the function to always query from admindb, ignoring the actual database parameter. This fixes the issue where runtime_global_variables table was not being populated on startup because the query was always hitting the same database regardless of the parameter.	3 months ago
Rene Cannao	4018a0ad3b	fix: Follow MCP pattern for GenAI variables runtime table population Update flush_genai_variables___database_to_runtime() to match the MCP pattern exactly: - Add 'lock' parameter (default true) for flexibility - Use ProxySQL_Admin's wrlock()/wrunlock() instead of GloGATH's - Use consistent variable naming (var_name = name + 6 for 'genai-' prefix) - Follow exact same locking pattern as MCP variables This fixes the issue where runtime_global_variables table was not being populated on startup because the locking pattern was incorrect.	3 months ago
Rene Cannao	1ea67900ab	fix: Populate runtime_global_variables for GenAI variables on startup The flush_genai_variables___database_to_runtime() function was only setting internal state in GloGATH but not populating the runtime_global_variables table. This caused variables to appear in global_variables but not in runtime_global_variables after startup. Fix: Add call to flush_genai_variables___runtime_to_database() with runtime=true to populate the runtime table, following the same pattern used by MCP variables.	3 months ago
Rene Cannao	51fd51e3f5	fix: Add missing GenAI_Thread.h include and fix variables reference Added missing #include for GenAI_Thread.h in AI_Features_Manager.cpp to resolve the compilation error in debug mode. Also fixed the remaining reference to variables.ai_features_enabled which should now use GloGATH->variables.genai_enabled. This fixes the "make debug" build failure.	3 months ago
Rene Cannao	a7dac5ef3d	feat: Make NL2SQL use async GenAI path instead of blocking calls This is a critical architectural fix - NL2SQL was making blocking calls to LLMs which would block the entire MySQL thread. Now NL2SQL uses the same async socketpair pattern as the GENAI embed/rerank operations. Changes: - Added nl2sql operation type to process_json_query() in GenAI module - Updated NL2SQL handler to construct JSON query and use async GENAI path - Added extern declaration for GloAI in GenAI_Thread.cpp - Falls back to synchronous path only on systems without epoll Architecture: - Before: NL2SQL: query → blocking nl2sql->convert() → blocks MySQL thread - After: NL2SQL: query → JSON GENAI request → async socketpair → non-blocking JSON protocol for NL2SQL: GENAI: {"type": "nl2sql", "query": "Show customers", "schema": "mydb"} The NL2SQL result is delivered asynchronously through the existing GENAI response handler, making the system fully non-blocking. Related to: https://github.com/ProxySQL/proxysql-vec/pull/13	3 months ago
Rene Cannao	527bfed297	fix: Migrate AI variables to GenAI module for proper architecture This commit fixes a serious design flaw where AI configuration variables were not integrated with the ProxySQL admin interface. All ai_* variables have been migrated to the GenAI module as genai-* variables. Changes: - Added 21 new genai_* variables to GenAI_Thread.h structure - Implemented get/set functions for all new variables in GenAI_Thread.cpp - Removed internal variables struct from AI_Features_Manager - AI_Features_Manager now reads from GloGATH instead of internal state - Updated documentation to reference genai-* variables - Fixed debug.cpp assertion for PROXY_DEBUG_NL2SQL and PROXY_DEBUG_ANOMALY Variable mapping: - ai_nl2sql_enabled → genai-nl2sql_enabled - ai_anomaly_detection_enabled → genai-anomaly_enabled - ai_features_enabled → genai-enabled - All other ai_* variables follow the same pattern The flush functions automatically handle all variables in the genai_thread_variables_names array, so database persistence works correctly without additional changes. Related to: https://github.com/ProxySQL/proxysql-vec/pull/13	3 months ago
Rene Cannao	2888ee3f45	Fix gemini-code-assist recommendations and implement comprehensive anomaly detection tests - Fix retry logic to use is_retryable_error function for proper HTTP error handling - Add exception handling to get_json_int function with try-catch around std::stoi - Improve validate_numeric_range to use strtol instead of atoi for better error reporting - Fix Chinese characters in documentation (non-zero -> non-zero) - Replace placeholder tests with actual comprehensive tests for anomaly detection functionality - Create new standalone unit test anomaly_detector_unit-t.cpp with 29 tests covering: * SQL injection pattern detection (12 tests) * Query normalization (8 tests) * Risk scoring calculations (5 tests) * Configuration validation (4 tests) - All tests pass successfully, providing meaningful validation of core anomaly detection logic Thanks to gemini-code-assist for the thorough code review and recommendations.	3 months ago
Rene Cannao	ae4200dbc0	Enhance AI features with improved validation, memory safety, error handling, and performance monitoring - Rename validate_provider_name to validate_provider_format for clarity - Add null checks and error handling for all strdup() operations - Enhance error messages with more context and HTTP status codes - Implement performance monitoring with timing metrics for LLM calls and cache operations - Add comprehensive test coverage for edge cases, retry scenarios, and performance - Extend status variables to track performance metrics - Update MySQL session to report timing information to AI manager	3 months ago
Rene Cannao	49092e9c8d	test: Add unit tests for AI configuration validation This commit adds comprehensive unit tests for the AI configuration validation functions used in AI_Features_Manager. Changes: - Add test/tap/tests/ai_validation-t.cpp with 61 unit tests - Test URL format validation (validate_url_format) - Test API key format validation (validate_api_key_format) - Test numeric range validation (validate_numeric_range) - Test provider name validation (validate_provider_name) - Test edge cases and boundary conditions The test file is self-contained with its own copies of the validation functions to avoid complex linking dependencies on libproxysql. Test Categories: - URL validation: 15 tests (http://, https:// protocols) - API key validation: 14 tests (OpenAI, Anthropic formats) - Numeric range: 13 tests (min/max boundaries) - Provider name: 8 tests (openai, anthropic) - Edge cases: 11 tests (NULL handling, long values) All 61 tests pass successfully. Part of: Phase 4 of NL2SQL improvement plan	3 months ago
Rene Cannao	8f38b8a577	feat: Add exponential backoff retry for transient LLM failures This commit adds configurable retry logic with exponential backoff for NL2SQL LLM API calls. Changes: - Add retry configuration to NL2SQLRequest (max_retries, retry_backoff_ms, retry_multiplier, retry_max_backoff_ms) - Add is_retryable_error() to identify retryable HTTP/CURL errors - Add sleep_with_jitter() for exponential backoff with 10% jitter - Add call_generic_openai_with_retry() wrapper - Add call_generic_anthropic_with_retry() wrapper - Update NL2SQL_Converter::convert() to use retry wrappers Default retry behavior: - 3 retries with 1000ms initial backoff - 2.0x multiplier, 30000ms max backoff - Retries on empty responses (transient failures) Part of: Phase 3 of NL2SQL improvement plan	3 months ago
Rene Cannao	d0dc36ac0b	feat: Add structured logging with timing and request IDs Add comprehensive structured logging for NL2SQL LLM API calls with request correlation, timing metrics, and detailed error context. Changes: - Add request_id field to NL2SQLRequest with UUID-like auto-generation - Add structured logging macros: * LOG_LLM_REQUEST: Logs URL, model, prompt length with request ID * LOG_LLM_RESPONSE: Logs HTTP status, duration_ms, response preview * LOG_LLM_ERROR: Logs error phase, message, and status code - Update call_generic_openai() signature to accept req_id parameter - Update call_generic_anthropic() signature to accept req_id parameter - Add timing metrics to both LLM call functions using clock_gettime() - Replace existing debug logging with structured logging macros - Update convert() to pass request_id to LLM calls Request IDs are generated as UUID-like strings (e.g., "12345678-9abc-def0-1234-567890abcdef") and are included in all log messages for correlation. This allows tracking a single NL2SQL request through all log lines from request to response. Timing is measured using CLOCK_MONOTONIC for accurate duration tracking of LLM API calls, reported in milliseconds. This provides much better debugging capability when troubleshooting NL2SQL issues, as administrators can now: - Correlate all log lines for a single request - See exact timing of LLM API calls - Identify which phase of processing failed - Track request/response metrics Fixes #2 - Add Structured Logging	3 months ago
Rene Cannao	45e592b623	feat: Add structured error messages with context to NL2SQL Add comprehensive error details to help users debug NL2SQL conversion issues. Changes: - Add error_code, error_details, http_status_code, provider_used fields to NL2SQLResult - Add NL2SQLErrorCode enum with structured error codes: * SUCCESS, ERR_API_KEY_MISSING, ERR_API_KEY_INVALID, ERR_TIMEOUT * ERR_CONNECTION_FAILED, ERR_RATE_LIMITED, ERR_SERVER_ERROR * ERR_EMPTY_RESPONSE, ERR_INVALID_RESPONSE, ERR_SQL_INJECTION_DETECTED * ERR_VALIDATION_FAILED, ERR_UNKNOWN_PROVIDER, ERR_REQUEST_TOO_LARGE - Add nl2sql_error_code_to_string() function for error code conversion - Add format_error_context() helper to create detailed error messages including: * Query (truncated if too long) * Schema name * Provider attempted * Endpoint URL * Specific error message - Add set_error_details() helper to populate error fields - Update error handling in convert() to use new error details - Track provider_used in successful conversions This provides much better debugging information when NL2SQL conversions fail, making it easier to identify misconfigurations and connectivity issues. Fixes #1 - Improve Error Messages	3 months ago
Rene Cannao	40b2608c2d	feat: Add configuration validation to AI_Features_Manager Add comprehensive validation for AI features configuration variables to prevent invalid states and improve error messages. Changes: - Add validate_url_format(): Checks for http:// or https:// prefix and host part - Add validate_api_key_format(): Validates API key format, checks for whitespace, minimum length, and incomplete key patterns (sk- with <20 chars, sk-ant- with <25 chars) - Add validate_numeric_range(): Validates numeric values are within min/max range - Add validate_provider_name(): Ensures provider is 'openai' or 'anthropic' - Update set_variable() to call validation functions before setting values Validated variables: - ai_nl2sql_provider: Must be 'openai' or 'anthropic' - ai_nl2sql_provider_url: Must have http:// or https:// prefix - ai_nl2sql_provider_key: No whitespace, minimum 10 chars - ai_nl2sql_cache_similarity_threshold: Range [0, 100] - ai_nl2sql_timeout_ms: Range [1000, 300000] (1 second to 5 minutes) - ai_nl2sql_max_cloud_requests_per_hour: Range [1, 10000] - ai_anomaly_similarity_threshold: Range [0, 100] - ai_anomaly_risk_threshold: Range [0, 100] - ai_anomaly_rate_limit: Range [1, 10000] - ai_vector_dimension: Range [128, 4096] This prevents misconfigurations and provides clear error messages to users when invalid values are provided. Fixes compilation issue by moving validation helper functions before set_variable() to resolve forward declaration errors.	3 months ago
Rene Cannao	36b11223b2	feat: Improve SQL validation with multi-factor scoring Add comprehensive SQL validation with confidence scoring based on: - SQL keyword detection (17 keywords covering DDL/DML/transactions) - Structural validation (balanced parentheses and quotes) - SQL injection pattern detection - Length and quality checks Confidence scoring: - Base 0.4 for valid SQL keyword - +0.15 for balanced parentheses - +0.15 for balanced quotes - +0.1 for minimum length - +0.1 for FROM clause in SELECT statements - +0.1 for no injection patterns - -0.3 penalty for injection patterns detected Low confidence (< 0.5) results are logged with detailed info. Cache storage threshold updated to 0.5 confidence (from implicit valid_sql). This improves detection of malformed or potentially malicious SQL while providing granular confidence scores for downstream use.	3 months ago
Rene Cannao	897d306d2d	Refactor: Simplify NL2SQL to use only generic providers Remove Ollama-specific provider code and use only generic OpenAI-compatible and Anthropic-compatible providers. Ollama is now used via its OpenAI-compatible endpoint at /v1/chat/completions. Changes: - Remove LOCAL_OLLAMA from ModelProvider enum - Remove ai_nl2sql_ollama_model and ai_nl2sql_ollama_url variables - Remove call_ollama() function from LLM_Clients.cpp - Update default configuration to use OpenAI provider with Ollama URL - Update all documentation to reflect generic-only approach Configuration: - ai_nl2sql_provider: 'openai' or 'anthropic' (default: 'openai') - ai_nl2sql_provider_url: endpoint URL (default: Ollama OpenAI-compatible) - ai_nl2sql_provider_model: model name - ai_nl2sql_provider_key: API key (optional for local endpoints) This simplifies the codebase by removing a separate code path for Ollama and aligns with the goal of avoiding provider-specific variables.	3 months ago
Rene Cannao	637b2a669c	feat: Implement NL2SQL vector cache and complete Anomaly threat pattern management NL2SQL_Converter improvements: - Implement get_query_embedding() using GenAI module - Implement check_vector_cache() with KNN search via sqlite-vec - Implement store_in_vector_cache() with embedding storage - All stub methods now fully functional Anomaly_Detector improvements: - Implement add_threat_pattern() with embedding generation - Stores patterns in both main table and virtual vec table - Returns pattern ID on success, -1 on error Documentation: - Add comprehensive VECTOR_FEATURES documentation - README.md (471 lines): User guide and quick start - API.md (736 lines): Complete API reference - ARCHITECTURE.md (358 lines): System architecture - TESTING.md (767 lines): Testing guide and procedures This completes the vector features implementation, enabling: - Semantic similarity caching for NL2SQL queries - Embedding-based threat pattern detection - Full CRUD operations for threat patterns	3 months ago
Rene Cannao	782f6cb66b	feat: Implement threat pattern management and improve statistics Improve Anomaly_Detector with full threat pattern CRUD operations: Changes to lib/Anomaly_Detector.cpp: - Implement list_threat_patterns(): * Returns JSON array of all threat patterns * Shows pattern_name, pattern_type, query_example, severity, created_at * Ordered by severity DESC (highest risk first) - Implement remove_threat_pattern(): * Deletes from both anomaly_patterns and anomaly_patterns_vec tables * Proper error handling with error messages * Returns true on success, false on failure - Improve get_statistics(): * Add threat_patterns_count to statistics * Add threat_patterns_by_type breakdown * Shows patterns grouped by type (sql_injection, dos, etc.) - Add count_by_pattern_type query for categorization Features: - Full CRUD operations for threat patterns - JSON-formatted output for API integration - Statistics include both counts and categorization - Proper cleanup of both main and virtual tables	3 months ago
Rene Cannao	1c7cd8c2b1	fix: Correct PROXY_DEBUG constant from AI_GENERIC to GENAI	3 months ago
Rene Cannao	f226c0e687	feat: Implement embedding-based threat similarity for Anomaly Detection Implemented embedding-based threat pattern detection using GenAI and sqlite-vec: Changes to lib/Anomaly_Detector.cpp: - Add GenAI_Thread.h include and GloGATH extern - Implement get_query_embedding(): * Calls GloGATH->embed_documents() via llama-server * Normalizes query before embedding for better quality * Returns std::vector<float> with embedding - Implement check_embedding_similarity(): * Generates embedding for query if not provided * Performs sqlite-vec KNN search against anomaly_patterns table * Uses cosine distance (vec_distance_cosine) for similarity * Calculates risk score based on severity and distance * Returns AnomalyResult with pattern details and blocking decision - Implement add_threat_pattern(): * Generates embedding for threat pattern example * Stores pattern with embedding in anomaly_patterns table * Updates anomaly_patterns_vec virtual table for KNN search * Returns pattern ID on success Features: - Semantic similarity detection against known threat patterns - Configurable similarity threshold (ai_anomaly_similarity_threshold) - Risk scoring based on pattern severity (1-10) and similarity - Automatic threat pattern management with vector indexing	3 months ago

1 2 3 4 5 ...

4672 Commits (cc3cc25532a00dd052a48cf757de7b1def1b95db)