9 Commits (9ffc3f8d711d2f1d3373c91a135d40cbea202841)

Author	SHA1	Message	Date
Rene Cannao	48bc7dd7bf	Merge v4.0 GenAI features into v3.0 with conditional compilation ... This commit merges the experimental v4.0 GenAI/MCP features into the stable v3.0 branch using conditional compilation. All v4.0 features are disabled by default and only enabled when PROXYSQLGENAI=1 is set at compile time. Changes: Build System: - Modified main Makefile to pass PROXYSQLGENAI flag to sub-makefiles - Modified deps/Makefile to conditionally build sqlite-vec and sqlite-rembed - Modified lib/Makefile to add PSQLGA flag and include GenAI object files - Modified src/Makefile to add PSQLGA flag and conditional linking Headers (wrapped with #ifdef PROXYSQLGENAI): - All 20 new GenAI header files in include/ - Modified cpp.h, proxysql_glovars.hpp, proxysql_admin.h - Modified ProxySQL_Admin_Tables_Definitions.h for GenAI/MCP tables Source Files: - All 22 new GenAI source files in lib/ wrapped with #ifdef PROXYSQLGENAI - Modified src/main.cpp for conditional global variables and init/shutdown - Modified Admin_Handler.cpp for conditional command handlers - Modified Admin_Bootstrap.cpp for conditional table registration - Modified Admin_FlushVariables.cpp for conditional variable flushing - Modified ProxySQL_Admin.cpp for conditional admin methods - Modified ProxySQL_Admin_Stats.cpp for conditional MCP stats functions - Modified proxy_sqlite3_symbols.cpp to always compile (needed by core) - Modified MySQL_Session.cpp for conditional GenAI function calls Test Files: - Renamed test_mcp_query_rules-t to test_mcp_query_rules-t.sh - Renamed test_mcp_rag_metrics-t to test_mcp_rag_metrics-t.sh - Modified anomaly_detection-t.cpp for conditional test execution Usage: # Build without GenAI (v3.0 mode - default) make clean && make build_deps -j$(nproc) && make build_lib -j$(nproc) && make build_src -j$(nproc) # Build with GenAI (v4.0 mode) make clean && PROXYSQLGENAI=1 make build_deps -j$(nproc) && PROXYSQLGENAI=1 make build_lib -j$(nproc) && PROXYSQLGENAI=1 make build_src -j$(nproc)	2 weeks ago
Rene Cannao	a24b8adaa3	Use proxy_sqlite3_* for SQLite calls in Anomaly_Detector.cpp (address PR review)	1 month ago
Rene Cannao	8dc4246bdc	Introduce canonical proxy_sqlite3 symbol TU; update lib Makefile; remove MAIN_PROXY_SQLITE3 from main.cpp	1 month ago
Rene Cannao	637b2a669c	feat: Implement NL2SQL vector cache and complete Anomaly threat pattern management ... NL2SQL_Converter improvements: - Implement get_query_embedding() using GenAI module - Implement check_vector_cache() with KNN search via sqlite-vec - Implement store_in_vector_cache() with embedding storage - All stub methods now fully functional Anomaly_Detector improvements: - Implement add_threat_pattern() with embedding generation - Stores patterns in both main table and virtual vec table - Returns pattern ID on success, -1 on error Documentation: - Add comprehensive VECTOR_FEATURES documentation - README.md (471 lines): User guide and quick start - API.md (736 lines): Complete API reference - ARCHITECTURE.md (358 lines): System architecture - TESTING.md (767 lines): Testing guide and procedures This completes the vector features implementation, enabling: - Semantic similarity caching for NL2SQL queries - Embedding-based threat pattern detection - Full CRUD operations for threat patterns	1 month ago
Rene Cannao	782f6cb66b	feat: Implement threat pattern management and improve statistics ... Improve Anomaly_Detector with full threat pattern CRUD operations: Changes to lib/Anomaly_Detector.cpp: - Implement list_threat_patterns(): * Returns JSON array of all threat patterns * Shows pattern_name, pattern_type, query_example, severity, created_at * Ordered by severity DESC (highest risk first) - Implement remove_threat_pattern(): * Deletes from both anomaly_patterns and anomaly_patterns_vec tables * Proper error handling with error messages * Returns true on success, false on failure - Improve get_statistics(): * Add threat_patterns_count to statistics * Add threat_patterns_by_type breakdown * Shows patterns grouped by type (sql_injection, dos, etc.) - Add count_by_pattern_type query for categorization Features: - Full CRUD operations for threat patterns - JSON-formatted output for API integration - Statistics include both counts and categorization - Proper cleanup of both main and virtual tables	1 month ago
Rene Cannao	f226c0e687	feat: Implement embedding-based threat similarity for Anomaly Detection ... Implemented embedding-based threat pattern detection using GenAI and sqlite-vec: Changes to lib/Anomaly_Detector.cpp: - Add GenAI_Thread.h include and GloGATH extern - Implement get_query_embedding(): * Calls GloGATH->embed_documents() via llama-server * Normalizes query before embedding for better quality * Returns std::vector<float> with embedding - Implement check_embedding_similarity(): * Generates embedding for query if not provided * Performs sqlite-vec KNN search against anomaly_patterns table * Uses cosine distance (vec_distance_cosine) for similarity * Calculates risk score based on severity and distance * Returns AnomalyResult with pattern details and blocking decision - Implement add_threat_pattern(): * Generates embedding for threat pattern example * Stores pattern with embedding in anomaly_patterns table * Updates anomaly_patterns_vec virtual table for KNN search * Returns pattern ID on success Features: - Semantic similarity detection against known threat patterns - Configurable similarity threshold (ai_anomaly_similarity_threshold) - Risk scoring based on pattern severity (1-10) and similarity - Automatic threat pattern management with vector indexing	1 month ago
Rene Cannao	52a70b0b09	feat: Implement AI-based Anomaly Detection for ProxySQL ... Phase 3: Anomaly Detection Implementation This commit implements a comprehensive multi-stage anomaly detection system for real-time SQL query security analysis. **Core Detection Methods:** 1. **SQL Injection Pattern Detection** (lib/Anomaly_Detector.cpp) - Regex-based detection of 11 SQL injection patterns - Suspicious keyword detection (11 patterns) - Covers: tautologies, union-based, comment-based, stacked queries 2. **Query Normalization** (lib/Anomaly_Detector.cpp:normalize_query) - Converts to lowercase - Removes SQL comments - Replaces string/numeric literals with placeholders - Normalizes whitespace 3. **Rate Limiting** (lib/Anomaly_Detector.cpp:check_rate_limiting) - Per user/host query rate tracking - Configurable time windows (3600s default) - Auto-block on threshold exceeded - Prevents DoS and brute force attacks 4. **Statistical Anomaly Detection** (lib/Anomaly_Detector.cpp:check_statistical_anomaly) - Z-score based outlier detection - Abnormal execution time detection (>5s) - Large result set detection (>10000 rows) - Behavioral profiling per user 5. **Embedding-based Similarity** (lib/Anomaly_Detector.cpp:check_embedding_similarity) - Placeholder for vector similarity search - Framework for sqlite-vec integration - Detects novel attack variations **Query Flow Integration:** - Added `detect_ai_anomaly()` to MySQL_Session (line 3626) - Integrated after libinjection SQLi detection (line 5150) - Blocks queries when risk threshold exceeded (default: 0.70) - Sends error response with anomaly details **Status Variables Added:** - `ai_detected_anomalies`: Total anomalies detected - `ai_blocked_queries`: Total queries blocked - Available via: `SELECT * FROM stats_mysql_global` **Configuration (defaults):** - `enabled`: true - `risk_threshold`: 70 (0-100) - `similarity_threshold`: 85 (0-100) - `rate_limit`: 100 queries/hour - `auto_block`: true - `log_only`: false **Detection Pipeline:** ``` Query → SQLi Check → AI Anomaly Check → [Block if needed] → Execute (libinjection) (Multi-stage) ``` **Files Modified:** - include/MySQL_Session.h: Added detect_ai_anomaly() declaration - include/MySQL_Thread.h: Added AI status variables - lib/Anomaly_Detector.cpp: Full implementation (700+ lines) - lib/MySQL_Session.cpp: Integration and query flow - lib/MySQL_Thread.cpp: Status variable definitions **Next Steps:** - Add unit tests for each detection method - Add integration tests with sample attacks - Add user and developer documentation Related: Phase 1-2 (NL2SQL foundation and testing) Related: Phase 4 (Vector storage for embeddings)	1 month ago
Rene Cannao	af68f347d4	fix: Add missing verbosity level to proxy_debug call in Anomaly_Detector ... The proxy_debug macro requires a verbosity level as the second parameter. Fixed the call in Anomaly_Detector::analyze() to include the level.	1 month ago
Rene Cannao	d9346fe64d	feat: Add AI features manager foundation ... - Add AI_Features_Manager coordinator class - Add AI_Vector_Storage interface (stub) - Add Anomaly_Detector class (stub for Phase 3) - Update includes and main initialization	1 month ago