proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	3daaa5c592	feat: Implement RAG (Retrieval-Augmented Generation) subsystem Adds a complete RAG subsystem to ProxySQL with: - RAG_Tool_Handler implementing all MCP tools for retrieval operations - Database schema with FTS and vector support - FTS, vector, and hybrid search capabilities - Fetch and refetch tools for document/chunk retrieval - Admin tools for monitoring - Configuration variables for RAG parameters - Comprehensive documentation and test scripts Implements v0 deliverables from RAG blueprint: - SQLite schema initialization - Source registry management - MCP tools: search_fts, search_vector, search_hybrid, get_chunks, get_docs, fetch_from_source, admin.stats - Unit/integration tests and examples	3 months ago
Rene Cannao	3f25fb6c4e	Enhance anomaly detector unit tests with additional edge case coverage - Add edge case tests for SQL injection pattern detection: * Empty and NULL query handling * Very long query processing - Add edge case tests for query normalization: * Empty query handling * Unicode character support * Nested comment processing * Multiple line comment handling - Add edge case tests for risk scoring: * Negative match handling * Large match count capping * Boundary condition validation - Fix test assertions for more robust validation - All 39 tests now pass successfully These improvements address potential feedback from code reviewers by providing more comprehensive test coverage for edge cases and boundary conditions.	3 months ago
Rene Cannao	2888ee3f45	Fix gemini-code-assist recommendations and implement comprehensive anomaly detection tests - Fix retry logic to use is_retryable_error function for proper HTTP error handling - Add exception handling to get_json_int function with try-catch around std::stoi - Improve validate_numeric_range to use strtol instead of atoi for better error reporting - Fix Chinese characters in documentation (non-zero -> non-zero) - Replace placeholder tests with actual comprehensive tests for anomaly detection functionality - Create new standalone unit test anomaly_detector_unit-t.cpp with 29 tests covering: * SQL injection pattern detection (12 tests) * Query normalization (8 tests) * Risk scoring calculations (5 tests) * Configuration validation (4 tests) - All tests pass successfully, providing meaningful validation of core anomaly detection logic Thanks to gemini-code-assist for the thorough code review and recommendations.	3 months ago
Rene Cannao	ae4200dbc0	Enhance AI features with improved validation, memory safety, error handling, and performance monitoring - Rename validate_provider_name to validate_provider_format for clarity - Add null checks and error handling for all strdup() operations - Enhance error messages with more context and HTTP status codes - Implement performance monitoring with timing metrics for LLM calls and cache operations - Add comprehensive test coverage for edge cases, retry scenarios, and performance - Extend status variables to track performance metrics - Update MySQL session to report timing information to AI manager	3 months ago
Rene Cannao	3032dffed4	test: Add NL2SQL internal functionality unit tests Add comprehensive TAP unit tests for NL2SQL internal functions: - Error code conversion (5 tests): Validate nl2sql_error_code_to_string() covers all 13 defined error codes plus UNKNOWN_ERROR - SQL validation patterns (17 tests): Test validate_and_score_sql() * Valid SELECT queries (4 tests) * Non-SELECT queries (4 tests) * Injection pattern detection (4 tests) * Edge cases (4 tests): empty, lone keyword, semicolons, complex queries - Request ID generation (12 tests): Test UUID-like ID generation * Format validation (20 assertions for 10 IDs) * Uniqueness (100 IDs checked for duplicates) * Hexadecimal character validation - Prompt building (8 tests): Test build_prompt() * Basic prompt structure (3 tests) * Schema context inclusion (3 tests) * Section ordering (1 test) * Special character handling (2 tests) Note: Tests are self-contained with standalone implementations matching the logic in NL2SQL_Converter.cpp.	3 months ago
Rene Cannao	49092e9c8d	test: Add unit tests for AI configuration validation This commit adds comprehensive unit tests for the AI configuration validation functions used in AI_Features_Manager. Changes: - Add test/tap/tests/ai_validation-t.cpp with 61 unit tests - Test URL format validation (validate_url_format) - Test API key format validation (validate_api_key_format) - Test numeric range validation (validate_numeric_range) - Test provider name validation (validate_provider_name) - Test edge cases and boundary conditions The test file is self-contained with its own copies of the validation functions to avoid complex linking dependencies on libproxysql. Test Categories: - URL validation: 15 tests (http://, https:// protocols) - API key validation: 14 tests (OpenAI, Anthropic formats) - Numeric range: 13 tests (min/max boundaries) - Provider name: 8 tests (openai, anthropic) - Edge cases: 11 tests (NULL handling, long values) All 61 tests pass successfully. Part of: Phase 4 of NL2SQL improvement plan	3 months ago
Rene Cannao	1a8b406d9b	fix: Correct SQL query for AI variables in vector features test - Change from runtime_mysql_servers with variable_name column - To mysql_servers with ai_* prefix columns - This matches the actual schema where AI variables are stored	3 months ago
Rene Cannao	4b0cb9d95a	test: Add vector features unit test Add unit test for vector features including: - Virtual vec0 table creation verification - NL2SQL vector cache configuration tests - Anomaly embedding configuration tests - Vector database file verification - Status variables validation - Cache statistics interface tests - GenAI module availability checks 20 tests covering configuration and infrastructure validation. Tests can be extended with actual embedding generation once llama-server is running in the test environment.	3 months ago
Rene Cannao	0be9715188	test: Add comprehensive tests and documentation for Anomaly Detection Added 95 tests (50 unit + 45 integration) and 4 documentation files: Test Files: - test/tap/tests/anomaly_detection-t.cpp (50 unit tests) * Initialization and configuration tests * SQL injection pattern detection * Query normalization * Rate limiting * Statistical anomaly detection * Integration scenarios * Configuration management * False positive handling - test/tap/tests/anomaly_detection_integration-t.cpp (45 integration tests) * Real SQL injection pattern detection with actual queries * Legitimate query passthrough verification * Multi-user rate limiting scenarios * Statistical anomaly detection * Log-only mode configuration Documentation (doc/ANOMALY_DETECTION/): - README.md: User guide with quick start, configuration, examples - API.md: Complete API reference for Anomaly_Detector class - ARCHITECTURE.md: System architecture and design documentation - TESTING.md: Testing guide with test categories and examples All tests compile successfully and follow the TAP framework pattern used throughout ProxySQL.	3 months ago
Rene Cannao	eccb2bfe4d	test: Add integration tests for NL2SQL - nl2sql_integration-t.cpp: Schema-aware conversion, multi-table queries - Tests JOIN queries, aggregations, complex patterns - Tests error recovery and cross-schema queries - 30 tests across 6 categories Tests require running ProxySQL instance with admin interface to create test schema and validate SQL generation.	3 months ago
Rene Cannao	6d2b0ab303	test: Fix vector keyword conflict in NL2SQL unit tests Add 'using std::vector;' declaration to resolve conflicts with macros defined in included headers.	3 months ago
Rene Cannao	a61f709c7b	test: Add comprehensive TAP unit tests for NL2SQL - nl2sql_unit_base-t.cpp: Initialization, configuration, persistence, error handling - nl2sql_prompt_builder-t.cpp: Prompt construction, schema context, edge cases - nl2sql_model_selection-t.cpp: Model routing logic, latency handling, fallback Tests follow ProxySQL TAP framework patterns and use CommandLine helper for environment-based configuration.	3 months ago
René Cannaò	313f637cf0	Merge branch 'v3.1-vec' into v3.1-MCP1 Signed-off-by: René Cannaò <rene.cannao@gmail.com>	3 months ago
Rene Cannao	40cff23c3b	Initialize MySQL Tool Handler and fix default MySQL port - Initialize MySQL_Tool_Handler in ProxySQL_MCP_Server constructor with MySQL configuration from MCP variables - Use GloVars.get_SSL_pem_mem() to get SSL certificates correctly - Add MySQL_Tool_Handler cleanup in destructor - Change configure_mcp.sh default MySQL port from 3307 to 3306 - Change configure_mcp.sh default password from test123 to empty - Update help text and examples to match new defaults	3 months ago
Rene Cannao	33a100c1db	Use relative path mcp_catalog.db in MCP test instead of absolute /var/lib/proxysql path	3 months ago
Rene Cannao	5a85ef04f6	Fix MCP variables persistence and add DISK command support This commit fixes several issues with MCP (Model Context Protocol) variables not being properly persisted across storage layers and adds support for DISK commands. Changes: 1. lib/Admin_FlushVariables.cpp: - Fixed flush_mcp_variables___runtime_to_database() to properly insert variables into runtime_global_variables using db->execute() with formatted strings (matching admin pattern) - Fixed SQL format string to avoid double-prefix bug (qualified_name already contains "mcp-" prefix) - Fixed lock ordering by releasing outer wrlock before calling runtime_to_database with use_lock=true, then re-acquiring - Removed explicit BEGIN/COMMIT transactions to match admin pattern 2. lib/Admin_Handler.cpp: - Added MCP DISK command handlers that rewrite commands to SQL queries: * LOAD MCP VARIABLES FROM DISK -> INSERT OR REPLACE INTO main.global_variables * SAVE MCP VARIABLES TO DISK -> INSERT OR REPLACE INTO disk.global_variables * SAVE MCP VARIABLES FROM MEMORY/MEM -> INSERT OR REPLACE INTO disk.global_variables - Separated DISK command handlers from MEMORY/RUNTIME handlers 3. lib/ProxySQL_Admin.cpp: - Added flush_mcp_variables___runtime_to_database() call to stats section to ensure MCP variables are repopulated when runtime_global_variables is cleared and refreshed 4. tests/mcp_module-t.cpp: - Added verbose diagnostic output throughout tests - Added section headers and test numbers for clarity - Added variable value logging and error logging All 52 MCP module tests now pass.	3 months ago
Rene Cannao	221ff23991	Add MySQL exploration MCP tools with SQLite catalog Implemented MCP (Model Context Protocol) server providing tools for LLM-based MySQL database exploration: - MySQL_Catalog: SQLite-based catalog for LLM external memory with upsert, get, search, list, merge, delete operations and FTS support - MySQL_Tool_Handler: 17+ database exploration tools with guardrails: * Inventory: list_schemas, list_tables * Structure: describe_table, get_constraints, describe_view * Profiling: table_profile, column_profile * Sampling: sample_rows (max 20), sample_distinct (max 50) * Query: run_sql_readonly (max 200 rows, 2s timeout, SELECT-only) * Relationship: suggest_joins, find_reference_candidates * Catalog: catalog_upsert, catalog_get, catalog_search, catalog_list, catalog_merge, catalog_delete - MCP Module Integration: * Added 6 new configuration variables for MySQL tool handler (mysql_hosts, mysql_ports, mysql_user, mysql_password, mysql_schema, catalog_path) * Added MySQL_Tool_Handler pointer to MCP_Threads_Handler * Implemented tool routing in MCP endpoint for tools/list, tools/describe, and tools/call methods - TAP Tests: Updated to expect 14 MCP variables (was 8) Files: - include/MySQL_Catalog.h, lib/MySQL_Catalog.cpp - include/MySQL_Tool_Handler.h, lib/MySQL_Tool_Handler.cpp - include/MCP_Thread.h, lib/MCP_Thread.cpp - include/MCP_Endpoint.h, lib/MCP_Endpoint.cpp - lib/Makefile, test/tap/tests/mcp_module-t.cpp	3 months ago
Rene Cannao	245e61ee86	Make MCP_Threads_Handler a standalone independent class Remove unnecessary inheritance from MySQL_Threads_Handler. The MCP module should be independent and not depend on MySQL/PostgreSQL thread handlers. Changes: - MCP_Threads_Handler now manages its own pthread_rwlock_t for synchronization - Simplified init() signature (removed unused num/stack parameters) - Added ProxySQL_Main_init_MCP_module() call in main initialization phase - Include only standard C++ headers (pthread.h, cstring, cstdlib)	3 months ago
Rene Cannao	de3fd05a5a	Reverted change to test/tap/tests/.env	3 months ago
Rene Cannao	87fff9e046	Add MCP (Model Context Protocol) module skeleton Add new MCP module supporting multiple MCP server endpoints over HTTPS with JSON-RPC 2.0 protocol skeleton. Each endpoint (/mcp/config, /mcp/observe, /mcp/query, /mcp/admin, /mcp/cache) is a distinct MCP server with its own authentication configuration. Features: - HTTPS server using existing ProxySQL TLS certificates - JSON-RPC 2.0 skeleton implementation (actual protocol TBD) - 5 MCP endpoints with per-endpoint auth configuration - LOAD/SAVE MCP VARIABLES admin commands - Configuration file support (mcp_variables section) Implementation follows GenAI module pattern: - MCP_Threads_Handler: Main module handler with variable management - ProxySQL_MCP_Server: HTTPS server wrapper using libhttpserver - MCP_JSONRPC_Resource: Base endpoint class with JSON-RPC skeleton	3 months ago
Rene Cannao	bdd2ef70e6	Add comprehensive TAP tests for GenAI async architecture This commit adds extensive TAP tests for the GenAI module's async architecture, verifying the non-blocking socketpair-based implementation. Test coverage (28 tests): - Single async request handling - Sequential embedding and rerank requests (5 each) - Batch requests (10 documents) - Mixed embedding and rerank requests - Request/response matching with varying document counts - Error handling (invalid JSON, missing fields, etc.) - Special characters (quotes, backslashes, unicode, etc.) - Large documents (5KB single, 5x1KB batch) - top_n parameter for limiting rerank results - columns parameter for result format - Concurrent connections (3 simultaneous requests) Test file: test/tap/tests/genai_async-t.cpp All tests pass successfully when GenAI module is initialized and llama-server is running at the configured endpoints. Note: These tests require: - GenAI module initialized (genai-threads > 0) - Running llama-server with embedding and rerank endpoints - Epoll support (Linux systems)	3 months ago
Rene Cannao	db2485be37	Add comprehensive doxygen documentation to GenAI async module This commit adds extensive doxygen-format documentation to all key functions in the GenAI async module to improve code maintainability and API clarity. Documented functions: - lib/GenAI_Thread.cpp: - unregister_client() - cleanup flow and usage - call_llama_batch_embedding() - HTTP client with JSON format - call_llama_rerank() - HTTP client with JSON format - execute_sql_for_documents() - stub for document_from_sql - process_json_query() - autonomous JSON query processing - lib/MySQL_Session.cpp: - genai_send_async() - async flow and error handling - handle_genai_response() - response handling flow - genai_cleanup_request() - resource cleanup details - check_genai_events() - main loop integration Enhanced header documentation: - GenAI_RequestHeader - communication flow details - GenAI_ResponseHeader - response format details - register_client() - registration flow - unregister_client() - cleanup flow - embed_documents() - BLOCKING warning - rerank_documents() - BLOCKING warning - process_json_query() - supported formats All documentation includes: - @brief descriptions - @param parameter details - @return return value explanations - @note important warnings and usage notes - @see cross-references to related functions - Detailed workflow descriptions - Error handling details - Memory management notes	3 months ago
Rene Cannao	bbad8ab4f3	Fix GenAI variable naming and add comprehensive TAP tests - Fix double prefix bug in genai_thread_variables_names[] where variable names included the "genai_" prefix, but flush functions added "genai-" prefix, creating names like "genai-genai_threads" - Update get_variable() and set_variable() to use names without prefix - Add comprehensive TAP tests for GenAI embedding and reranking with 40 tests covering configuration, single/batch embedding, reranking, error handling, and GENAI: query syntax variations - Fix test expectations for leading space behavior (should be rejected) - Add tests for genai-embedding_timeout_ms and genai-rerank_timeout_ms	3 months ago
Rene Cannao	99dbd0a358	Add TAP test for GenAI module Add comprehensive TAP test for the GenAI module that validates: LOAD/SAVE Commands: - All 14 standard command variants (TO MEMORY/FROM DISK/TO RUNTIME, etc.) - Verifies commands execute successfully Variable Access: - SET operations for genai-var1 and genai-var2 - SELECT operations to retrieve variable values - Default values and special character handling Persistence Layers: - Memory layer (main.global_variables) - Disk layer (disk.global_variables) - Runtime layer (GloGATH handler object) - SAVE/LOAD operations between layers CHECKSUM Commands: - CHECKSUM DISK GENAI VARIABLES - CHECKSUM MEM/MEMORY GENAI VARIABLES - CHECKSUM GENAI VARIABLES (default) The test includes proper documentation and follows ProxySQL's TAP test conventions. Tests will fully pass once the GenAI module is completely implemented with variable registration.	3 months ago
Javier Jaramago Fernández	e73ba2b64c	Fix changing server defaults for 'reg_test_unexp_ping_pkt' - Fixed was changing server defaults ('max_allowed_packet') in the same connection as the subsequent INSERT. - Use direct MySQL connection for table/data preparation. - ProxySQL connection pool required cleanup 'connection_pool' to ensure defaults reload.	4 months ago
Javier Jaramago Fernández	7efc85cee3	Fix TAP 'groups.json' file syntax	4 months ago
Javier Jaramago Fernández	5c8a32a0b0	Merge branch 'v3.0' of github.com:sysown/proxysql into v3.0-handle_unexp_ping	4 months ago
Javier Jaramago Fernández	b9cf5b1689	Remove empty spaces at end of lines for TAP tests 'groups.json'	4 months ago
Javier Jaramago Fernández	54e2b3e6dc	Add new test 'reg_test_unexp_ping_pkt-t' to TAP groups	4 months ago
René Cannaò	0f7ff1f374	Merge branch 'v3.0' into v3.0_pgsql-query-digest-gen-5253 Signed-off-by: René Cannaò <rene@proxysql.com>	4 months ago
René Cannaò	2667540fcc	Merge pull request #5237 from sysown/v3.0_pgsql-monitor-sslsupport-5205 Add SSL support for backend connections in PGSQL monitor	4 months ago
Javier Jaramago Fernández	503b0975c6	Add new TAP test exercising unexpected COM_PING packets logic Also fixes 'random_string' utility function for large strings, avoiding the previous shuffle repetition from the dictionary.	4 months ago
Rahim Kanji	febb650abb	Added Multiple Keep Comments test	4 months ago
Rahim Kanji	5e75264bb3	Updated TAP test	4 months ago
Rahim Kanji	e70fcbf021	* Add dedicated handling for double-quoted PostgreSQL identifiers * Added crash payload testing * Fixed unterminated comments handling	4 months ago
Rahim Kanji	fd53642f12	Added pgsql-query_digests_stages_test-t to groups.json	4 months ago
Rahim Kanji	f9b9ede64c	Fixed TAP test	4 months ago
Rahim Kanji	1b16a3938b	Added TAP test	4 months ago
Rene Cannao	66119b74fb	Add comments to select @@version queries to bypass ProxySQL interception ProxySQL now responds directly to `select @@version` queries, which prevents these queries from reaching the backend MySQL server. This commit adds `/* set_testing */` comments to these queries in test files to prevent perfect match interception and allow them to run on the backend. Modified files: - test/tap/tap/utils.cpp - test/tap/tests/set_testing-240.h - test/tap/tests/set_testing.h - test/tap/tests/test_firewall-t.cpp	4 months ago
René Cannaò	5a314d2364	Merge pull request #4889 from sysown/v3.0_get_server_version Added handling of SELECT @@version and SELECT VERSION() without backend	4 months ago
Rahim Kanji	d1b003a0e4	Added TAP test to groups.json	4 months ago
Rahim Kanji	0e7b5e2ba8	Added TAP test	4 months ago
Rene Cannao	a206828609	Add test_load_from_config_prefix_stripping-t to test groups Adds the new prefix stripping validation test to the default-g4 test group along with mysql-auto_increment_delay_multiplex=0-g4, mysql-multiplexing=false-g4, mysql-query_digests=0-g4, and mysql-query_digests_keep_comment=1-g4 groups. This ensures the test is run as part of the regular test suite.	4 months ago
Rene Cannao	7ebdf561ca	Fix automatic prefix stripping to work with libconfig lookup The previous implementation stripped the prefix before calling group.lookupValue(), which would fail because the config file contains the prefixed name (e.g., "mysql-log_unhealthy_connections"). The lookup must use the original name from the config file. This commit moves the prefix stripping logic to after the value lookup but before constructing the SQL query, ensuring both: 1. The correct value is retrieved from the config using the original prefixed name 2. The variable is stored in the database with a single prefix Also includes a test to verify the fix works for mysql_variables, pgsql_variables, and admin_variables sections.	4 months ago
René Cannaò	b73160ef5f	Merge pull request #4901 from sysown/v3.0_wait_timeout [WIP] Setting client side wait_timeout	5 months ago
René Cannaò	0d55ab5ea2	Merge branch 'v3.0' into v3.0_get_server_version Signed-off-by: René Cannaò <rene@proxysql.com>	5 months ago
Rene Cannao	2b900b3465	Update test groups configuration	5 months ago
Rene Cannao	dc4694d656	Refactor idle session scanning and improve test precision Code improvements: - Extract SESS_TO_SCAN_idle_thread constant to header file for better maintainability - Replace magic number 128 with named constant in idle_thread_to_kill_idle_sessions() - Improve code readability and consistency in session scanning logic Test enhancements: - Add mysql-poll_timeout configuration for more precise timeout testing - Reduce test sleep times to 13 seconds for faster test execution - Add diagnostic messages to clearly show timeout configurations in test output - Ensure tests properly validate timeout enforcement with precise timing The changes improve code maintainability and make tests more reliable and faster while maintaining accurate timeout validation.	5 months ago
Rene Cannao	0c5e75a064	Fix wait_timeout timeout calculations and add proper newline characters Key improvements: - Fix timeout comparison in MySQL_Thread::idle_thread_to_kill_idle_sessions() to prevent underflow - Use effective wait_timeout (minimum of global and session values) for idle timeout calculations - Add proper newline characters to proxy_warning messages for consistent log formatting - Increase test sleep times to account for global timeout enforcement - Fix session timeout test durations to properly test timeout behavior Technical changes: - Replace broken min_idle calculation with proper effective wait_timeout logic - Add std::min() usage to determine effective timeout from global and session values - Ensure warning messages end with newline characters for proper log formatting - Update test sleep durations to ensure proper timeout testing Resolves potential timeout calculation bugs and ensures consistent timeout enforcement behavior.	5 months ago
René Cannaò	6534a152b2	Merge pull request #5241 from sysown/v3.0-5062_2 Add fast forward replication deprecate EOF test and update test infrastructure (closes #5062)	5 months ago

1 2 3 4 5 ...

1554 Commits (ad166c6b8a3a30d61ddd0dbbfc0b7b45f32f4df6)