proxysql

Commit Graph

Author	SHA1	Message	Date
René Cannaò	75a62f334d	Merge branch 'v4.0' into v3.1-vec Signed-off-by: René Cannaò <rene@proxysql.com>	1 month ago
Rene Cannao	a3f0bade4e	feat: Convert NL2SQL to generic LLM bridge - Rename NL2SQL_Converter to LLM_Bridge for generic prompt processing - Update MySQL protocol handler from /* NL2SQL: / to / LLM: / - Remove SQL-specific fields (sql_query, confidence, tables_used) - Add GENAI_OP_LLM operation type to GenAI module - Rename all genai_nl2sql_ variables to genai_llm_* - Update AI_Features_Manager to use LLM_Bridge - Deprecate ai_nl2sql_convert MCP tool with error message - LLM bridge now handles any prompt type via MySQL protocol This enables generic LLM access (summarization, code generation, translation, analysis) while preserving infrastructure for future NL2SQL implementation via Web UI + external agents.	1 month ago
Rene Cannao	52a70b0b09	feat: Implement AI-based Anomaly Detection for ProxySQL Phase 3: Anomaly Detection Implementation This commit implements a comprehensive multi-stage anomaly detection system for real-time SQL query security analysis. Core Detection Methods: 1. SQL Injection Pattern Detection (lib/Anomaly_Detector.cpp) - Regex-based detection of 11 SQL injection patterns - Suspicious keyword detection (11 patterns) - Covers: tautologies, union-based, comment-based, stacked queries 2. Query Normalization (lib/Anomaly_Detector.cpp:normalize_query) - Converts to lowercase - Removes SQL comments - Replaces string/numeric literals with placeholders - Normalizes whitespace 3. Rate Limiting (lib/Anomaly_Detector.cpp:check_rate_limiting) - Per user/host query rate tracking - Configurable time windows (3600s default) - Auto-block on threshold exceeded - Prevents DoS and brute force attacks 4. Statistical Anomaly Detection (lib/Anomaly_Detector.cpp:check_statistical_anomaly) - Z-score based outlier detection - Abnormal execution time detection (>5s) - Large result set detection (>10000 rows) - Behavioral profiling per user 5. Embedding-based Similarity (lib/Anomaly_Detector.cpp:check_embedding_similarity) - Placeholder for vector similarity search - Framework for sqlite-vec integration - Detects novel attack variations Query Flow Integration: - Added `detect_ai_anomaly()` to MySQL_Session (line 3626) - Integrated after libinjection SQLi detection (line 5150) - Blocks queries when risk threshold exceeded (default: 0.70) - Sends error response with anomaly details Status Variables Added: - `ai_detected_anomalies`: Total anomalies detected - `ai_blocked_queries`: Total queries blocked - Available via: `SELECT * FROM stats_mysql_global` Configuration (defaults): - `enabled`: true - `risk_threshold`: 70 (0-100) - `similarity_threshold`: 85 (0-100) - `rate_limit`: 100 queries/hour - `auto_block`: true - `log_only`: false Detection Pipeline: ``` Query → SQLi Check → AI Anomaly Check → [Block if needed] → Execute (libinjection) (Multi-stage) ``` Files Modified: - include/MySQL_Session.h: Added detect_ai_anomaly() declaration - include/MySQL_Thread.h: Added AI status variables - lib/Anomaly_Detector.cpp: Full implementation (700+ lines) - lib/MySQL_Session.cpp: Integration and query flow - lib/MySQL_Thread.cpp: Status variable definitions Next Steps: - Add unit tests for each detection method - Add integration tests with sample attacks - Add user and developer documentation Related: Phase 1-2 (NL2SQL foundation and testing) Related: Phase 4 (Vector storage for embeddings)	1 month ago
Rene Cannao	bc4fff12ce	feat: Add NL2SQL query interception in MySQL_Session - Add NL2SQL handler declaration - Add routing for 'NL2SQL:' prefix - Return resultset with generated SQL and metadata	1 month ago
Rene Cannao	db2485be37	Add comprehensive doxygen documentation to GenAI async module This commit adds extensive doxygen-format documentation to all key functions in the GenAI async module to improve code maintainability and API clarity. Documented functions: - lib/GenAI_Thread.cpp: - unregister_client() - cleanup flow and usage - call_llama_batch_embedding() - HTTP client with JSON format - call_llama_rerank() - HTTP client with JSON format - execute_sql_for_documents() - stub for document_from_sql - process_json_query() - autonomous JSON query processing - lib/MySQL_Session.cpp: - genai_send_async() - async flow and error handling - handle_genai_response() - response handling flow - genai_cleanup_request() - resource cleanup details - check_genai_events() - main loop integration Enhanced header documentation: - GenAI_RequestHeader - communication flow details - GenAI_ResponseHeader - response format details - register_client() - registration flow - unregister_client() - cleanup flow - embed_documents() - BLOCKING warning - rerank_documents() - BLOCKING warning - process_json_query() - supported formats All documentation includes: - @brief descriptions - @param parameter details - @return return value explanations - @note important warnings and usage notes - @see cross-references to related functions - Detailed workflow descriptions - Error handling details - Memory management notes	1 month ago
Rene Cannao	8405027124	Integrate GenAI async event handling into main MySQL session loop - Add check_genai_events() function for non-blocking epoll_wait on GenAI response fds - Integrate GenAI event checking into main handler() WAITING_CLIENT_DATA case - Add goto handler_again to process multiple GenAI responses in one iteration The async GenAI architecture is now fully integrated. MySQL threads no longer block when processing GENAI: queries - they send requests asynchronously via socketpair and continue processing other queries while GenAI workers handle the embedding/reranking operations.	1 month ago
Rene Cannao	0ff2e38e22	Implement async GenAI module with socketpair-based non-blocking architecture - Add GenAI_RequestHeader and GenAI_ResponseHeader protocol structures for socketpair communication - Implement GenAI listener_loop to read requests from epoll and queue to workers - Implement GenAI worker_loop to process requests and send responses via socketpair - Add GenAI_PendingRequest state management to MySQL_Session/Base_Session - Implement MySQL_Session async handlers: genai_send_async(), handle_genai_response(), genai_cleanup_request() - Modify MySQL_Session genai handler to use async path when epoll is available - Initialize GenAI epoll fd in Base_Session::init() This completes the async architecture that was planned but never fully implemented (previously had only placeholder comments). The GenAI module now processes requests asynchronously without blocking MySQL threads.	1 month ago
Rene Cannao	cc3e97b7b8	Merge EMBED and RERANK into unified GENAI: query syntax This commit refactors the experimental GenAI query syntax to use a single GENAI: keyword with type-based operations instead of separate EMBED: and RERANK: keywords. Changes: - Replace EMBED: and RERANK: detection with unified GENAI: detection - Merge genai_embedding and genai_rerank handlers into single genai handler - Add 'type' field to operation JSON ("embed" or "rerank") - Add 'columns' field for rerank operation (2 or 3, default 3) - columns=2: Returns only index and score - columns=3: Returns index, score, and document (default) Old syntax: EMBED: ["doc1", "doc2"] RERANK: {"query": "...", "documents": [...], "top_n": 5} New syntax: GENAI: {"type": "embed", "documents": ["doc1", "doc2"]} GENAI: {"type": "rerank", "query": "...", "documents": [...], "top_n": 5, "columns": 2} This provides a cleaner, more extensible API for future GenAI operations.	1 month ago
Rene Cannao	39939f598b	Add experimental GenAI RERANK: query support for MySQL This commit adds experimental support for reranking documents directly from MySQL queries using a special RERANK: syntax. Changes: - Add handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___genai_rerank() - Add RERANK: query detection alongside EMBED: detection - Implement JSON parsing for query, documents array, and optional top_n - Build resultset with index, score, and document columns - Use MySQL ERR_Packet for error handling Query format: RERANK: {"query": "search query", "documents": ["doc1", "doc2", ...], "top_n": 5} Result format: 1 row per result, 3 columns (index, score, document)	1 month ago
Rene Cannao	253591d262	Add experimental GenAI EMBED: query support for MySQL This commit adds experimental support for generating embeddings directly from MySQL queries using a special EMBED: syntax. Changes: - Add MYDS_INTERNAL_GENAI to MySQL_DS_type enum for GenAI connections - Add handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___genai_embedding() - Implement EMBED: query detection and JSON parsing for document arrays - Build CSV resultset with embeddings (1 row per document, 1 column) - Add myconn NULL check in MySQL_Thread for INTERNAL_GENAI type - Add "debug_genai" name to debug module array - Remove HAVE_LIBCURL checks (libcurl is always statically linked) - Use static curl header: "curl/curl.h" instead of <curl/curl.h> - Remove curl_global_cleanup() from GenAI module (should only be in main()) Query format: EMBED: ["doc1", "doc2", ...] Result format: 1 row per document, 1 column with CSV embeddings Error handling uses MySQL ERR_Packet instead of resultsets.	1 month ago
Rene Cannao	fc73ec1c50	Code review improvements: Add enum and refactor SELECT VERSION() handling - Add SelectVersionForwardingMode enum to replace magic numbers (0,1,2,3) - Refactor modes 2 and 3 to eliminate code duplication - Improve code readability and maintainability Addresses feedback from gemini-code-assist on PR #5277	1 month ago
Rene Cannao	366164ab26	Fix #5272 : Add mysql-select_version_forwarding variable for SELECT VERSION() Since ProxySQL 3.0.4, SELECT VERSION() queries were intercepted and returned ProxySQL's mysql-server_version variable instead of proxying to backends. This broke SQLAlchemy for MariaDB which expects "MariaDB" in the version string. This commit adds a new variable `mysql-select_version_forwarding` with 4 modes: - 0 = never: Always return ProxySQL's version (3.0.4+ behavior) - 1 = always: Always proxy to backend (3.0.3 behavior) - 2 = smart (fallback to 0): Try backend connection, else ProxySQL version - 3 = smart (fallback to 1): Try backend connection, else proxy (default) The implementation includes: - New global variable mysql_thread___select_version_forwarding - New function get_backend_version_for_hostgroup() to peek at backend connection versions without removing them from the pool - Modified SELECT VERSION() handler to support all 4 modes - ProxySQL backend detection to avoid recursion Mode 3 (default) ensures SQLAlchemy always gets the real MariaDB version string while maintaining fast response when connections are available.	2 months ago
Rene Cannao	44aa606caa	Implement fast forward grace close feature to prevent data loss Problem: In fast forward mode, ProxySQL forwards packets directly from client to backend without buffering them. If the backend connection closes unexpectedly (e.g., due to server crash, network failure, or other issues), ProxySQL immediately closes the client session. This can result in data loss because the client may have sent additional data that hasn't been fully transmitted yet, as ProxySQL does not wait for the output buffers to drain. Solution: Implement a configurable grace period for session closure in fast forward mode. When the backend closes unexpectedly, instead of closing the session immediately, ProxySQL waits for a configurable timeout (fast_forward_grace_close_ms, default 5000ms) to allow any pending client output data to be sent. During this grace period: - If the client output buffers become empty, the session closes gracefully. - If the timeout expires, the session closes anyway to prevent indefinite hanging. Changes: - Added global variable mysql_thread___fast_forward_grace_close_ms (0-3600000ms) - Added session flags: backend_closed_in_fast_forward, fast_forward_grace_start_time - Added data stream flag: defer_close_due_to_fast_forward - Modified MySQL_Data_Stream::read_from_net() to detect backend EOF and initiate grace close if client buffers are not empty - Modified MySQL_Session::handler() FAST_FORWARD case to implement grace close logic with timeout and buffer checks - Added extensive inline documentation explaining the feature and its mechanics This prevents data loss in fast forward scenarios while maintaining bounded session lifetime.	3 months ago
Wazir Ahmed	5219d4be93	processlist: Prevent crash due to SQLITE_TOOBIG - Add new mysql/pgsql variable `processlist_max_query_length`. - Min: 1K - Max: 32M - Default: 2M - Truncate current query based on the configuration before inserting into `stats_*_processlist` tables. - Refactor/fix code related to other processlist configurations. 1. `session_idle_show_processlist` value was not updated in `ProxySQL_Admin.variables`. 2. Pass processlist config as an argument to `MySQL_Threads_Handler::SQL3_Processlist` instead of using thread-local variables. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	4 months ago
René Cannaò	ccf72b606c	Merge pull request #5134 from wazir-ahmed/mysqldump Improve `mysqldump` compatibility	4 months ago
Rahim Kanji	f88b781905	Removed pending_connect	4 months ago
Wazir Ahmed	b6d796c9f2	Improve `mysqldump` compatibility Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	5 months ago
René Cannaò	e1b7afe282	Merge pull request #5099 from wazir-ahmed/clock-monotonic-raw Use `CLOCK_MONOTONIC_RAW` instead of `CLOCK_MONOTONIC`	5 months ago
Rahim Kanji	a7df0d3b3c	Fixed clang compilation	5 months ago
Wazir Ahmed	8d29742e20	`QueryInfo::end_time` should always be greater than or equal to `start_time` Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago
Rahim Kanji	96c989a854	* Replaced virtual RequestEnd method with CRTP (static polymorphism) * Removed multi_pkt from PgSQL_session	6 months ago
Rahim Kanji	a29eaf1eb2	Moved warning_in_hg from Base_Session to MySQL_Session	7 months ago
Rahim Kanji	441504190d	Moved sess_STMTs_meta and SLDH from Base_Session to MySQL_Session	7 months ago
Rahim Kanji	87ad15b01c	* Removed autocommit_on_hostgroup from PostgreSQL modules * Introduced an error generator method to reduce code duplication * Added several comments for clarity * Addressed edge cases related to libpq limitations	7 months ago
Javier Jaramago Fernández	811874bad1	Merge branch 'v3.0' of github.com:sysown/proxysql into v3.0-ff_conns_eof_caps_match	11 months ago
René Cannaò	2551d350e9	Port to 3.0 of in-memory query logging See #4713 for further details	11 months ago
Javier Jaramago Fernández	d49a96dade	Enforce 'CLIENT_DEPRECATE_EOF' capability match for fast-forward - Ensures matching for 'CLIENT_DEPRECATE_EOF' capability when creating or acquiring backend conns for 'fast-forward' sessions. - Since 'CLIENT_DEPRECATE_EOF' capability isn't enforced on backend connections for regular sessions, a mismatch between client and backend capabilities is possible. A verification has been introduced when a session switches to 'fast-forward', in case of capabilities mismatch, an error is logged and client disconnected.	1 year ago
René Cannaò	7ef5993c35	Revert "Enforce 'CLIENT_DEPRECATE_EOF' capability match for conn-pool" This reverts commit `50ce205d21`.	1 year ago
Javier Jaramago Fernández	50ce205d21	Enforce 'CLIENT_DEPRECATE_EOF' capability match for conn-pool Add new config option ('mysql-connpool_match_client_deprecate_eof') that allows to control whether to enforce a match between fronted and backend connections for 'CLIENT_DEPRECATE_EOF' capability. By default, this option is 'true', allowing smooth transitions from regular sessions to 'fast_forward'.	1 year ago
Rahim Kanji	79c7f13768	Clean up	1 year ago
René Cannaò	04abd43a57	Initial commit with PostgreSQL support	1 year ago
Rahim Kanji	c5bc9ed2c6	DNS Cache Fixes * If a MySQL connection fails or times out for a specific hostname, the corresponding DNS record will be removed from DNS Cache. This ensures that outdated DNS records are not retained. (Rest of the DNS resolution process remains unchanged) * DNS cache functionality will automatically be disabled when ProxySQL is run with the -M or --no-monitor option.	1 year ago
René Cannaò	dc2e9fae60	Further code cleanup Initial split of MySQL_Session::handler___status_CONNECTING_CLIENT___STATE_SERVER_HANDSHAKE()	2 years ago
René Cannaò	c2f9df4182	Splitting MySQL_Session::handler()	2 years ago
René Cannaò	557b39912d	Spit of MySQL_Session::get_pkts_from_client() Split of get_pkts_from_client() into smaller functions	2 years ago
René Cannaò	27a1bbb810	Adding MySQL_Session::set_previous_status_mode3() This function helps removing some redundant code	2 years ago
René Cannaò	1bc9d9e5f4	Several code documentation	2 years ago
René Cannaò	e958946b82	Handle STATUS command without backends Handles special queries executed by the STATUS command in mysql cli . Specifically: - "select DATABASE(), USER() limit 1" - "select @@character_set_client, @@character_set_connection, @@character_set_server, @@character_set_database limit 1" Closes #4396 Closes #4426	2 years ago
Javier Jaramago Fernández	e8d2254689	Merge branch 'v2.x' of github.com:sysown/proxysql into v2.x-aurora_autodiscovery-refactor_cluster_mysql_servers-gr_bootstrap_mode	2 years ago
Rahim Kanji	045d6bcaeb	* Execute all test cases with 'CLIENT_DEPRECATE_EOF' both enabled and disabled. * Introduced 'ps_type' enum to differentiate between preparing a statement and executing a statement * Fixed issue where the warning count fails to reset when the SET statement generates a warning. * Resolved the issue where the 'server_status' and 'warning count' were not accurately represented in the statement. * Replaced 'mysql_warning_count' with 'mysql_stmt_warning_count' for retrieving the warning count in statements. * Added 'handle_warning' field in hostgroup_attributes TAP test * Updated warnings TAP test	2 years ago
Rahim Kanji	05208e2839	* Update warning_count as soon as query is executed.	2 years ago
Rahim Kanji	1807315950	Merge branch 'v2.x_hostgroup_attr_settings' into v2.x_warnings_support	2 years ago
Rahim Kanji	6b295477d6	Enhanced Warnings handling in ProxySQL: * Implemented 'SHOW WARNINGS' and 'SHOW COUNT() FROM WARNINGS' query handling. When a query produces warnings, the warning count is now included in the query response. Additionally, a new variable 'mysql-query_cache_with_warnings_support' has been introduced for the query cache feature: * A value of '0' means that if a query execution produces warnings, the resultset will not be saved in the query cache. * A value of '1' indicates that if a query produces warnings, the resultset will be saved in the query cache, but the warning count in that resultset will be set to '0'. ** Note: * Warning handling is dependent on the query digest and 'mysql-query_digests' must be enabled for it to work. * Warnings saved in logfile ['mysql-log_mysql_warnings_enabled'] and the current feature can work in conjunction with each other.	2 years ago
Miro Stauder	4ffb96bf00	remove dead code	3 years ago
Miro Stauder	b1dc534757	refactor out pktH	3 years ago
Javier Jaramago Fernández	bb40e9e431	Fix 'COMMIT\|ROLLBACK' forwarding to backend connections - Closes #4264 These commands are now forwarded to the backend connections from a session with potential transaction statuses, using the following priority order: 1. Connections flagged with 'SERVER_STATUS_IN_TRANS', or 'autocommit=0' when 'autocommit_false_is_transaction' is set. 2. Connections with 'autocommit=0' holding a 'SAVEPOINT'. 3. Connections with 'unknown transaction status', e.g: connections with errors.	3 years ago
Javier Jaramago Fernández	858c430b59	Merge branch 'v2.x' of github.com:sysown/proxysql into v2.x-gr_bootstrap_mode	3 years ago
Rahim Kanji	95b22b96fb	Added SSL connection support in kill_query_thread	3 years ago
Javier Jaramago Fernández	ff2e1a2b85	Fix crash when attempting to log failed 'SET NAMES' Session 'status' and 'client_myds->DSS' should never be modified for a regular session prior to calling 'RequestEnd'. It's 'RequestEnd' responsibility to appropriately change these statuses after the rest of cleanup operations for the request has taken place, including logging.	3 years ago
René Cannaò	c0745b42c2	Adding last_insert_id and last_gtid to MySQL_Logger	3 years ago

1 2 3 4 5

231 Commits (9ffc3f8d711d2f1d3373c91a135d40cbea202841)