proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	52a70b0b09	feat: Implement AI-based Anomaly Detection for ProxySQL Phase 3: Anomaly Detection Implementation This commit implements a comprehensive multi-stage anomaly detection system for real-time SQL query security analysis. Core Detection Methods: 1. SQL Injection Pattern Detection (lib/Anomaly_Detector.cpp) - Regex-based detection of 11 SQL injection patterns - Suspicious keyword detection (11 patterns) - Covers: tautologies, union-based, comment-based, stacked queries 2. Query Normalization (lib/Anomaly_Detector.cpp:normalize_query) - Converts to lowercase - Removes SQL comments - Replaces string/numeric literals with placeholders - Normalizes whitespace 3. Rate Limiting (lib/Anomaly_Detector.cpp:check_rate_limiting) - Per user/host query rate tracking - Configurable time windows (3600s default) - Auto-block on threshold exceeded - Prevents DoS and brute force attacks 4. Statistical Anomaly Detection (lib/Anomaly_Detector.cpp:check_statistical_anomaly) - Z-score based outlier detection - Abnormal execution time detection (>5s) - Large result set detection (>10000 rows) - Behavioral profiling per user 5. Embedding-based Similarity (lib/Anomaly_Detector.cpp:check_embedding_similarity) - Placeholder for vector similarity search - Framework for sqlite-vec integration - Detects novel attack variations Query Flow Integration: - Added `detect_ai_anomaly()` to MySQL_Session (line 3626) - Integrated after libinjection SQLi detection (line 5150) - Blocks queries when risk threshold exceeded (default: 0.70) - Sends error response with anomaly details Status Variables Added: - `ai_detected_anomalies`: Total anomalies detected - `ai_blocked_queries`: Total queries blocked - Available via: `SELECT * FROM stats_mysql_global` Configuration (defaults): - `enabled`: true - `risk_threshold`: 70 (0-100) - `similarity_threshold`: 85 (0-100) - `rate_limit`: 100 queries/hour - `auto_block`: true - `log_only`: false Detection Pipeline: ``` Query → SQLi Check → AI Anomaly Check → [Block if needed] → Execute (libinjection) (Multi-stage) ``` Files Modified: - include/MySQL_Session.h: Added detect_ai_anomaly() declaration - include/MySQL_Thread.h: Added AI status variables - lib/Anomaly_Detector.cpp: Full implementation (700+ lines) - lib/MySQL_Session.cpp: Integration and query flow - lib/MySQL_Thread.cpp: Status variable definitions Next Steps: - Add unit tests for each detection method - Add integration tests with sample attacks - Add user and developer documentation Related: Phase 1-2 (NL2SQL foundation and testing) Related: Phase 4 (Vector storage for embeddings)	3 months ago
Rene Cannao	253591d262	Add experimental GenAI EMBED: query support for MySQL This commit adds experimental support for generating embeddings directly from MySQL queries using a special EMBED: syntax. Changes: - Add MYDS_INTERNAL_GENAI to MySQL_DS_type enum for GenAI connections - Add handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___genai_embedding() - Implement EMBED: query detection and JSON parsing for document arrays - Build CSV resultset with embeddings (1 row per document, 1 column) - Add myconn NULL check in MySQL_Thread for INTERNAL_GENAI type - Add "debug_genai" name to debug module array - Remove HAVE_LIBCURL checks (libcurl is always statically linked) - Use static curl header: "curl/curl.h" instead of <curl/curl.h> - Remove curl_global_cleanup() from GenAI module (should only be in main()) Query format: EMBED: ["doc1", "doc2", ...] Result format: 1 row per document, 1 column with CSV embeddings Error handling uses MySQL ERR_Packet instead of resultsets.	3 months ago
Rene Cannao	b5598d8d53	Add comprehensive ProxySQL_Poll usage documentation throughout codebase Enhance ProxySQL_Poll class documentation with detailed usage patterns: - lib/ProxySQL_Poll.cpp: Enhanced file-level documentation with architecture overview, template specialization, memory management, and event processing pipeline explanations - lib/MySQL_Thread.cpp: Added usage documentation for listener registration, removal patterns, client session setup, and main poll loop - lib/PgSQL_Thread.cpp: Added equivalent PostgreSQL usage documentation mirroring MySQL patterns with protocol-specific details - lib/mysql_data_stream.cpp: Documented cleanup, receive activity tracking, and send activity tracking patterns - lib/PgSQL_Data_Stream.cpp: Documented equivalent PostgreSQL data stream patterns for cleanup and activity tracking All documentation is placed directly where code is used, avoiding specific line numbers for better maintainability. Includes comprehensive explanations of when, why, and how ProxySQL_Poll methods are used throughout ProxySQL's event-driven architecture. [skip-ci]	3 months ago
Javier Jaramago Fernández	6fea828e86	Improve logging in unexpected COM_PING packet handling Logging messages now include 'client address', 'session status' and 'data stream status'. Client address is also logged when OK packets are dispatched, this should help tracking if a client has received the expected packets or not.	4 months ago
Javier Jaramago Fernández	d0e88599ee	Add special handling for unexpected COM_PING packets Implements a workaround for the handling of unexpected 'COM_PING' packets received during query processing, while a resultset is yet being streamed to the client. Received 'COM_PING' packets are queued in the form of a counter. This counter is later used to sent the corresponding number of 'OK' packets to the client after 'MySQL_Session' has finished processing the current query.	4 months ago
Rene Cannao	5a7b22181f	Fix metrics collection for wait_timeout counters The get_status_variable() function was only scanning worker threads but ignoring auxiliary threads (idle threads) where timeout terminations are detected. This caused the timeout termination counter to show incorrect/zero values. - Added idle thread scanning to both overloaded versions of get_status_variable() function - Now properly collects metrics from both worker and idle threads - Fixes the issue where proxysql_mysql_timeout_terminated_connections_total showed zero despite actual timeout terminations Resolves the metrics reading issue identified in the previous commits.	4 months ago
Rene Cannao	dc4694d656	Refactor idle session scanning and improve test precision Code improvements: - Extract SESS_TO_SCAN_idle_thread constant to header file for better maintainability - Replace magic number 128 with named constant in idle_thread_to_kill_idle_sessions() - Improve code readability and consistency in session scanning logic Test enhancements: - Add mysql-poll_timeout configuration for more precise timeout testing - Reduce test sleep times to 13 seconds for faster test execution - Add diagnostic messages to clearly show timeout configurations in test output - Ensure tests properly validate timeout enforcement with precise timing The changes improve code maintainability and make tests more reliable and faster while maintaining accurate timeout validation.	4 months ago
Rene Cannao	0c5e75a064	Fix wait_timeout timeout calculations and add proper newline characters Key improvements: - Fix timeout comparison in MySQL_Thread::idle_thread_to_kill_idle_sessions() to prevent underflow - Use effective wait_timeout (minimum of global and session values) for idle timeout calculations - Add proper newline characters to proxy_warning messages for consistent log formatting - Increase test sleep times to account for global timeout enforcement - Fix session timeout test durations to properly test timeout behavior Technical changes: - Replace broken min_idle calculation with proper effective wait_timeout logic - Add std::min() usage to determine effective timeout from global and session values - Ensure warning messages end with newline characters for proper log formatting - Update test sleep durations to ensure proper timeout testing Resolves potential timeout calculation bugs and ensures consistent timeout enforcement behavior.	4 months ago
René Cannaò	588b0a0ae6	Merge branch 'v3.0' into v3.0_wait_timeout Signed-off-by: René Cannaò <rene@proxysql.com>	5 months ago
Rene Cannao	ae93966603	Add TAP test for fast forward grace close feature - Rename and modify test to use MySQL C API mysql_binlog_* functions - Implement throttled binlog reading with 5 iterations (no limit, 2s, 5s, 20s, 60s targets) - Add diagnostics for debugging binlog fetch issues - Set RPL options for file, position, server_id, and non-blocking flag - Update Makefile to compile with MySQL client library	5 months ago
Rene Cannao	44aa606caa	Implement fast forward grace close feature to prevent data loss Problem: In fast forward mode, ProxySQL forwards packets directly from client to backend without buffering them. If the backend connection closes unexpectedly (e.g., due to server crash, network failure, or other issues), ProxySQL immediately closes the client session. This can result in data loss because the client may have sent additional data that hasn't been fully transmitted yet, as ProxySQL does not wait for the output buffers to drain. Solution: Implement a configurable grace period for session closure in fast forward mode. When the backend closes unexpectedly, instead of closing the session immediately, ProxySQL waits for a configurable timeout (fast_forward_grace_close_ms, default 5000ms) to allow any pending client output data to be sent. During this grace period: - If the client output buffers become empty, the session closes gracefully. - If the timeout expires, the session closes anyway to prevent indefinite hanging. Changes: - Added global variable mysql_thread___fast_forward_grace_close_ms (0-3600000ms) - Added session flags: backend_closed_in_fast_forward, fast_forward_grace_start_time - Added data stream flag: defer_close_due_to_fast_forward - Modified MySQL_Data_Stream::read_from_net() to detect backend EOF and initiate grace close if client buffers are not empty - Modified MySQL_Session::handler() FAST_FORWARD case to implement grace close logic with timeout and buffer checks - Added extensive inline documentation explaining the feature and its mechanics This prevents data loss in fast forward scenarios while maintaining bounded session lifetime.	5 months ago
René Cannaò	dfd46fee4b	Merge pull request #5150 from proton-lisandro-pin/ignore_min_gtid Implement a `ignore_min_gtid_annotation` global variable, fix memory leak/overflow in `min_gtid` annotation handler.	5 months ago
René Cannaò	8be91071ef	Merge pull request #5190 from sysown/v3.0_default_interface_global_var_5189 Ensure default mysql-interfaces=0.0.0.0:6033 and pgsql-interfaces=0.0.0.0:6133 are set	5 months ago
Lisandro Pin	6cb522283d	Implement a ignore_min_gtid_annotation global variable for ProxySQL When true, all `min_gtid` query annotations are ignored; see https://proxysql.com/documentation/query-annotations/ for details. This is useful on ProxySQL setups with multiple layers, where some layers mandate GTID-based routing while others don't.	5 months ago
Rahim Kanji	48928ce04e	Ensure default mysql-interfaces=0.0.0.0:6033 and pgsql-interfaces=0.0.0.0:6133 are set	5 months ago
Rahim Kanji	86c17f04d7	Replace single sleep with looped delay for simulated watchdog test	5 months ago
Wazir Ahmed	5219d4be93	processlist: Prevent crash due to SQLITE_TOOBIG - Add new mysql/pgsql variable `processlist_max_query_length`. - Min: 1K - Max: 32M - Default: 2M - Truncate current query based on the configuration before inserting into `stats_*_processlist` tables. - Refactor/fix code related to other processlist configurations. 1. `session_idle_show_processlist` value was not updated in `ProxySQL_Admin.variables`. 2. Pass processlist config as an argument to `MySQL_Threads_Handler::SQL3_Processlist` instead of using thread-local variables. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago
Wazir Ahmed	97eb19c0f0	fix: Close idle session based on effective timeout - effective_timeout = min (mysql-wait_timeout, session-wait_timeout) Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	9 months ago
Rahim Kanji	344ba25777	Add Watchdog PROXYSQLTEST covering MySQL and PgSQL threads	9 months ago
René Cannaò	2c26a42897	Merge pull request #4819 from sysown/v3.0-ff_conns_eof_caps_match Enforce `CLIENT_DEPRECATE_EOF` capability match for `fast-forward` sessions	11 months ago
René Cannaò	9737e0630a	Add variable mysql-eventslog_stmt_parameters #4923 Variable mysql-eventslog_stmt_parameters configures the logging of parameters in prepared statements: - 0 : logging disabled - 1 : logging enabled	12 months ago
Javier Jaramago Fernández	811874bad1	Merge branch 'v3.0' of github.com:sysown/proxysql into v3.0-ff_conns_eof_caps_match	1 year ago
Yashwant Sahu	d7ec497afd	Now taking effective timeout taking as minimum of global timeout and session timeout	1 year ago
Yashwant Sahu	abe16e6692	Parsing and setting wait timeout for session	1 year ago
René Cannaò	d3f06c537f	Merge branch 'v3.0' into v3.0-ff_conns_eof_caps_match	1 year ago
René Cannaò	2551d350e9	Port to 3.0 of in-memory query logging See #4713 for further details	1 year ago
Rahim Kanji	f9232ae50d	Merge remote-tracking branch 'Master/v3.0' into v3.0_session_parameter_tracking	1 year ago
Javier Jaramago Fernández	d49a96dade	Enforce 'CLIENT_DEPRECATE_EOF' capability match for fast-forward - Ensures matching for 'CLIENT_DEPRECATE_EOF' capability when creating or acquiring backend conns for 'fast-forward' sessions. - Since 'CLIENT_DEPRECATE_EOF' capability isn't enforced on backend connections for regular sessions, a mismatch between client and backend capabilities is possible. A verification has been introduced when a session switches to 'fast-forward', in case of capabilities mismatch, an error is logged and client disconnected.	1 year ago
Rahim Kanji	35b2dd593d	Separate implementation of SET statement parser for MySQL and PostgreSQL	1 year ago
René Cannaò	7ef5993c35	Revert "Enforce 'CLIENT_DEPRECATE_EOF' capability match for conn-pool" This reverts commit `50ce205d21`.	1 year ago
René Cannaò	b53b5b24cc	Merge pull request #4810 from sysown/v3.0-SetThreadName_optional Make set_thread_name() optional	1 year ago
René Cannaò	c2f1871adf	Make set_thread_name() optional	1 year ago
Javier Jaramago Fernández	50ce205d21	Enforce 'CLIENT_DEPRECATE_EOF' capability match for conn-pool Add new config option ('mysql-connpool_match_client_deprecate_eof') that allows to control whether to enforce a match between fronted and backend connections for 'CLIENT_DEPRECATE_EOF' capability. By default, this option is 'true', allowing smooth transitions from regular sessions to 'fast_forward'.	1 year ago
Rahim Kanji	b4e9060f71	Modified Set Parser	1 year ago
Yashwant Sahu	2d580746c5	changed protocol_compression_level default to (-1 to 9 )	1 year ago
Yashwant Sahu	2f3b2dacae	Merge branch 'v3.0' into v3.0_compression	1 year ago
Rahim Kanji	794ec0f48d	Merge remote-tracking branch 'Master/v3.0' into v3.0_dynamic_fast_forward	1 year ago
Yashwant Sahu	a0db6c3a1f	1. Changed name to protocol_compression_level. 2. Added TAP test	1 year ago
Yashwant Sahu	a846447a22	Added MySQL compression level	1 year ago
Rahim Kanji	ea059eadbf	Added SESSION_FORWARD_TYPE enum to define the type of session forwarding	1 year ago
Javier Jaramago Fernández	74c74adf5e	Prevent busy-waiting in active wait during 'listener_del' Since the operation of stopping each worker thread listeners is performed during 'maintenance_loops', the active wait taking place in 'listener_del' is likely to take some time. A sensible delay has been added to reduce unneccesary load.	1 year ago
Javier Jaramago Fernández	8bc60ed283	Fix worker threads stalling race condition on RESUME command During a RESUME operation, if a 'MySQL_Thread' is bootstrapping listeners (in `MySQL_Thread::run_BootstrapListener`) and detect that `MySQL_Threads_Handler::bootstrapping_listeners` is 'false', the thread prematurely shuts down its own bootstrapping flag from 'mypolls' (`ProxySQL_Poll::bootstrapping_listeners`). Since this thread wont ever bootstrap its corresponding listening sockets, the other working threads will be stalled waiting on it, eventually triggering the watchdog and crashing the instance. Since the bootstrapping operation is sequential, it's expected that all the threads but the one starting their listening sockets are in an active wait. A sensible delay has been introduced to reduce the overhead of such wait.	1 year ago
René Cannaò	d4bd93672e	Merge pull request #79 from sysown/v2.x_refactore_query_processor Refactored Query Processor [WIP]	2 years ago
Rahim Kanji	e723082a1b	Refactored Query Processor Architecture - Introduced separate query processors for MySQL and PostgreSQL. - Removed unnecessary GTID logic from `PgSQL_Session`. - Added a command counter for PostgreSQL queries. - Refactored `Query_Processor` to serve as a base class containing core functionalities applicable to both MySQL and PostgreSQL. Changes: - Removed GTID-related code from `PgSQL_Logger` and `PgSQL_Session`: - bool with_gtid; - char gtid_buf[128]; - int gtid_hid; - Added new PostgreSQL-specific tables: - `pgsql_query_rules`, `pgsql_firewall_whitelist_rules`, `pgsql_query_rules_fast_routing` - `runtime_pgsql_query_rules`, `runtime_pgsql_firewall_whitelist_rules`, `runtime_pgsql_query_rules_fast_routing` - Added PostgreSQL-related variables: pgsql_thread___commands_stats pgsql_thread___query_digests pgsql_thread___query_digests_lowercase pgsql_thread___query_digests_no_digits pgsql_thread___query_digests_replace_null pgsql_thread___query_digests_normalize_digest_text pgsql_thread___query_digests_track_hostname pgsql_thread___query_digests_keep_comment pgsql_thread___query_digests_max_digest_length pgsql_thread___query_digests_max_query_length pgsql_thread___query_digests_grouping_limit pgsql_thread___query_digests_groups_grouping_limit pgsql_thread___enable_load_data_local_infile pgsql_thread___auditlog_filename pgsql_thread___auditlog_filesize pgsql_thread___eventslog_filename pgsql_thread___eventslog_filesize pgsql_thread___eventslog_default_log pgsql_thread___eventslog_format pgsql_thread___firewall_whitelist_errormsg pgsql_thread___firewall_whitelist_enabled pgsql_thread___query_processor_iterations pgsql_thread___query_processor_regex	2 years ago
René Cannaò	cbf4628ea3	Merge remote-tracking branch 'proxysql27/v2.7' into v2.x_postgres_merge27_20240917	2 years ago
René Cannaò	38350292e2	Merge pull request #4600 from sysown/v2.x_proxy Implementation of PROXY protocol V1	2 years ago
René Cannaò	cf9d5fe21a	Merge pull request #4574 from sysown/set-thread-names Set thread names	2 years ago
Javier Jaramago Fernández	bbda0a86eb	Add initial impl for dual-password support	2 years ago
René Cannaò	94b0da1768	In SQL3_Processlist() use saved IP/port	2 years ago
Rene Cannao	7fe013fac2	Merge branch 'proxysql_v2.x_240813' into v2.x_postgres_merge_v2.x_240813 This merges ProxySQL v2.x up to 2024-08-13	2 years ago

1 2 3 4 5 ...

743 Commits (803115f5045f1ca03bbd8b65d313bdffde6f1c21)