752 Commits (e8253cac1fea6fd6a868e121db72c13b7ebaa460)

Author	SHA1	Message	Date
Rene Cannao	caf324f911	logger: harden buffered logging flush/rotation semantics ... This commit addresses three correctness regressions introduced by thread-local log buffering in PR #5364 (query logging performance): 1) stale logfile pointer/UAF race during concurrent rotate/close 2) stale logfile-open state after close 3) non-global flush behavior in admin/format-switch paths What was fixed - Make `flush_and_rotate()` consume the current logfile pointer under lock - Signature changed from `std::fstream*` to `std::fstream*&` - Prevents dereferencing a stale stream pointer captured before lock acquisition while another thread rotates/closes the file - Updated declaration/definition and all call sites - Add explicit synchronization for cross-thread buffer draining - Added `LogBufferThreadContext::buffer_lock` - Any path that appends or flushes a thread buffer now locks this mutex - Guarantees force-flush from admin/config paths cannot race with worker-thread appends on the same context - Restore global forced flush semantics where required - Extended `MySQL_Logger::flush` and `PgSQL_Logger::flush` to `flush(bool force = false)` - `force=false`: preserves existing low-overhead worker-loop behavior (per-thread timeout-based flush) - `force=true`: snapshots all known thread contexts and drains both events/audit buffers regardless of timeout - `flush_log()` now calls `flush(true)` before file rotation, so admin flush and format-switch operations no longer miss pending thread buffers - Avoid unintended rotation during forced draining - In `force=true` path, flush uses `rotate_fn=nullptr` - Drains buffered payload into the current file first - `flush_log()` then performs one controlled rotate/open step - Fix logfile state tracking after close - `events_close_log_unlocked()` and `audit_close_log_unlocked()` now set `logfile_open=false` when the stream is closed - Prevents write paths from treating a closed stream as open - Remove per-thread-context dependency during metadata header write - `events_open_log_unlocked()` now uses a local `LogBuffer` for metadata emission in format=1 instead of reusing a thread context buffer - Keeps open/rotate path independent from worker context lifecycle - Keep callers consistent and non-duplicative - Since `flush_log()` now force-drains internally, removed redundant explicit `flush()` calls from: - MySQL/PgSQL `eventslog_format` switch handlers - `ProxySQL_Admin::flush_logs` Behavioral outcome - No stale stream pointer use when close/rotate interleaves with flush - No false-positive logfile-open state after close - `FLUSH LOGS` and eventslog format switch now drain all known thread buffers before rotating, preventing dropped/misplaced buffered records Validation - Built modified objects directly - Ran full debug build with GENAI enabled: make clean && export PROXYSQLGENAI=1 && make debug -j24 Build completed successfully.	3 months ago
René Cannaò	568fe1524e	Merge pull request #5364 from mevishalr/feature/query-logging-improvements ... Optimize query logging performance (#5243)	3 months ago
Rene Cannao	cc53ddca23	Merge branch 'v3.0-openssl-fix' into v3.0-test0213	3 months ago
vramesha	15a4ed66e9	Optimize query logging performance (#5243) ... Detailed Changes: - Introduced LogBuffer and LogBufferThreadContext to implement per-thread buffering, reducing lock contention. - Replaced localtime() with localtime_r() for improved performance and thread safety during timestamp generation. - Implemented configurable sampling for event logs to reduce I/O overhead. - Added global configuration variables to control flush size, flush interval, and sampling rate for event and audit logs.	3 months ago
Rene Cannao	778e01174c	Add protocol labels to Thread Handler metrics and enable PostgreSQL metrics ... This commit extends PR #5069 by adding protocol labels to distinguish MySQL and PostgreSQL metrics in the Thread Handler modules. Changes: - Add { "protocol", "mysql" } labels to 65 MySQL_Thread.cpp metrics - Add { "protocol", "pgsql" } labels to 60 PgSQL_Thread.cpp metrics - Enable GloPTH (PostgreSQL threads handler) metrics export - Document Query Cache limitation (template base class collision) This resolves Issue #5068 by ensuring PostgreSQL prometheus metrics are properly exported with protocol labels, distinguishing them from MySQL metrics. Related: #5068, #5069	4 months ago
Rene Cannao	4c788dcbb3	fix: signed/unsigned comparison warnings (-Wsign-compare) ... Fix two HIGH priority signed/unsigned comparison warnings: 1. MySQL_HostGroups_Manager.cpp:1792 - Change loop variable from 'int j' to 'unsigned int j' to match the unsigned comparison with servers->len. 2. MySQL_Thread.cpp:4357 - Cast int to size_t when comparing with getBufferSize() which returns size_t. These changes prevent potential logic errors from negative values being interpreted as large unsigned values. Found during compiler warning analysis with 'make debug'.	4 months ago
René Cannaò	75a62f334d	Merge branch 'v4.0' into v3.1-vec ... Signed-off-by: René Cannaò <rene@proxysql.com>	4 months ago
Rene Cannao	52a70b0b09	feat: Implement AI-based Anomaly Detection for ProxySQL ... Phase 3: Anomaly Detection Implementation This commit implements a comprehensive multi-stage anomaly detection system for real-time SQL query security analysis. **Core Detection Methods:** 1. **SQL Injection Pattern Detection** (lib/Anomaly_Detector.cpp) - Regex-based detection of 11 SQL injection patterns - Suspicious keyword detection (11 patterns) - Covers: tautologies, union-based, comment-based, stacked queries 2. **Query Normalization** (lib/Anomaly_Detector.cpp:normalize_query) - Converts to lowercase - Removes SQL comments - Replaces string/numeric literals with placeholders - Normalizes whitespace 3. **Rate Limiting** (lib/Anomaly_Detector.cpp:check_rate_limiting) - Per user/host query rate tracking - Configurable time windows (3600s default) - Auto-block on threshold exceeded - Prevents DoS and brute force attacks 4. **Statistical Anomaly Detection** (lib/Anomaly_Detector.cpp:check_statistical_anomaly) - Z-score based outlier detection - Abnormal execution time detection (>5s) - Large result set detection (>10000 rows) - Behavioral profiling per user 5. **Embedding-based Similarity** (lib/Anomaly_Detector.cpp:check_embedding_similarity) - Placeholder for vector similarity search - Framework for sqlite-vec integration - Detects novel attack variations **Query Flow Integration:** - Added `detect_ai_anomaly()` to MySQL_Session (line 3626) - Integrated after libinjection SQLi detection (line 5150) - Blocks queries when risk threshold exceeded (default: 0.70) - Sends error response with anomaly details **Status Variables Added:** - `ai_detected_anomalies`: Total anomalies detected - `ai_blocked_queries`: Total queries blocked - Available via: `SELECT * FROM stats_mysql_global` **Configuration (defaults):** - `enabled`: true - `risk_threshold`: 70 (0-100) - `similarity_threshold`: 85 (0-100) - `rate_limit`: 100 queries/hour - `auto_block`: true - `log_only`: false **Detection Pipeline:** ``` Query → SQLi Check → AI Anomaly Check → [Block if needed] → Execute (libinjection) (Multi-stage) ``` **Files Modified:** - include/MySQL_Session.h: Added detect_ai_anomaly() declaration - include/MySQL_Thread.h: Added AI status variables - lib/Anomaly_Detector.cpp: Full implementation (700+ lines) - lib/MySQL_Session.cpp: Integration and query flow - lib/MySQL_Thread.cpp: Status variable definitions **Next Steps:** - Add unit tests for each detection method - Add integration tests with sample attacks - Add user and developer documentation Related: Phase 1-2 (NL2SQL foundation and testing) Related: Phase 4 (Vector storage for embeddings)	4 months ago
Rene Cannao	253591d262	Add experimental GenAI EMBED: query support for MySQL ... This commit adds experimental support for generating embeddings directly from MySQL queries using a special EMBED: syntax. Changes: - Add MYDS_INTERNAL_GENAI to MySQL_DS_type enum for GenAI connections - Add handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___genai_embedding() - Implement EMBED: query detection and JSON parsing for document arrays - Build CSV resultset with embeddings (1 row per document, 1 column) - Add myconn NULL check in MySQL_Thread for INTERNAL_GENAI type - Add "debug_genai" name to debug module array - Remove HAVE_LIBCURL checks (libcurl is always statically linked) - Use static curl header: "curl/curl.h" instead of <curl/curl.h> - Remove curl_global_cleanup() from GenAI module (should only be in main()) Query format: EMBED: ["doc1", "doc2", ...] Result format: 1 row per document, 1 column with CSV embeddings Error handling uses MySQL ERR_Packet instead of resultsets.	5 months ago
Rene Cannao	b5598d8d53	Add comprehensive ProxySQL_Poll usage documentation throughout codebase ... Enhance ProxySQL_Poll class documentation with detailed usage patterns: - lib/ProxySQL_Poll.cpp: Enhanced file-level documentation with architecture overview, template specialization, memory management, and event processing pipeline explanations - lib/MySQL_Thread.cpp: Added usage documentation for listener registration, removal patterns, client session setup, and main poll loop - lib/PgSQL_Thread.cpp: Added equivalent PostgreSQL usage documentation mirroring MySQL patterns with protocol-specific details - lib/mysql_data_stream.cpp: Documented cleanup, receive activity tracking, and send activity tracking patterns - lib/PgSQL_Data_Stream.cpp: Documented equivalent PostgreSQL data stream patterns for cleanup and activity tracking All documentation is placed directly where code is used, avoiding specific line numbers for better maintainability. Includes comprehensive explanations of when, why, and how ProxySQL_Poll methods are used throughout ProxySQL's event-driven architecture. [skip-ci]	5 months ago
Rene Cannao	0699c7ba11	Add comprehensive ProxySQL_Poll usage documentation throughout codebase ... Enhance ProxySQL_Poll class documentation with detailed usage patterns: - lib/ProxySQL_Poll.cpp: Enhanced file-level documentation with architecture overview, template specialization, memory management, and event processing pipeline explanations - lib/MySQL_Thread.cpp: Added usage documentation for listener registration, removal patterns, client session setup, and main poll loop - lib/PgSQL_Thread.cpp: Added equivalent PostgreSQL usage documentation mirroring MySQL patterns with protocol-specific details - lib/mysql_data_stream.cpp: Documented cleanup, receive activity tracking, and send activity tracking patterns - lib/PgSQL_Data_Stream.cpp: Documented equivalent PostgreSQL data stream patterns for cleanup and activity tracking All documentation is placed directly where code is used, avoiding specific line numbers for better maintainability. Includes comprehensive explanations of when, why, and how ProxySQL_Poll methods are used throughout ProxySQL's event-driven architecture. [skip-ci]	5 months ago
Rene Cannao	366164ab26	Fix #5272: Add mysql-select_version_forwarding variable for SELECT VERSION() ... Since ProxySQL 3.0.4, SELECT VERSION() queries were intercepted and returned ProxySQL's mysql-server_version variable instead of proxying to backends. This broke SQLAlchemy for MariaDB which expects "MariaDB" in the version string. This commit adds a new variable `mysql-select_version_forwarding` with 4 modes: - 0 = never: Always return ProxySQL's version (3.0.4+ behavior) - 1 = always: Always proxy to backend (3.0.3 behavior) - 2 = smart (fallback to 0): Try backend connection, else ProxySQL version - 3 = smart (fallback to 1): Try backend connection, else proxy (default) The implementation includes: - New global variable mysql_thread___select_version_forwarding - New function get_backend_version_for_hostgroup() to peek at backend connection versions without removing them from the pool - Modified SELECT VERSION() handler to support all 4 modes - ProxySQL backend detection to avoid recursion Mode 3 (default) ensures SQLAlchemy always gets the real MariaDB version string while maintaining fast response when connections are available.	5 months ago
Javier Jaramago Fernández	6fea828e86	Improve logging in unexpected COM_PING packet handling ... Logging messages now include 'client address', 'session status' and 'data stream status'. Client address is also logged when OK packets are dispatched, this should help tracking if a client has received the expected packets or not.	5 months ago
Javier Jaramago Fernández	d0e88599ee	Add special handling for unexpected COM_PING packets ... Implements a workaround for the handling of unexpected 'COM_PING' packets received during query processing, while a resultset is yet being streamed to the client. Received 'COM_PING' packets are queued in the form of a counter. This counter is later used to sent the corresponding number of 'OK' packets to the client after 'MySQL_Session' has finished processing the current query.	5 months ago
Rene Cannao	5a7b22181f	Fix metrics collection for wait_timeout counters ... The get_status_variable() function was only scanning worker threads but ignoring auxiliary threads (idle threads) where timeout terminations are detected. This caused the timeout termination counter to show incorrect/zero values. - Added idle thread scanning to both overloaded versions of get_status_variable() function - Now properly collects metrics from both worker and idle threads - Fixes the issue where proxysql_mysql_timeout_terminated_connections_total showed zero despite actual timeout terminations Resolves the metrics reading issue identified in the previous commits.	6 months ago
Rene Cannao	dc4694d656	Refactor idle session scanning and improve test precision ... Code improvements: - Extract SESS_TO_SCAN_idle_thread constant to header file for better maintainability - Replace magic number 128 with named constant in idle_thread_to_kill_idle_sessions() - Improve code readability and consistency in session scanning logic Test enhancements: - Add mysql-poll_timeout configuration for more precise timeout testing - Reduce test sleep times to 13 seconds for faster test execution - Add diagnostic messages to clearly show timeout configurations in test output - Ensure tests properly validate timeout enforcement with precise timing The changes improve code maintainability and make tests more reliable and faster while maintaining accurate timeout validation.	6 months ago
Rene Cannao	0c5e75a064	Fix wait_timeout timeout calculations and add proper newline characters ... Key improvements: - Fix timeout comparison in MySQL_Thread::idle_thread_to_kill_idle_sessions() to prevent underflow - Use effective wait_timeout (minimum of global and session values) for idle timeout calculations - Add proper newline characters to proxy_warning messages for consistent log formatting - Increase test sleep times to account for global timeout enforcement - Fix session timeout test durations to properly test timeout behavior Technical changes: - Replace broken min_idle calculation with proper effective wait_timeout logic - Add std::min() usage to determine effective timeout from global and session values - Ensure warning messages end with newline characters for proper log formatting - Update test sleep durations to ensure proper timeout testing Resolves potential timeout calculation bugs and ensures consistent timeout enforcement behavior.	6 months ago
René Cannaò	588b0a0ae6	Merge branch 'v3.0' into v3.0_wait_timeout ... Signed-off-by: René Cannaò <rene@proxysql.com>	6 months ago
Rene Cannao	ae93966603	Add TAP test for fast forward grace close feature ... - Rename and modify test to use MySQL C API mysql_binlog_* functions - Implement throttled binlog reading with 5 iterations (no limit, 2s, 5s, 20s, 60s targets) - Add diagnostics for debugging binlog fetch issues - Set RPL options for file, position, server_id, and non-blocking flag - Update Makefile to compile with MySQL client library	7 months ago
Rene Cannao	44aa606caa	Implement fast forward grace close feature to prevent data loss ... Problem: In fast forward mode, ProxySQL forwards packets directly from client to backend without buffering them. If the backend connection closes unexpectedly (e.g., due to server crash, network failure, or other issues), ProxySQL immediately closes the client session. This can result in data loss because the client may have sent additional data that hasn't been fully transmitted yet, as ProxySQL does not wait for the output buffers to drain. Solution: Implement a configurable grace period for session closure in fast forward mode. When the backend closes unexpectedly, instead of closing the session immediately, ProxySQL waits for a configurable timeout (fast_forward_grace_close_ms, default 5000ms) to allow any pending client output data to be sent. During this grace period: - If the client output buffers become empty, the session closes gracefully. - If the timeout expires, the session closes anyway to prevent indefinite hanging. Changes: - Added global variable mysql_thread___fast_forward_grace_close_ms (0-3600000ms) - Added session flags: backend_closed_in_fast_forward, fast_forward_grace_start_time - Added data stream flag: defer_close_due_to_fast_forward - Modified MySQL_Data_Stream::read_from_net() to detect backend EOF and initiate grace close if client buffers are not empty - Modified MySQL_Session::handler() FAST_FORWARD case to implement grace close logic with timeout and buffer checks - Added extensive inline documentation explaining the feature and its mechanics This prevents data loss in fast forward scenarios while maintaining bounded session lifetime.	7 months ago
René Cannaò	dfd46fee4b	Merge pull request #5150 from proton-lisandro-pin/ignore_min_gtid ... Implement a `ignore_min_gtid_annotation` global variable, fix memory leak/overflow in `min_gtid` annotation handler.	7 months ago
René Cannaò	8be91071ef	Merge pull request #5190 from sysown/v3.0_default_interface_global_var_5189 ... Ensure default mysql-interfaces=0.0.0.0:6033 and pgsql-interfaces=0.0.0.0:6133 are set	7 months ago
Lisandro Pin	6cb522283d	Implement a ignore_min_gtid_annotation global variable for ProxySQL ... When true, all `min_gtid` query annotations are ignored; see https://proxysql.com/documentation/query-annotations/ for details. This is useful on ProxySQL setups with multiple layers, where some layers mandate GTID-based routing while others don't.	7 months ago
Rahim Kanji	48928ce04e	Ensure default mysql-interfaces=0.0.0.0:6033 and pgsql-interfaces=0.0.0.0:6133 are set	7 months ago
Rahim Kanji	86c17f04d7	Replace single sleep with looped delay for simulated watchdog test	7 months ago
Wazir Ahmed	5219d4be93	processlist: Prevent crash due to SQLITE_TOOBIG ... - Add new mysql/pgsql variable `processlist_max_query_length`. - Min: 1K - Max: 32M - Default: 2M - Truncate current query based on the configuration before inserting into `stats_*_processlist` tables. - Refactor/fix code related to other processlist configurations. 1. `session_idle_show_processlist` value was not updated in `ProxySQL_Admin.variables`. 2. Pass processlist config as an argument to `MySQL_Threads_Handler::SQL3_Processlist` instead of using thread-local variables. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	7 months ago
Wazir Ahmed	97eb19c0f0	fix: Close idle session based on effective timeout ... - effective_timeout = min (mysql-wait_timeout, session-wait_timeout) Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	10 months ago
Rahim Kanji	344ba25777	Add Watchdog PROXYSQLTEST covering MySQL and PgSQL threads	10 months ago
René Cannaò	2c26a42897	Merge pull request #4819 from sysown/v3.0-ff_conns_eof_caps_match ... Enforce `CLIENT_DEPRECATE_EOF` capability match for `fast-forward` sessions	1 year ago
René Cannaò	9737e0630a	Add variable mysql-eventslog_stmt_parameters #4923 ... Variable mysql-eventslog_stmt_parameters configures the logging of parameters in prepared statements: - 0 : logging disabled - 1 : logging enabled	1 year ago
Javier Jaramago Fernández	811874bad1	Merge branch 'v3.0' of github.com:sysown/proxysql into v3.0-ff_conns_eof_caps_match	1 year ago
Yashwant Sahu	d7ec497afd	Now taking effective timeout taking as minimum of global timeout and session timeout	1 year ago
Yashwant Sahu	abe16e6692	Parsing and setting wait timeout for session	1 year ago
René Cannaò	d3f06c537f	Merge branch 'v3.0' into v3.0-ff_conns_eof_caps_match	1 year ago
René Cannaò	2551d350e9	Port to 3.0 of in-memory query logging ... See #4713 for further details	1 year ago
Rahim Kanji	f9232ae50d	Merge remote-tracking branch 'Master/v3.0' into v3.0_session_parameter_tracking	1 year ago
Javier Jaramago Fernández	d49a96dade	Enforce 'CLIENT_DEPRECATE_EOF' capability match for fast-forward ... - Ensures matching for 'CLIENT_DEPRECATE_EOF' capability when creating or acquiring backend conns for 'fast-forward' sessions. - Since 'CLIENT_DEPRECATE_EOF' capability isn't enforced on backend connections for regular sessions, a mismatch between client and backend capabilities is possible. A verification has been introduced when a session switches to 'fast-forward', in case of capabilities mismatch, an error is logged and client disconnected.	1 year ago
Rahim Kanji	35b2dd593d	Separate implementation of SET statement parser for MySQL and PostgreSQL	1 year ago
René Cannaò	7ef5993c35	Revert "Enforce 'CLIENT_DEPRECATE_EOF' capability match for conn-pool" ... This reverts commit 50ce205d21.	1 year ago
René Cannaò	b53b5b24cc	Merge pull request #4810 from sysown/v3.0-SetThreadName_optional ... Make set_thread_name() optional	1 year ago
René Cannaò	c2f1871adf	Make set_thread_name() optional	1 year ago
Javier Jaramago Fernández	50ce205d21	Enforce 'CLIENT_DEPRECATE_EOF' capability match for conn-pool ... Add new config option ('mysql-connpool_match_client_deprecate_eof') that allows to control whether to enforce a match between fronted and backend connections for 'CLIENT_DEPRECATE_EOF' capability. By default, this option is 'true', allowing smooth transitions from regular sessions to 'fast_forward'.	1 year ago
Rahim Kanji	b4e9060f71	Modified Set Parser	1 year ago
Yashwant Sahu	2d580746c5	changed protocol_compression_level default to (-1 to 9 )	1 year ago
Yashwant Sahu	2f3b2dacae	Merge branch 'v3.0' into v3.0_compression	1 year ago
Rahim Kanji	794ec0f48d	Merge remote-tracking branch 'Master/v3.0' into v3.0_dynamic_fast_forward	1 year ago
Yashwant Sahu	a0db6c3a1f	1. Changed name to protocol_compression_level. ... 2. Added TAP test	2 years ago
Yashwant Sahu	a846447a22	Added MySQL compression level	2 years ago
Rahim Kanji	ea059eadbf	Added SESSION_FORWARD_TYPE enum to define the type of session forwarding	2 years ago
Javier Jaramago Fernández	74c74adf5e	Prevent busy-waiting in active wait during 'listener_del' ... Since the operation of stopping each worker thread listeners is performed during 'maintenance_loops', the active wait taking place in 'listener_del' is likely to take some time. A sensible delay has been added to reduce unneccesary load.	2 years ago