proxysql

Commit Graph

Author	SHA1	Message	Date
Rene Cannao	3458cbd28b	fix: address zstd review feedback	3 weeks ago
Rene Cannao	c2f678a9de	feat: implement release tiers and feature guards for v3.0, v3.1, and v4.0 This commit introduces a structured release tier system controlled by environment variables, allowing three distinct ProxySQL versions to be built from the same codebase while maintaining a clear upgrade path. Key Changes: - Tier 1 (v3.0.x Stable): Default build including bug fixes and core enhancements. - Tier 2 (v3.1.x Innovative): Enabled via PROXYSQL31=1. Adds FFTO and TSDB. - Tier 3 (v4.0.x AI/MCP): Enabled via PROXYSQLGENAI=1. Adds the AI stack. Implementation Details: - Makefile: Implemented dynamic versioning logic that increments the minor version for v3.1 and the major version for v4.0. Added tier documentation. - Feature Guards: Introduced PROXYSQLFFTO and PROXYSQLTSDB guards across ProxySQL_Admin, ProxySQL_Statistics, REST API, and HTTP server. - FFTO: Guarded Fast Forward Traffic Observer in MySQL and PgSQL sessions, including thread-local variables and configuration parameters. - TSDB: Guarded the entire Time Series Database subsystem, including SQLite table definitions, background loops, and REST API endpoints. - Build Fixes: * Fixed 'proxy_debug_func' linkage by adding extern "C" and moving the definition outside of DEBUG guards to ensure non-debug builds link correctly. * Corrected misplaced/nested DEBUG and PROXYSQLFFTO guards. * Verified build stability for all tiers (unset, PROXYSQL31=1, PROXYSQLGENAI=1).	1 month ago
Rene Cannao	acef17a8c1	Merge branch 'v3.0-ff_inspect' into v3.0-unified-large-pr # Conflicts: # test/tap/tests/Makefile	2 months ago
Rene Cannao	e2ebc0c449	Merge branch 'v4.0-mcp-stats2' into v3.0-unified-large-pr	2 months ago
Rene Cannao	1c25787b88	Introduce {mysql,pgsql}-query_processor_first_comment_parsing variable (#5384 ) This universal fix allows controlling when the first comment of a query is processed relative to the query rules. By setting this variable to 1, ProxySQL-specific annotations (like GTID) can be parsed immediately, allowing subsequent query rules to strip them from the query string. This prevents prepared statement cache bloat and improves backend statement reuse when annotations with unique literals are used.	2 months ago
Rene Cannao	1258a1f2f2	Address PR reviews: improve performance, safety, and robustness of FFTO	2 months ago
Rene Cannao	e762f91cda	MCP stats: add in-memory processlist filtering and PgSQL concurrency stress TAP Replace direct stats-schema reads in MCP show_processlist with in-memory processlist access via SQL3_Processlist query options. Add processlist query option types in proxysql_admin interfaces and implement filtering, sorting, and pagination post-processing in both MySQL and PgSQL thread implementations. Introduce configurable MCP cap mcp_stats_show_processlist_max_rows (default 200, max 1000), wire it through MCP variable loading, and expose it in default config. Expand Stats_Tool_Handler processlist handling to parse filter/sort arguments, enforce cap metadata, and return consistent MCP payload fields from in-memory snapshots. Add extensive doxygen documentation in the touched headers and source blocks to describe API contracts, filtering behavior, and runtime constraints. Add new TAP test mcp_pgsql_concurrency_stress-t that generates sustained concurrent PgSQL traffic (simple reads, read/write table workload, randomized pg_sleep) while polling MCP show_processlist and show_queries in parallel with filter and ordering assertions. Validation performed locally: mcp_pgsql_concurrency_stress-t (31 assertions) and mcp_show_queries_topk-t (12 assertions).	2 months ago
Rene Cannao	3a3be5a559	Implement Fast Forward Traffic Observer (FFTO) for MySQL and PostgreSQL - Added TrafficObserver base interface - Implemented MySQLFFTO and PgSQLFFTO classes for protocol-aware observation - Integrated FFTO hooks into MySQL and PostgreSQL session handlers in FAST_FORWARD state - Added configuration variables: mysql-ffto_enabled, mysql-ffto_max_buffer_size, pgsql-ffto_enabled, pgsql-ffto_max_buffer_size - Implemented query digest extraction and reporting parity with internal stats - Added session-level bypass when packet size exceeds buffer threshold - Included architectural documentation in doc/ffto_design.md	2 months ago
vramesha	15a4ed66e9	Optimize query logging performance (#5243 ) Detailed Changes: - Introduced LogBuffer and LogBufferThreadContext to implement per-thread buffering, reducing lock contention. - Replaced localtime() with localtime_r() for improved performance and thread safety during timestamp generation. - Implemented configurable sampling for event logs to reduce I/O overhead. - Added global configuration variables to control flush size, flush interval, and sampling rate for event and audit logs.	2 months ago
René Cannaò	75a62f334d	Merge branch 'v4.0' into v3.1-vec Signed-off-by: René Cannaò <rene@proxysql.com>	3 months ago
Rene Cannao	52a70b0b09	feat: Implement AI-based Anomaly Detection for ProxySQL Phase 3: Anomaly Detection Implementation This commit implements a comprehensive multi-stage anomaly detection system for real-time SQL query security analysis. Core Detection Methods: 1. SQL Injection Pattern Detection (lib/Anomaly_Detector.cpp) - Regex-based detection of 11 SQL injection patterns - Suspicious keyword detection (11 patterns) - Covers: tautologies, union-based, comment-based, stacked queries 2. Query Normalization (lib/Anomaly_Detector.cpp:normalize_query) - Converts to lowercase - Removes SQL comments - Replaces string/numeric literals with placeholders - Normalizes whitespace 3. Rate Limiting (lib/Anomaly_Detector.cpp:check_rate_limiting) - Per user/host query rate tracking - Configurable time windows (3600s default) - Auto-block on threshold exceeded - Prevents DoS and brute force attacks 4. Statistical Anomaly Detection (lib/Anomaly_Detector.cpp:check_statistical_anomaly) - Z-score based outlier detection - Abnormal execution time detection (>5s) - Large result set detection (>10000 rows) - Behavioral profiling per user 5. Embedding-based Similarity (lib/Anomaly_Detector.cpp:check_embedding_similarity) - Placeholder for vector similarity search - Framework for sqlite-vec integration - Detects novel attack variations Query Flow Integration: - Added `detect_ai_anomaly()` to MySQL_Session (line 3626) - Integrated after libinjection SQLi detection (line 5150) - Blocks queries when risk threshold exceeded (default: 0.70) - Sends error response with anomaly details Status Variables Added: - `ai_detected_anomalies`: Total anomalies detected - `ai_blocked_queries`: Total queries blocked - Available via: `SELECT * FROM stats_mysql_global` Configuration (defaults): - `enabled`: true - `risk_threshold`: 70 (0-100) - `similarity_threshold`: 85 (0-100) - `rate_limit`: 100 queries/hour - `auto_block`: true - `log_only`: false Detection Pipeline: ``` Query → SQLi Check → AI Anomaly Check → [Block if needed] → Execute (libinjection) (Multi-stage) ``` Files Modified: - include/MySQL_Session.h: Added detect_ai_anomaly() declaration - include/MySQL_Thread.h: Added AI status variables - lib/Anomaly_Detector.cpp: Full implementation (700+ lines) - lib/MySQL_Session.cpp: Integration and query flow - lib/MySQL_Thread.cpp: Status variable definitions Next Steps: - Add unit tests for each detection method - Add integration tests with sample attacks - Add user and developer documentation Related: Phase 1-2 (NL2SQL foundation and testing) Related: Phase 4 (Vector storage for embeddings)	3 months ago
Rene Cannao	366164ab26	Fix #5272 : Add mysql-select_version_forwarding variable for SELECT VERSION() Since ProxySQL 3.0.4, SELECT VERSION() queries were intercepted and returned ProxySQL's mysql-server_version variable instead of proxying to backends. This broke SQLAlchemy for MariaDB which expects "MariaDB" in the version string. This commit adds a new variable `mysql-select_version_forwarding` with 4 modes: - 0 = never: Always return ProxySQL's version (3.0.4+ behavior) - 1 = always: Always proxy to backend (3.0.3 behavior) - 2 = smart (fallback to 0): Try backend connection, else ProxySQL version - 3 = smart (fallback to 1): Try backend connection, else proxy (default) The implementation includes: - New global variable mysql_thread___select_version_forwarding - New function get_backend_version_for_hostgroup() to peek at backend connection versions without removing them from the pool - Modified SELECT VERSION() handler to support all 4 modes - ProxySQL backend detection to avoid recursion Mode 3 (default) ensures SQLAlchemy always gets the real MariaDB version string while maintaining fast response when connections are available.	3 months ago
Javier Jaramago Fernández	d0e88599ee	Add special handling for unexpected COM_PING packets Implements a workaround for the handling of unexpected 'COM_PING' packets received during query processing, while a resultset is yet being streamed to the client. Received 'COM_PING' packets are queued in the form of a counter. This counter is later used to sent the corresponding number of 'OK' packets to the client after 'MySQL_Session' has finished processing the current query.	4 months ago
Rene Cannao	5a7b22181f	Fix metrics collection for wait_timeout counters The get_status_variable() function was only scanning worker threads but ignoring auxiliary threads (idle threads) where timeout terminations are detected. This caused the timeout termination counter to show incorrect/zero values. - Added idle thread scanning to both overloaded versions of get_status_variable() function - Now properly collects metrics from both worker and idle threads - Fixes the issue where proxysql_mysql_timeout_terminated_connections_total showed zero despite actual timeout terminations Resolves the metrics reading issue identified in the previous commits.	4 months ago
Rene Cannao	dc4694d656	Refactor idle session scanning and improve test precision Code improvements: - Extract SESS_TO_SCAN_idle_thread constant to header file for better maintainability - Replace magic number 128 with named constant in idle_thread_to_kill_idle_sessions() - Improve code readability and consistency in session scanning logic Test enhancements: - Add mysql-poll_timeout configuration for more precise timeout testing - Reduce test sleep times to 13 seconds for faster test execution - Add diagnostic messages to clearly show timeout configurations in test output - Ensure tests properly validate timeout enforcement with precise timing The changes improve code maintainability and make tests more reliable and faster while maintaining accurate timeout validation.	4 months ago
Rene Cannao	44aa606caa	Implement fast forward grace close feature to prevent data loss Problem: In fast forward mode, ProxySQL forwards packets directly from client to backend without buffering them. If the backend connection closes unexpectedly (e.g., due to server crash, network failure, or other issues), ProxySQL immediately closes the client session. This can result in data loss because the client may have sent additional data that hasn't been fully transmitted yet, as ProxySQL does not wait for the output buffers to drain. Solution: Implement a configurable grace period for session closure in fast forward mode. When the backend closes unexpectedly, instead of closing the session immediately, ProxySQL waits for a configurable timeout (fast_forward_grace_close_ms, default 5000ms) to allow any pending client output data to be sent. During this grace period: - If the client output buffers become empty, the session closes gracefully. - If the timeout expires, the session closes anyway to prevent indefinite hanging. Changes: - Added global variable mysql_thread___fast_forward_grace_close_ms (0-3600000ms) - Added session flags: backend_closed_in_fast_forward, fast_forward_grace_start_time - Added data stream flag: defer_close_due_to_fast_forward - Modified MySQL_Data_Stream::read_from_net() to detect backend EOF and initiate grace close if client buffers are not empty - Modified MySQL_Session::handler() FAST_FORWARD case to implement grace close logic with timeout and buffer checks - Added extensive inline documentation explaining the feature and its mechanics This prevents data loss in fast forward scenarios while maintaining bounded session lifetime.	5 months ago
Lisandro Pin	6cb522283d	Implement a ignore_min_gtid_annotation global variable for ProxySQL When true, all `min_gtid` query annotations are ignored; see https://proxysql.com/documentation/query-annotations/ for details. This is useful on ProxySQL setups with multiple layers, where some layers mandate GTID-based routing while others don't.	5 months ago
Wazir Ahmed	5219d4be93	processlist: Prevent crash due to SQLITE_TOOBIG - Add new mysql/pgsql variable `processlist_max_query_length`. - Min: 1K - Max: 32M - Default: 2M - Truncate current query based on the configuration before inserting into `stats_*_processlist` tables. - Refactor/fix code related to other processlist configurations. 1. `session_idle_show_processlist` value was not updated in `ProxySQL_Admin.variables`. 2. Pass processlist config as an argument to `MySQL_Threads_Handler::SQL3_Processlist` instead of using thread-local variables. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>	6 months ago
René Cannaò	9737e0630a	Add variable mysql-eventslog_stmt_parameters #4923 Variable mysql-eventslog_stmt_parameters configures the logging of parameters in prepared statements: - 0 : logging disabled - 1 : logging enabled	12 months ago
René Cannaò	2551d350e9	Port to 3.0 of in-memory query logging See #4713 for further details	1 year ago
Rahim Kanji	f9232ae50d	Merge remote-tracking branch 'Master/v3.0' into v3.0_session_parameter_tracking	1 year ago
Rahim Kanji	35b2dd593d	Separate implementation of SET statement parser for MySQL and PostgreSQL	1 year ago
René Cannaò	7ef5993c35	Revert "Enforce 'CLIENT_DEPRECATE_EOF' capability match for conn-pool" This reverts commit `50ce205d21`.	1 year ago
Javier Jaramago Fernández	50ce205d21	Enforce 'CLIENT_DEPRECATE_EOF' capability match for conn-pool Add new config option ('mysql-connpool_match_client_deprecate_eof') that allows to control whether to enforce a match between fronted and backend connections for 'CLIENT_DEPRECATE_EOF' capability. By default, this option is 'true', allowing smooth transitions from regular sessions to 'fast_forward'.	1 year ago
Rahim Kanji	b4e9060f71	Modified Set Parser	1 year ago
Yashwant Sahu	a0db6c3a1f	1. Changed name to protocol_compression_level. 2. Added TAP test	1 year ago
Yashwant Sahu	a846447a22	Added MySQL compression level	1 year ago
René Cannaò	cbf4628ea3	Merge remote-tracking branch 'proxysql27/v2.7' into v2.x_postgres_merge27_20240917	2 years ago
Rene Cannao	d85430b709	Implementation of PROXY protocol V1 This commit introduces: - class ProxyProtocolInfo() , that: - performs parsing - validates subnet - run automated tests in DEBUG build - variable mysql-proxy_protocol_networks . Accepted values: - empty string: disables PROXY protocol - '*' : allows connections from any IP - comma separated list of subnets - automated testing in DEBUG build during start - export of PROXY protocol information in internal session, using PROXY_V1 - a TAP test to verify various connections	2 years ago
Miro Stauder	565848c4a0	use local includes - prometheus	2 years ago
Miro Stauder	8394fd444e	use local includes - prometheus	2 years ago
René Cannaò	2d7806c325	Moved run_SetAllSession_ToProcess0 to Base_Thread	2 years ago
René Cannaò	483b8335d3	More functions moved to Base_Thread * move_session_to_idle_mysql_sessions * find_session_idx_in_mysql_sessions * ProcessAllMyDS_BeforePoll	2 years ago
René Cannaò	a881936cad	More functions moved to Base_Thread * ProcessAllMyDS_AfterPoll * read_one_byte_from_pipe * tune_timeout_for_myds_needs_pause * tune_timeout_for_session_needs_pause * configure_pollout * set_backend_to_be_skipped_if_frontend_is_slow	2 years ago
René Cannaò	db6cb93fb9	Moved ProcessAllSessions_SortingSessions to Base_Thread	2 years ago
René Cannaò	8389f76413	More functions moved to Base_Thread() * check_timing_out_session() * check_for_invalid_fd()	2 years ago
René Cannaò	c9326564c0	Minor changes to try to optimize compiling	2 years ago
René Cannaò	c903a45633	Second attempt to use Base_Thread Moved register_session() into Base_Thread. Moved Session_Regex() from Client_Session() to Base_Session() just to fix some compiler issue. Also removed some dead code from previous commit.	2 years ago
René Cannaò	0cfab2616f	First attempt to use Base_Thread and Base_Session	2 years ago
René Cannaò	de345ecaed	Merge remote-tracking branch 'public/v2.x' into v2.x_postgres_240422	2 years ago
René Cannaò	6020216dd2	Various refactoring in MySQL_Thread.cpp	2 years ago
Rahim Kanji	936e27139b	Merge branch 'v2.x_PostgreSQL_Merge' into v2.x_postgres # Conflicts: # include/MySQL_Session.h # lib/Makefile # lib/MySQL_Session.cpp # lib/MySQL_Thread.cpp # lib/ProxySQL_Admin.cpp # lib/ProxySQL_Poll.cpp	2 years ago
Rahim Kanji	ebf20b6a60	* Added PostgreSQL configuration (port: 6034 Admin and 6035: PostgreSQL client connections) * Fixed few bugs * Added Load and Save commands for PostgreSQL tables	2 years ago
Rahim Kanji	e029d39007	Initial PostgreSQL commit * Added plain text and SCRAM authentication	2 years ago
Anphuc Bui	e0db02df27	Merge branch 'v2.x' into v2.x	2 years ago
Javier Jaramago Fernández	caed360eeb	Merge branch 'v2.x' of github.com:sysown/proxysql into v2.x_sha2pass_draft3	2 years ago
René Cannaò	984582eaae	Renamed variable evaluate_replication_lag_on_servers_load Renamed variable from: mysql-recover_replication_lag_shunned_servers_on_comit to: mysql-evaluate_replication_lag_on_servers_load	2 years ago
Rahim Kanji	b75b05c392	Restore servers that are in a shunned state due to replication lag only upon commit (triggered by the command LOAD MYSQL SERVERS TO RUNTIME), provided that the replication lag is below the maximum threshold.	2 years ago
Anphuc Bui	0bcddd0002	Merge branch 'v2.x' into v2.x	2 years ago
anphucbui	bb35b20719	Moving core discovery logic to fit into read_only framework as a new task type	2 years ago

1 2 3 4 5 ...

345 Commits (bd51ff4f37ffbb1c64745b5573fa359ef9dc29c3)