73 Commits (5b1363624c272b4e6284f38f76fff91a70fc74a8)

Author	SHA1	Message	Date
Rene Cannao	17092487cf	Add protocol-labeled logger counters and FLUSH LOGS race TAP ... - Introduce proxysql_query_logger_logged_queries_total with protocol label (mysql, pgsql) for direct query-log accounting via Prometheus. - Wire MySQL and PgSQL loggers to increment/export the counter and hook PgSQL logger metrics into admin scrape updates. - Add reg_test_5389-flush_logs_no_drop-t.cpp to stress concurrent client traffic + PROXYSQL FLUSH LOGS and assert no dropped logs via metric delta == executed queries. - Document the new TAP with doxygen comments for env knobs and flow.	2 months ago
Rene Cannao	c5191d21f8	logger: avoid log drops during rotate close/open window	2 months ago
Rene Cannao	caf324f911	logger: harden buffered logging flush/rotation semantics ... This commit addresses three correctness regressions introduced by thread-local log buffering in PR #5364 (query logging performance): 1) stale logfile pointer/UAF race during concurrent rotate/close 2) stale logfile-open state after close 3) non-global flush behavior in admin/format-switch paths What was fixed - Make `flush_and_rotate()` consume the current logfile pointer under lock - Signature changed from `std::fstream*` to `std::fstream*&` - Prevents dereferencing a stale stream pointer captured before lock acquisition while another thread rotates/closes the file - Updated declaration/definition and all call sites - Add explicit synchronization for cross-thread buffer draining - Added `LogBufferThreadContext::buffer_lock` - Any path that appends or flushes a thread buffer now locks this mutex - Guarantees force-flush from admin/config paths cannot race with worker-thread appends on the same context - Restore global forced flush semantics where required - Extended `MySQL_Logger::flush` and `PgSQL_Logger::flush` to `flush(bool force = false)` - `force=false`: preserves existing low-overhead worker-loop behavior (per-thread timeout-based flush) - `force=true`: snapshots all known thread contexts and drains both events/audit buffers regardless of timeout - `flush_log()` now calls `flush(true)` before file rotation, so admin flush and format-switch operations no longer miss pending thread buffers - Avoid unintended rotation during forced draining - In `force=true` path, flush uses `rotate_fn=nullptr` - Drains buffered payload into the current file first - `flush_log()` then performs one controlled rotate/open step - Fix logfile state tracking after close - `events_close_log_unlocked()` and `audit_close_log_unlocked()` now set `logfile_open=false` when the stream is closed - Prevents write paths from treating a closed stream as open - Remove per-thread-context dependency during metadata header write - `events_open_log_unlocked()` now uses a local `LogBuffer` for metadata emission in format=1 instead of reusing a thread context buffer - Keeps open/rotate path independent from worker context lifecycle - Keep callers consistent and non-duplicative - Since `flush_log()` now force-drains internally, removed redundant explicit `flush()` calls from: - MySQL/PgSQL `eventslog_format` switch handlers - `ProxySQL_Admin::flush_logs` Behavioral outcome - No stale stream pointer use when close/rotate interleaves with flush - No false-positive logfile-open state after close - `FLUSH LOGS` and eventslog format switch now drain all known thread buffers before rotating, preventing dropped/misplaced buffered records Validation - Built modified objects directly - Ran full debug build with GENAI enabled: make clean && export PROXYSQLGENAI=1 && make debug -j24 Build completed successfully.	2 months ago
René Cannaò	568fe1524e	Merge pull request #5364 from mevishalr/feature/query-logging-improvements ... Optimize query logging performance (#5243)	2 months ago
Rene Cannao	200fdecb8d	Apply AI agent review fixes to PR #5374 ... Critical Fixes: - Fix NULL pointer dereferences in MySQL_Logger.cpp: * Add null checks for myds->myconn->parent in log_audit_entry() * Fix cleanup code in log_request() and log_audit_entry() * Change sprintf to snprintf for buffer safety - Remove redundant pthread_rwlock_init in debug.cpp: * filters_rwlock already initialized with PTHREAD_RWLOCK_INITIALIZER * Re-initialization caused undefined behavior per POSIX Major Improvements: - Add bounded wait helper wait_for_glo_mth() in proxysql_utils.h: * Replaces indefinite while loops with 10s timeout * Applied to 26 locations across 4 files Build System: - Require C++17 in Makefile (remove C++11 fallback): * Code uses C++17 structured bindings extensively * Cleanest solution vs adding fallback guards to 27+ locations Compilation fixes: - Add unistd.h include for usleep - Remove conflicting GloMyMon extern declaration Files changed: - Makefile - include/proxysql_utils.h - lib/MySQL_Logger.cpp - lib/debug.cpp - lib/MySQL_Monitor.cpp - lib/ProxySQL_Admin.cpp - lib/ClickHouse_Server.cpp - src/SQLite3_Server.cpp	2 months ago
Rene Cannao	4e39297ef9	Fix more Valgrind uninitialized value errors ... 1. Base_Thread.cpp: Initialize member variables in constructor - curtime, last_move_to_idle_thread_time, epoll_thread, shutdown, mysql_sessions were not initialized, causing uninitialized value use in ProcessAllMyDS_BeforePoll and ProcessAllMyDS_AfterPoll. 2. MySQL_Logger.cpp: Initialize query_ptr, query_len, and buf in MySQL_Event constructor. These fields were used without initialization, causing Valgrind errors in JSON serialization and other operations.	2 months ago
Rene Cannao	c2f82b3d4e	Fix more Valgrind uninitialized value errors ... 1. MySQL_Logger.cpp: Initialize username_len, schemaname_len, and server_len to 0 in MySQL_Event constructor. These fields were uninitialized and used in copy constructor and JSON serialization. 2. debug.cpp: Add default member initializers to DebugLogEntry struct to ensure POD fields are zero-initialized. Move errno save/restore to after early return checks to avoid potential TLS access issues before thread is fully initialized.	2 months ago
Rene Cannao	ee7ee0b387	Fix multiple Valgrind-reported uninitialized memory issues ... 1. MySQL_Logger.cpp: Add NULL checks for myconn->parent before accessing address/port. Client connections (frontend) have parent=NULL, causing uninitialized value reads when logging. 2. debug.cpp: Add NULL check for GloVars.global.gdbg_lvl before accessing debug level configuration. Prevents use of uninitialized data when proxy_debug_func is called before init_debug_struct() completes. 3. ProxySQL_Admin.cpp: Add NULL checks for debug filter database fields before constructing filter strings. Prevents undefined behavior when constructing std::string from NULL pointers.	2 months ago
vramesha	71143c2e84	Address AI review comments ... - Added <pthread.h> to log_utils.h and removed the header from log_utils.cpp - Pass 0 to reset_time argument in flush_and_rotate inside logger's destructor	2 months ago
vramesha	091942e9d6	Address AI review suggestions	2 months ago
vramesha	15a4ed66e9	Optimize query logging performance (#5243) ... Detailed Changes: - Introduced LogBuffer and LogBufferThreadContext to implement per-thread buffering, reducing lock contention. - Replaced localtime() with localtime_r() for improved performance and thread safety during timestamp generation. - Implemented configurable sampling for event logs to reduce I/O overhead. - Added global configuration variables to control flush size, flush interval, and sampling rate for event and audit logs.	2 months ago
Rene Cannao	9262089060	refactor: migrate prepare_v2 to RAII API in MySQL_Logger ... Migrate insertMysqlEventsIntoDb() function from deprecated prepare_v2() to the new RAII-based API using raw pointer aliases. Pattern used: auto [rc, statement_unique] = db->prepare_v2(query); sqlite3_stmt *statement = statement_unique.get(); // ... use statement directly ... // No manual finalize - RAII handles it This addresses 2 of the ~150 deprecated prepare_v2 warnings.	2 months ago
Zuofei Wang	241f17eb98	fix mysql binary logging 0 parameters	10 months ago
René Cannaò	216b31feae	Added new event_type PROXYSQL_METADATA ... When creating an event log in binary format, a metadata packet is written. The metadata is in JSON format. It currently only provide the ProxySQL version. For illustration purposes, tool eventlog_reader_to_json.cpp supports it too. This commit includes also some reformatting	11 months ago
René Cannaò	9737e0630a	Add variable mysql-eventslog_stmt_parameters #4923 ... Variable mysql-eventslog_stmt_parameters configures the logging of parameters in prepared statements: - 0 : logging disabled - 1 : logging enabled	12 months ago
René Cannaò	97304ffa91	prepared statements logging: use HEX for STRING type ... When parameters are strings or blobs, they are always converted to hex	12 months ago
René Cannaò	65f79a07d2	Changes on write_query_format_1() for datetime ... When logging the parameters of COM_STMT_EXECUTE , use sizeof(MYSQL_TIME) for these types: * MYSQL_TYPE_TIMESTAMP * MYSQL_TYPE_DATE * MYSQL_TYPE_TIME * MYSQL_TYPE_DATETIME Notes: * sizeof(MYSQL_TIME) is 40 * client library may use MYSQL_TYPE_DATETIME for all of these types	12 months ago
René Cannaò	cbb8505ca1	Improvement in prepared statements logging ... In bufferTypeInfoMap , use sizeof(MYSQL_TIME) for the size of these types: * MYSQL_TYPE_TIMESTAMP * MYSQL_TYPE_DATE * MYSQL_TYPE_TIME * MYSQL_TYPE_DATETIME Notes: * sizeof(MYSQL_TIME) is 40 * client library may use MYSQL_TYPE_DATETIME for all of these types	12 months ago
René Cannaò	a39580ffe1	Several fixes on prepared statements parameters logging ... In write_query_format_1() : * write the number of parameters (previously erroneously skipped) * write null_bitmap only if there are parameters * write the parameters values (this was previously erroneously skipped) In test_ps_logging-t.cpp : * verify that query logging is configured to file In eventlog_reader_to_json.cpp : * added verbose logging * added enum `log_event_type` for properly identifying event type * parse all events in the input file	1 year ago
René Cannaò	be0fbeedc8	Adding documention for write_query_format_2_json()	1 year ago
René Cannaò	39031f614b	Uses 0 params in write_query_format_1() ... When logging COM_STMT_EXECUTE parameters, we check stored parameters. In the unexpected event (it should never happen) that either `session` or `session->CurrentQuery.stmt_meta` are `nullptr` , we log 0 parameters to ensure a deterministic format in the query logging file.	1 year ago
René Cannaò	579f04de30	Inline documentation in write_query_format_1() ... Added inline documentation in write_query_format_1() related to the logging of parameters when COM_STMT_EXECUTE is logged	1 year ago
René Cannaò	ad190e9b56	WIP for logging of prepared statements parameters ... Added: * bufferTypeInfoMap : to map mysql type to typename and function to convert to JSON * extractStmtExecuteMetadataToJson() : function to generate a JSON with parameters Modified write_query_format_1() to write parameters. This is still a WIP	1 year ago
René Cannaò	f47b0e6f6c	Changed to MySQL_Event ... Adding session Addin a placeholder for extractStmtExecuteMetadataToJson()	1 year ago
René Cannaò	3475f8a75f	Port of PR #4822 to v3.0	1 year ago
René Cannaò	cb53db81ca	Export MySQL_Logger metrics for v3.0 ... Port of #4802	1 year ago
René Cannaò	2551d350e9	Port to 3.0 of in-memory query logging ... See #4713 for further details	1 year ago
Rahim Kanji	e723082a1b	Refactored Query Processor Architecture ... - Introduced separate query processors for MySQL and PostgreSQL. - Removed unnecessary GTID logic from `PgSQL_Session`. - Added a command counter for PostgreSQL queries. - Refactored `Query_Processor` to serve as a base class containing core functionalities applicable to both MySQL and PostgreSQL. Changes: - Removed GTID-related code from `PgSQL_Logger` and `PgSQL_Session`: - bool with_gtid; - char gtid_buf[128]; - int gtid_hid; - Added new PostgreSQL-specific tables: - `pgsql_query_rules`, `pgsql_firewall_whitelist_rules`, `pgsql_query_rules_fast_routing` - `runtime_pgsql_query_rules`, `runtime_pgsql_firewall_whitelist_rules`, `runtime_pgsql_query_rules_fast_routing` - Added PostgreSQL-related variables: pgsql_thread___commands_stats pgsql_thread___query_digests pgsql_thread___query_digests_lowercase pgsql_thread___query_digests_no_digits pgsql_thread___query_digests_replace_null pgsql_thread___query_digests_normalize_digest_text pgsql_thread___query_digests_track_hostname pgsql_thread___query_digests_keep_comment pgsql_thread___query_digests_max_digest_length pgsql_thread___query_digests_max_query_length pgsql_thread___query_digests_grouping_limit pgsql_thread___query_digests_groups_grouping_limit pgsql_thread___enable_load_data_local_infile pgsql_thread___auditlog_filename pgsql_thread___auditlog_filesize pgsql_thread___eventslog_filename pgsql_thread___eventslog_filesize pgsql_thread___eventslog_default_log pgsql_thread___eventslog_format pgsql_thread___firewall_whitelist_errormsg pgsql_thread___firewall_whitelist_enabled pgsql_thread___query_processor_iterations pgsql_thread___query_processor_regex	2 years ago
René Cannaò	fde8ace076	Removed json.hpp from include files ... This seems to speedup compile time, and also lead to smaller binary	2 years ago
Miro Stauder	b1dc534757	refactor out pktH	3 years ago
René Cannaò	b546d51d90	Fix uninitialized variable	3 years ago
René Cannaò	fcb2acd67c	Added TAP test for logging last_insert_id ... Also modified the Logger to log last_insert_id in JSON format only if not 0	3 years ago
René Cannaò	c0745b42c2	Adding last_insert_id and last_gtid to MySQL_Logger	3 years ago
René Cannaò	3837de61e2	Various minor fixes ... Fixes: * minor memory leak when running CHECKSUM command * minor memory leak in load_mysql_servers_to_runtime() * not initialized json in MySQL_Event (valgrind doesn't like it) * not initialized variable MySQL_Thread::shutdown in the constructor	3 years ago
Javier Jaramago Fernández	2ecbad3244	Fix warning due to catching of polymorphic type by value	3 years ago
Javier Jaramago Fernández	a661d33391	Fix invalid digest value in 'eventslog' for 'COM_STMT_EXECUTE' ... 'COM_STMT_EXECUTE' invalidly reported the 'digest' value from the previous query when preceded by a 'COM_QUERY'.	3 years ago
Javier Jaramago Fernández	41ef582fc5	Add new field 'stmt_id' to eventslog for 'STMT_PREPARE|STMT_EXECUTE' events	3 years ago
Javier Jaramago Fernández	f993402c62	Fix crash when failed 'connection_reset' is tried to be logged	4 years ago
René Cannaò	aceeaf79ed	Disabled prometheus metrics dump , see #3571 ... This commit also fixes some minor memory corruptions	5 years ago
Rene Cannao	41b173a83c	Fixing a lot of compiler warnings	6 years ago
René Cannaò	e062f84c3e	Merge pull request #2399 from val214/v2.0.9-mem-leaks ... [WIP] V2.0.9 mem leaks fixes	6 years ago
peter	6bb057fbda	Fix add a mutex lock twice in the process of function call(log_audit_entry() -> write() -> write_auth()) ... Bug description: Since fix write event log and audit log coredump(#2427)， i think i left out a code that was not processed, resulting in coredump again in the program of proxysql , when i confige these parameters of auditlog_filename and auditlog_filesize in proxysql.cnf file or i set these parameters in proxysql-admin command line. the coredump contest: (gdb) bt #0 0x00002b224e9981f7 in raise () from /lib64/libc.so.6 #1 0x00002b224e9998e8 in abort () from /lib64/libc.so.6 #2 0x00002b224e991266 in __assert_fail_base () from /lib64/libc.so.6 #3 0x00002b224e991312 in __assert_fail () from /lib64/libc.so.6 #4 0x00000000005b2dc3 in main (argc=5, argv=0x7ffea714fd48) at main.cpp:1664 the proxysql.log contest: 2019-12-06 11:03:07 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:10 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:13 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:16 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:19 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:22 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:25 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:28 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:31 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:34 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:34 main.cpp:1663:main(): [ERROR] Watchdog: reached 10 missed heartbeats. Aborting! proxysql: main.cpp:1664: int main(int, const char**): Assertion `0' failed. Error: signal 6: proxysql(_Z13crash_handleri+0x25)[0x5b607c] /lib64/libc.so.6(+0x35270)[0x2b224e998270] /lib64/libc.so.6(gsignal+0x37)[0x2b224e9981f7] /lib64/libc.so.6(abort+0x148)[0x2b224e9998e8] /lib64/libc.so.6(+0x2e266)[0x2b224e991266] /lib64/libc.so.6(+0x2e312)[0x2b224e991312] proxysql(main+0xe3a)[0x5b2dc3] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b224e984c05] proxysql[0x5adaa5] 2019-12-06 11:03:34 main.cpp:1396:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2019-12-06 11:03:34 main.cpp:1397:ProxySQL_daemonize_phase3(): [INFO] ProxySQL version _DEBUG Impact of this fix： I analyzed the call relationship of write_auth function of MySQL_Logger.cpp file in the whole project of proxysql with the source insight ide. I found write_auth function that was only called by the write function of MySQL_Logger.cpp file in the whole project of proxysql, and this write function was only called by log_audit_entry function or log_request function of MySQL_Logger.cpp file in the whole project of proxysql. However, i've added a mutex lock separately in log_audit_entry function and log_request function. And so i think it should delete the code for the wrlock function call in write_auth function, and it doesn't have impact for this time fix. Bux fix: Fix add a mutex lock twice in the process of function call(log_audit_entry() -> write() -> write_auth())	6 years ago
bskllzh	d71d01cb63	Fix write event log and audit log coredump(the events.logfile&audit.logfile null pointer) ... Bug description: When i make a benchmark test with a sysbench(version: 1.0.17) in proxysql 2.0.8 + mysql 5.7.26, i get the follow error info: Error: signal 11: proxysql(_Z13crash_handleri+0x25)[0x5b607c] /lib64/libc.so.6(+0x35270)[0x2b673de34270] /lib64/libstdc++.so.6(_ZNSo6sentryC2ERSo+0xf)[0x2b673d67521f] /lib64/libstdc++.so.6(_ZNSo5writeEPKcl+0x31)[0x2b673d675621] proxysql(_ZN11MySQL_Event20write_query_format_1EPSt13basic_fstreamIcSt11char_traitsIcEE+0x27d)[0x6e2d81] proxysql(_ZN11MySQL_Event5writeEPSt13basic_fstreamIcSt11char_traitsIcEEP13MySQL_Session+0x79)[0x6e19e9] proxysql(_ZN12MySQL_Logger11log_requestEP13MySQL_SessionP17MySQL_Data_Stream+0x710)[0x6e5416] proxysql(_ZN13MySQL_Session8LogQueryEP17MySQL_Data_Stream+0x6d)[0x654055] proxysql(_ZN13MySQL_Session10RequestEndEP17MySQL_Data_Stream+0x35)[0x6540d7] proxysql(_ZN13MySQL_Session7handlerEv+0x4397)[0x6459ed] proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x830)[0x623180] proxysql(_ZN12MySQL_Thread3runEv+0x2cbe)[0x62208a] proxysql(_Z24mysql_worker_thread_funcPv+0xbb)[0x5afb05] /lib64/libpthread.so.0(+0x7e25)[0x2b673cda8e25] /lib64/libc.so.6(clone+0x6d)[0x2b673def734d] 2019-12-02 09:52:26 main.cpp:1396:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! I analyze the coredump with proxysql 2.0.8 debug version very carefully, and i found the variable of events.logfile is null pointer in log_request function when it call write function(me.write(events.logfile, sess)). The following is the coredump（write (this=0x2b67402fd1f0, f=0x0, sess=0x2b67414a8c00)）: (gdb) bt #0 0x00002b673d67521f in std::ostream::sentry::sentry(std::ostream&) () from /lib64/libstdc++.so.6 #1 0x00002b673d675621 in std::ostream::write(char const*, long) () from /lib64/libstdc++.so.6 #2 0x00000000006e2d81 in MySQL_Event::write_query_format_1 (this=0x2b67402fd1f0, f=0x0) at MySQL_Logger.cpp:265 #3 0x00000000006e19e9 in MySQL_Event::write (this=0x2b67402fd1f0, f=0x0, sess=0x2b67414a8c00) at MySQL_Logger.cpp:89 #4 0x00000000006e5416 in MySQL_Logger::log_request (this=0x2b673e419680, sess=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Logger.cpp:717 #5 0x0000000000654055 in MySQL_Session::LogQuery (this=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Session.cpp:6691 #6 0x00000000006540d7 in MySQL_Session::RequestEnd (this=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Session.cpp:6711 #7 0x00000000006459ed in MySQL_Session::handler (this=0x2b67414a8c00) at MySQL_Session.cpp:4086 #8 0x0000000000623180 in MySQL_Thread::process_all_sessions (this=0x2b6741301000) at MySQL_Thread.cpp:4392 #9 0x000000000062208a in MySQL_Thread::run (this=0x2b6741301000) at MySQL_Thread.cpp:4125 #10 0x00000000005afb05 in mysql_worker_thread_func (arg=0x2b673e461f30) at main.cpp:648 #11 0x00002b673cda8e25 in start_thread () from /lib64/libpthread.so.0 #12 0x00002b673def734d in clone () from /lib64/libc.so.6 And then i add a line of code in log_request function of MySQL_Logger.cpp file to check up whether events.logfile is null pointer ，as follow: void MySQL_Logger::log_request(MySQL_Session *sess, MySQL_Data_Stream *myds) { if (events.enabled==false) return; if (events.logfile==NULL) return; proxy_info("log_request: First: events.logfile is null %d\n", (events.logfile == NULL) ? 1 : 0); MySQL_Connection_userinfo *ui=sess->client_myds->myconn->userinfo; And i make a benchmark test again with proxysql 2.0.8 debug version, the coredump of proxysql happen again, then i get the follow print info from proxysql.log [root@XXXXX proxysql]# grep -n -w 'log_request: First: events.logfile is null 1' proxysql.log 7358447:2019-12-02 09:52:26 MySQL_Logger.cpp:617:log_request(): [INFO] log_request: First: events.logfile is null 1 7358448:2019-12-02 09:52:26 MySQL_Logger.cpp:617:log_request(): [INFO] log_request: First: events.logfile is null 1 Bux fix: Fix write event log and audit log coredump, because the proxysql get the null pointer variable of events.logfile（audit.logfile）in a multithreaded environment.	6 years ago
Valentin Rakush	6a6873b342	Fix leaks in logger and admin	6 years ago
René Cannaò	ce4a4a2464	Code cleanup and speeding compile time	6 years ago
René Cannaò	45e701b54d	Code cleanup and speeding compile time	6 years ago
René Cannaò	80e92b144d	[WIP] Second commit for mysql firewall whitelist ... Improved also compiling time	6 years ago
René Cannaò	3ad899c758	Avoid crash in JSON library for invalid UTF8	7 years ago
René Cannaò	3b3def9f78	Fix unitialized file	7 years ago
René Cannaò	255d9247b8	Reduce locking on MySQL_Logger writes to disk	7 years ago