62 Commits (92620890603fe7f9efd6f01bdbf9abb85eed09ca)

Author	SHA1	Message	Date
Rene Cannao	9262089060	refactor: migrate prepare_v2 to RAII API in MySQL_Logger ... Migrate insertMysqlEventsIntoDb() function from deprecated prepare_v2() to the new RAII-based API using raw pointer aliases. Pattern used: auto [rc, statement_unique] = db->prepare_v2(query); sqlite3_stmt *statement = statement_unique.get(); // ... use statement directly ... // No manual finalize - RAII handles it This addresses 2 of the ~150 deprecated prepare_v2 warnings.	2 weeks ago
Zuofei Wang	241f17eb98	fix mysql binary logging 0 parameters	8 months ago
René Cannaò	216b31feae	Added new event_type PROXYSQL_METADATA ... When creating an event log in binary format, a metadata packet is written. The metadata is in JSON format. It currently only provide the ProxySQL version. For illustration purposes, tool eventlog_reader_to_json.cpp supports it too. This commit includes also some reformatting	10 months ago
René Cannaò	9737e0630a	Add variable mysql-eventslog_stmt_parameters #4923 ... Variable mysql-eventslog_stmt_parameters configures the logging of parameters in prepared statements: - 0 : logging disabled - 1 : logging enabled	10 months ago
René Cannaò	97304ffa91	prepared statements logging: use HEX for STRING type ... When parameters are strings or blobs, they are always converted to hex	10 months ago
René Cannaò	65f79a07d2	Changes on write_query_format_1() for datetime ... When logging the parameters of COM_STMT_EXECUTE , use sizeof(MYSQL_TIME) for these types: * MYSQL_TYPE_TIMESTAMP * MYSQL_TYPE_DATE * MYSQL_TYPE_TIME * MYSQL_TYPE_DATETIME Notes: * sizeof(MYSQL_TIME) is 40 * client library may use MYSQL_TYPE_DATETIME for all of these types	10 months ago
René Cannaò	cbb8505ca1	Improvement in prepared statements logging ... In bufferTypeInfoMap , use sizeof(MYSQL_TIME) for the size of these types: * MYSQL_TYPE_TIMESTAMP * MYSQL_TYPE_DATE * MYSQL_TYPE_TIME * MYSQL_TYPE_DATETIME Notes: * sizeof(MYSQL_TIME) is 40 * client library may use MYSQL_TYPE_DATETIME for all of these types	10 months ago
René Cannaò	a39580ffe1	Several fixes on prepared statements parameters logging ... In write_query_format_1() : * write the number of parameters (previously erroneously skipped) * write null_bitmap only if there are parameters * write the parameters values (this was previously erroneously skipped) In test_ps_logging-t.cpp : * verify that query logging is configured to file In eventlog_reader_to_json.cpp : * added verbose logging * added enum `log_event_type` for properly identifying event type * parse all events in the input file	11 months ago
René Cannaò	be0fbeedc8	Adding documention for write_query_format_2_json()	11 months ago
René Cannaò	39031f614b	Uses 0 params in write_query_format_1() ... When logging COM_STMT_EXECUTE parameters, we check stored parameters. In the unexpected event (it should never happen) that either `session` or `session->CurrentQuery.stmt_meta` are `nullptr` , we log 0 parameters to ensure a deterministic format in the query logging file.	11 months ago
René Cannaò	579f04de30	Inline documentation in write_query_format_1() ... Added inline documentation in write_query_format_1() related to the logging of parameters when COM_STMT_EXECUTE is logged	11 months ago
René Cannaò	ad190e9b56	WIP for logging of prepared statements parameters ... Added: * bufferTypeInfoMap : to map mysql type to typename and function to convert to JSON * extractStmtExecuteMetadataToJson() : function to generate a JSON with parameters Modified write_query_format_1() to write parameters. This is still a WIP	11 months ago
René Cannaò	f47b0e6f6c	Changed to MySQL_Event ... Adding session Addin a placeholder for extractStmtExecuteMetadataToJson()	11 months ago
René Cannaò	3475f8a75f	Port of PR #4822 to v3.0	12 months ago
René Cannaò	cb53db81ca	Export MySQL_Logger metrics for v3.0 ... Port of #4802	12 months ago
René Cannaò	2551d350e9	Port to 3.0 of in-memory query logging ... See #4713 for further details	12 months ago
Rahim Kanji	e723082a1b	Refactored Query Processor Architecture ... - Introduced separate query processors for MySQL and PostgreSQL. - Removed unnecessary GTID logic from `PgSQL_Session`. - Added a command counter for PostgreSQL queries. - Refactored `Query_Processor` to serve as a base class containing core functionalities applicable to both MySQL and PostgreSQL. Changes: - Removed GTID-related code from `PgSQL_Logger` and `PgSQL_Session`: - bool with_gtid; - char gtid_buf[128]; - int gtid_hid; - Added new PostgreSQL-specific tables: - `pgsql_query_rules`, `pgsql_firewall_whitelist_rules`, `pgsql_query_rules_fast_routing` - `runtime_pgsql_query_rules`, `runtime_pgsql_firewall_whitelist_rules`, `runtime_pgsql_query_rules_fast_routing` - Added PostgreSQL-related variables: pgsql_thread___commands_stats pgsql_thread___query_digests pgsql_thread___query_digests_lowercase pgsql_thread___query_digests_no_digits pgsql_thread___query_digests_replace_null pgsql_thread___query_digests_normalize_digest_text pgsql_thread___query_digests_track_hostname pgsql_thread___query_digests_keep_comment pgsql_thread___query_digests_max_digest_length pgsql_thread___query_digests_max_query_length pgsql_thread___query_digests_grouping_limit pgsql_thread___query_digests_groups_grouping_limit pgsql_thread___enable_load_data_local_infile pgsql_thread___auditlog_filename pgsql_thread___auditlog_filesize pgsql_thread___eventslog_filename pgsql_thread___eventslog_filesize pgsql_thread___eventslog_default_log pgsql_thread___eventslog_format pgsql_thread___firewall_whitelist_errormsg pgsql_thread___firewall_whitelist_enabled pgsql_thread___query_processor_iterations pgsql_thread___query_processor_regex	1 year ago
René Cannaò	fde8ace076	Removed json.hpp from include files ... This seems to speedup compile time, and also lead to smaller binary	2 years ago
Miro Stauder	b1dc534757	refactor out pktH	3 years ago
René Cannaò	b546d51d90	Fix uninitialized variable	3 years ago
René Cannaò	fcb2acd67c	Added TAP test for logging last_insert_id ... Also modified the Logger to log last_insert_id in JSON format only if not 0	3 years ago
René Cannaò	c0745b42c2	Adding last_insert_id and last_gtid to MySQL_Logger	3 years ago
René Cannaò	3837de61e2	Various minor fixes ... Fixes: * minor memory leak when running CHECKSUM command * minor memory leak in load_mysql_servers_to_runtime() * not initialized json in MySQL_Event (valgrind doesn't like it) * not initialized variable MySQL_Thread::shutdown in the constructor	3 years ago
Javier Jaramago Fernández	2ecbad3244	Fix warning due to catching of polymorphic type by value	3 years ago
Javier Jaramago Fernández	a661d33391	Fix invalid digest value in 'eventslog' for 'COM_STMT_EXECUTE' ... 'COM_STMT_EXECUTE' invalidly reported the 'digest' value from the previous query when preceded by a 'COM_QUERY'.	3 years ago
Javier Jaramago Fernández	41ef582fc5	Add new field 'stmt_id' to eventslog for 'STMT_PREPARE|STMT_EXECUTE' events	3 years ago
Javier Jaramago Fernández	f993402c62	Fix crash when failed 'connection_reset' is tried to be logged	4 years ago
René Cannaò	aceeaf79ed	Disabled prometheus metrics dump , see #3571 ... This commit also fixes some minor memory corruptions	5 years ago
Rene Cannao	41b173a83c	Fixing a lot of compiler warnings	6 years ago
René Cannaò	e062f84c3e	Merge pull request #2399 from val214/v2.0.9-mem-leaks ... [WIP] V2.0.9 mem leaks fixes	6 years ago
peter	6bb057fbda	Fix add a mutex lock twice in the process of function call(log_audit_entry() -> write() -> write_auth()) ... Bug description: Since fix write event log and audit log coredump(#2427)， i think i left out a code that was not processed, resulting in coredump again in the program of proxysql , when i confige these parameters of auditlog_filename and auditlog_filesize in proxysql.cnf file or i set these parameters in proxysql-admin command line. the coredump contest: (gdb) bt #0 0x00002b224e9981f7 in raise () from /lib64/libc.so.6 #1 0x00002b224e9998e8 in abort () from /lib64/libc.so.6 #2 0x00002b224e991266 in __assert_fail_base () from /lib64/libc.so.6 #3 0x00002b224e991312 in __assert_fail () from /lib64/libc.so.6 #4 0x00000000005b2dc3 in main (argc=5, argv=0x7ffea714fd48) at main.cpp:1664 the proxysql.log contest: 2019-12-06 11:03:07 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:10 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:13 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:16 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:19 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:22 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:25 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:28 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:31 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:34 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:34 main.cpp:1663:main(): [ERROR] Watchdog: reached 10 missed heartbeats. Aborting! proxysql: main.cpp:1664: int main(int, const char**): Assertion `0' failed. Error: signal 6: proxysql(_Z13crash_handleri+0x25)[0x5b607c] /lib64/libc.so.6(+0x35270)[0x2b224e998270] /lib64/libc.so.6(gsignal+0x37)[0x2b224e9981f7] /lib64/libc.so.6(abort+0x148)[0x2b224e9998e8] /lib64/libc.so.6(+0x2e266)[0x2b224e991266] /lib64/libc.so.6(+0x2e312)[0x2b224e991312] proxysql(main+0xe3a)[0x5b2dc3] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b224e984c05] proxysql[0x5adaa5] 2019-12-06 11:03:34 main.cpp:1396:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2019-12-06 11:03:34 main.cpp:1397:ProxySQL_daemonize_phase3(): [INFO] ProxySQL version _DEBUG Impact of this fix： I analyzed the call relationship of write_auth function of MySQL_Logger.cpp file in the whole project of proxysql with the source insight ide. I found write_auth function that was only called by the write function of MySQL_Logger.cpp file in the whole project of proxysql, and this write function was only called by log_audit_entry function or log_request function of MySQL_Logger.cpp file in the whole project of proxysql. However, i've added a mutex lock separately in log_audit_entry function and log_request function. And so i think it should delete the code for the wrlock function call in write_auth function, and it doesn't have impact for this time fix. Bux fix: Fix add a mutex lock twice in the process of function call(log_audit_entry() -> write() -> write_auth())	6 years ago
bskllzh	d71d01cb63	Fix write event log and audit log coredump(the events.logfile&audit.logfile null pointer) ... Bug description: When i make a benchmark test with a sysbench(version: 1.0.17) in proxysql 2.0.8 + mysql 5.7.26, i get the follow error info: Error: signal 11: proxysql(_Z13crash_handleri+0x25)[0x5b607c] /lib64/libc.so.6(+0x35270)[0x2b673de34270] /lib64/libstdc++.so.6(_ZNSo6sentryC2ERSo+0xf)[0x2b673d67521f] /lib64/libstdc++.so.6(_ZNSo5writeEPKcl+0x31)[0x2b673d675621] proxysql(_ZN11MySQL_Event20write_query_format_1EPSt13basic_fstreamIcSt11char_traitsIcEE+0x27d)[0x6e2d81] proxysql(_ZN11MySQL_Event5writeEPSt13basic_fstreamIcSt11char_traitsIcEEP13MySQL_Session+0x79)[0x6e19e9] proxysql(_ZN12MySQL_Logger11log_requestEP13MySQL_SessionP17MySQL_Data_Stream+0x710)[0x6e5416] proxysql(_ZN13MySQL_Session8LogQueryEP17MySQL_Data_Stream+0x6d)[0x654055] proxysql(_ZN13MySQL_Session10RequestEndEP17MySQL_Data_Stream+0x35)[0x6540d7] proxysql(_ZN13MySQL_Session7handlerEv+0x4397)[0x6459ed] proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x830)[0x623180] proxysql(_ZN12MySQL_Thread3runEv+0x2cbe)[0x62208a] proxysql(_Z24mysql_worker_thread_funcPv+0xbb)[0x5afb05] /lib64/libpthread.so.0(+0x7e25)[0x2b673cda8e25] /lib64/libc.so.6(clone+0x6d)[0x2b673def734d] 2019-12-02 09:52:26 main.cpp:1396:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! I analyze the coredump with proxysql 2.0.8 debug version very carefully, and i found the variable of events.logfile is null pointer in log_request function when it call write function(me.write(events.logfile, sess)). The following is the coredump（write (this=0x2b67402fd1f0, f=0x0, sess=0x2b67414a8c00)）: (gdb) bt #0 0x00002b673d67521f in std::ostream::sentry::sentry(std::ostream&) () from /lib64/libstdc++.so.6 #1 0x00002b673d675621 in std::ostream::write(char const*, long) () from /lib64/libstdc++.so.6 #2 0x00000000006e2d81 in MySQL_Event::write_query_format_1 (this=0x2b67402fd1f0, f=0x0) at MySQL_Logger.cpp:265 #3 0x00000000006e19e9 in MySQL_Event::write (this=0x2b67402fd1f0, f=0x0, sess=0x2b67414a8c00) at MySQL_Logger.cpp:89 #4 0x00000000006e5416 in MySQL_Logger::log_request (this=0x2b673e419680, sess=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Logger.cpp:717 #5 0x0000000000654055 in MySQL_Session::LogQuery (this=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Session.cpp:6691 #6 0x00000000006540d7 in MySQL_Session::RequestEnd (this=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Session.cpp:6711 #7 0x00000000006459ed in MySQL_Session::handler (this=0x2b67414a8c00) at MySQL_Session.cpp:4086 #8 0x0000000000623180 in MySQL_Thread::process_all_sessions (this=0x2b6741301000) at MySQL_Thread.cpp:4392 #9 0x000000000062208a in MySQL_Thread::run (this=0x2b6741301000) at MySQL_Thread.cpp:4125 #10 0x00000000005afb05 in mysql_worker_thread_func (arg=0x2b673e461f30) at main.cpp:648 #11 0x00002b673cda8e25 in start_thread () from /lib64/libpthread.so.0 #12 0x00002b673def734d in clone () from /lib64/libc.so.6 And then i add a line of code in log_request function of MySQL_Logger.cpp file to check up whether events.logfile is null pointer ，as follow: void MySQL_Logger::log_request(MySQL_Session *sess, MySQL_Data_Stream *myds) { if (events.enabled==false) return; if (events.logfile==NULL) return; proxy_info("log_request: First: events.logfile is null %d\n", (events.logfile == NULL) ? 1 : 0); MySQL_Connection_userinfo *ui=sess->client_myds->myconn->userinfo; And i make a benchmark test again with proxysql 2.0.8 debug version, the coredump of proxysql happen again, then i get the follow print info from proxysql.log [root@XXXXX proxysql]# grep -n -w 'log_request: First: events.logfile is null 1' proxysql.log 7358447:2019-12-02 09:52:26 MySQL_Logger.cpp:617:log_request(): [INFO] log_request: First: events.logfile is null 1 7358448:2019-12-02 09:52:26 MySQL_Logger.cpp:617:log_request(): [INFO] log_request: First: events.logfile is null 1 Bux fix: Fix write event log and audit log coredump, because the proxysql get the null pointer variable of events.logfile（audit.logfile）in a multithreaded environment.	6 years ago
Valentin Rakush	6a6873b342	Fix leaks in logger and admin	6 years ago
René Cannaò	ce4a4a2464	Code cleanup and speeding compile time	6 years ago
René Cannaò	45e701b54d	Code cleanup and speeding compile time	6 years ago
René Cannaò	80e92b144d	[WIP] Second commit for mysql firewall whitelist ... Improved also compiling time	6 years ago
René Cannaò	3ad899c758	Avoid crash in JSON library for invalid UTF8	6 years ago
René Cannaò	3b3def9f78	Fix unitialized file	7 years ago
René Cannaò	255d9247b8	Reduce locking on MySQL_Logger writes to disk	7 years ago
René Cannaò	0831740e45	Several enhancements on Query logger ... Enhancements: - added metrics rows_affected and rows_sent - added global variable mysql-eventslog_default_log : if 1 , logging is enabled for every query unless explicitly disabled in mysql_query_rules.log . Default is 0 - added global variable mysql-eventslog_format : default is 1 (legacy format). A value of 2 enables logging in JSON format. Issue #871 Changing value at runtime causes the current file to be closed and a new one created - fixed logging for prepared statements: till 2.0.5 only some percentage of prepared statements was correctly logged Extended tables stats_mysql_query_digest and stats_mysql_query_digest_reset to also include sum_rows_affected and sum_rows_sent Extended `eventslog_reader_sample.cpp` to support the new enhancements	7 years ago
René Cannaò	37d79ad23a	Fixed compiler warning: ISO C++ forbids converting a string constant to char pointer	7 years ago
René Cannaò	7ea33a1312	Completing initial implementation of Audit module	7 years ago
René Cannaò	9e2c1fdda1	Draft implementation of audit log	7 years ago
Nick Vyzas	28a9e3f984	Fixes - "Event log file is overwritten on ProxySQL service restart" #1201	8 years ago
René Cannaò	5e05348b22	Remove spinlock from MySQL Logger #977	9 years ago
René Cannaò	a6c53d0d12	Revert "Refactor header includes."	9 years ago
Maxim Bublis	30dbcc21a8	Refactor header includes. ... * reduce build time by 25%; * remove circular dependency on `cpp.h` and other headers. * fixes issue with `my_global.h` from mariadbclient which redefines `__attribute__`, that leads to broken STL and GCC intrinsics if `<algorithm>` wasn't included prior including this header; * fixes another issue with redefined `__attribute__` which leads to `_conn_exchange_t` not being aligned to cache line size; * removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize` to be `pthread_dummy(0)` which led to stack size being never adjusted; * add missing include guards to some headers; * remove unused proxysql_hash.h header.	9 years ago
Maxim Bublis	043a321e55	Cleanup legacy codebase.	9 years ago
René Cannaò	fb12719ba7	Quick return from MySQL_Logger::flush_log() #644	10 years ago
René Cannaò	8e81495bee	Improved logging ... Added logging information for: - client - server - hostgroup_id - digest - duration	10 years ago