proxysql

Commit Graph

Author	SHA1	Message	Date
Rahim Kanji	e723082a1b	Refactored Query Processor Architecture - Introduced separate query processors for MySQL and PostgreSQL. - Removed unnecessary GTID logic from `PgSQL_Session`. - Added a command counter for PostgreSQL queries. - Refactored `Query_Processor` to serve as a base class containing core functionalities applicable to both MySQL and PostgreSQL. Changes: - Removed GTID-related code from `PgSQL_Logger` and `PgSQL_Session`: - bool with_gtid; - char gtid_buf[128]; - int gtid_hid; - Added new PostgreSQL-specific tables: - `pgsql_query_rules`, `pgsql_firewall_whitelist_rules`, `pgsql_query_rules_fast_routing` - `runtime_pgsql_query_rules`, `runtime_pgsql_firewall_whitelist_rules`, `runtime_pgsql_query_rules_fast_routing` - Added PostgreSQL-related variables: pgsql_thread___commands_stats pgsql_thread___query_digests pgsql_thread___query_digests_lowercase pgsql_thread___query_digests_no_digits pgsql_thread___query_digests_replace_null pgsql_thread___query_digests_normalize_digest_text pgsql_thread___query_digests_track_hostname pgsql_thread___query_digests_keep_comment pgsql_thread___query_digests_max_digest_length pgsql_thread___query_digests_max_query_length pgsql_thread___query_digests_grouping_limit pgsql_thread___query_digests_groups_grouping_limit pgsql_thread___enable_load_data_local_infile pgsql_thread___auditlog_filename pgsql_thread___auditlog_filesize pgsql_thread___eventslog_filename pgsql_thread___eventslog_filesize pgsql_thread___eventslog_default_log pgsql_thread___eventslog_format pgsql_thread___firewall_whitelist_errormsg pgsql_thread___firewall_whitelist_enabled pgsql_thread___query_processor_iterations pgsql_thread___query_processor_regex	1 year ago
René Cannaò	fde8ace076	Removed json.hpp from include files This seems to speedup compile time, and also lead to smaller binary	2 years ago
Miro Stauder	b1dc534757	refactor out pktH	3 years ago
René Cannaò	b546d51d90	Fix uninitialized variable	3 years ago
René Cannaò	fcb2acd67c	Added TAP test for logging last_insert_id Also modified the Logger to log last_insert_id in JSON format only if not 0	3 years ago
René Cannaò	c0745b42c2	Adding last_insert_id and last_gtid to MySQL_Logger	3 years ago
René Cannaò	3837de61e2	Various minor fixes Fixes: * minor memory leak when running CHECKSUM command * minor memory leak in load_mysql_servers_to_runtime() * not initialized json in MySQL_Event (valgrind doesn't like it) * not initialized variable MySQL_Thread::shutdown in the constructor	3 years ago
Javier Jaramago Fernández	2ecbad3244	Fix warning due to catching of polymorphic type by value	3 years ago
Javier Jaramago Fernández	a661d33391	Fix invalid digest value in 'eventslog' for 'COM_STMT_EXECUTE' 'COM_STMT_EXECUTE' invalidly reported the 'digest' value from the previous query when preceded by a 'COM_QUERY'.	3 years ago
Javier Jaramago Fernández	41ef582fc5	Add new field 'stmt_id' to eventslog for 'STMT_PREPARE\|STMT_EXECUTE' events	3 years ago
Javier Jaramago Fernández	f993402c62	Fix crash when failed 'connection_reset' is tried to be logged	4 years ago
René Cannaò	aceeaf79ed	Disabled prometheus metrics dump , see #3571 This commit also fixes some minor memory corruptions	5 years ago
Rene Cannao	41b173a83c	Fixing a lot of compiler warnings	6 years ago
René Cannaò	e062f84c3e	Merge pull request #2399 from val214/v2.0.9-mem-leaks [WIP] V2.0.9 mem leaks fixes	6 years ago
peter	6bb057fbda	Fix add a mutex lock twice in the process of function call(log_audit_entry() -> write() -> write_auth()) Bug description: Since fix write event log and audit log coredump(#2427)， i think i left out a code that was not processed, resulting in coredump again in the program of proxysql , when i confige these parameters of auditlog_filename and auditlog_filesize in proxysql.cnf file or i set these parameters in proxysql-admin command line. the coredump contest: (gdb) bt #0 0x00002b224e9981f7 in raise () from /lib64/libc.so.6 #1 0x00002b224e9998e8 in abort () from /lib64/libc.so.6 #2 0x00002b224e991266 in __assert_fail_base () from /lib64/libc.so.6 #3 0x00002b224e991312 in __assert_fail () from /lib64/libc.so.6 #4 0x00000000005b2dc3 in main (argc=5, argv=0x7ffea714fd48) at main.cpp:1664 the proxysql.log contest: 2019-12-06 11:03:07 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:10 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:13 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:16 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:19 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:22 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:25 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:28 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:31 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:34 main.cpp:1659:main(): [ERROR] Watchdog: 8 threads missed a heartbeat 2019-12-06 11:03:34 main.cpp:1663:main(): [ERROR] Watchdog: reached 10 missed heartbeats. Aborting! proxysql: main.cpp:1664: int main(int, const char**): Assertion `0' failed. Error: signal 6: proxysql(_Z13crash_handleri+0x25)[0x5b607c] /lib64/libc.so.6(+0x35270)[0x2b224e998270] /lib64/libc.so.6(gsignal+0x37)[0x2b224e9981f7] /lib64/libc.so.6(abort+0x148)[0x2b224e9998e8] /lib64/libc.so.6(+0x2e266)[0x2b224e991266] /lib64/libc.so.6(+0x2e312)[0x2b224e991312] proxysql(main+0xe3a)[0x5b2dc3] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b224e984c05] proxysql[0x5adaa5] 2019-12-06 11:03:34 main.cpp:1396:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2019-12-06 11:03:34 main.cpp:1397:ProxySQL_daemonize_phase3(): [INFO] ProxySQL version _DEBUG Impact of this fix： I analyzed the call relationship of write_auth function of MySQL_Logger.cpp file in the whole project of proxysql with the source insight ide. I found write_auth function that was only called by the write function of MySQL_Logger.cpp file in the whole project of proxysql, and this write function was only called by log_audit_entry function or log_request function of MySQL_Logger.cpp file in the whole project of proxysql. However, i've added a mutex lock separately in log_audit_entry function and log_request function. And so i think it should delete the code for the wrlock function call in write_auth function, and it doesn't have impact for this time fix. Bux fix: Fix add a mutex lock twice in the process of function call(log_audit_entry() -> write() -> write_auth())	6 years ago
bskllzh	d71d01cb63	Fix write event log and audit log coredump(the events.logfile&audit.logfile null pointer) Bug description: When i make a benchmark test with a sysbench(version: 1.0.17) in proxysql 2.0.8 + mysql 5.7.26, i get the follow error info: Error: signal 11: proxysql(_Z13crash_handleri+0x25)[0x5b607c] /lib64/libc.so.6(+0x35270)[0x2b673de34270] /lib64/libstdc++.so.6(_ZNSo6sentryC2ERSo+0xf)[0x2b673d67521f] /lib64/libstdc++.so.6(_ZNSo5writeEPKcl+0x31)[0x2b673d675621] proxysql(_ZN11MySQL_Event20write_query_format_1EPSt13basic_fstreamIcSt11char_traitsIcEE+0x27d)[0x6e2d81] proxysql(_ZN11MySQL_Event5writeEPSt13basic_fstreamIcSt11char_traitsIcEEP13MySQL_Session+0x79)[0x6e19e9] proxysql(_ZN12MySQL_Logger11log_requestEP13MySQL_SessionP17MySQL_Data_Stream+0x710)[0x6e5416] proxysql(_ZN13MySQL_Session8LogQueryEP17MySQL_Data_Stream+0x6d)[0x654055] proxysql(_ZN13MySQL_Session10RequestEndEP17MySQL_Data_Stream+0x35)[0x6540d7] proxysql(_ZN13MySQL_Session7handlerEv+0x4397)[0x6459ed] proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x830)[0x623180] proxysql(_ZN12MySQL_Thread3runEv+0x2cbe)[0x62208a] proxysql(_Z24mysql_worker_thread_funcPv+0xbb)[0x5afb05] /lib64/libpthread.so.0(+0x7e25)[0x2b673cda8e25] /lib64/libc.so.6(clone+0x6d)[0x2b673def734d] 2019-12-02 09:52:26 main.cpp:1396:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! I analyze the coredump with proxysql 2.0.8 debug version very carefully, and i found the variable of events.logfile is null pointer in log_request function when it call write function(me.write(events.logfile, sess)). The following is the coredump（write (this=0x2b67402fd1f0, f=0x0, sess=0x2b67414a8c00)）: (gdb) bt #0 0x00002b673d67521f in std::ostream::sentry::sentry(std::ostream&) () from /lib64/libstdc++.so.6 #1 0x00002b673d675621 in std::ostream::write(char const, long) () from /lib64/libstdc++.so.6 #2 0x00000000006e2d81 in MySQL_Event::write_query_format_1 (this=0x2b67402fd1f0, f=0x0) at MySQL_Logger.cpp:265 #3 0x00000000006e19e9 in MySQL_Event::write (this=0x2b67402fd1f0, f=0x0, sess=0x2b67414a8c00) at MySQL_Logger.cpp:89 #4 0x00000000006e5416 in MySQL_Logger::log_request (this=0x2b673e419680, sess=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Logger.cpp:717 #5 0x0000000000654055 in MySQL_Session::LogQuery (this=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Session.cpp:6691 #6 0x00000000006540d7 in MySQL_Session::RequestEnd (this=0x2b67414a8c00, myds=0x2b6741346480) at MySQL_Session.cpp:6711 #7 0x00000000006459ed in MySQL_Session::handler (this=0x2b67414a8c00) at MySQL_Session.cpp:4086 #8 0x0000000000623180 in MySQL_Thread::process_all_sessions (this=0x2b6741301000) at MySQL_Thread.cpp:4392 #9 0x000000000062208a in MySQL_Thread::run (this=0x2b6741301000) at MySQL_Thread.cpp:4125 #10 0x00000000005afb05 in mysql_worker_thread_func (arg=0x2b673e461f30) at main.cpp:648 #11 0x00002b673cda8e25 in start_thread () from /lib64/libpthread.so.0 #12 0x00002b673def734d in clone () from /lib64/libc.so.6 And then i add a line of code in log_request function of MySQL_Logger.cpp file to check up whether events.logfile is null pointer ，as follow: void MySQL_Logger::log_request(MySQL_Session sess, MySQL_Data_Stream myds) { if (events.enabled==false) return; if (events.logfile==NULL) return; proxy_info("log_request: First: events.logfile is null %d\n", (events.logfile == NULL) ? 1 : 0); MySQL_Connection_userinfo ui=sess->client_myds->myconn->userinfo; And i make a benchmark test again with proxysql 2.0.8 debug version, the coredump of proxysql happen again, then i get the follow print info from proxysql.log [root@XXXXX proxysql]# grep -n -w 'log_request: First: events.logfile is null 1' proxysql.log 7358447:2019-12-02 09:52:26 MySQL_Logger.cpp:617:log_request(): [INFO] log_request: First: events.logfile is null 1 7358448:2019-12-02 09:52:26 MySQL_Logger.cpp:617:log_request(): [INFO] log_request: First: events.logfile is null 1 Bux fix: Fix write event log and audit log coredump, because the proxysql get the null pointer variable of events.logfile（audit.logfile）in a multithreaded environment.	6 years ago
Valentin Rakush	6a6873b342	Fix leaks in logger and admin	6 years ago
René Cannaò	ce4a4a2464	Code cleanup and speeding compile time	6 years ago
René Cannaò	45e701b54d	Code cleanup and speeding compile time	6 years ago
René Cannaò	80e92b144d	[WIP] Second commit for mysql firewall whitelist Improved also compiling time	6 years ago
René Cannaò	3ad899c758	Avoid crash in JSON library for invalid UTF8	6 years ago
René Cannaò	3b3def9f78	Fix unitialized file	7 years ago
René Cannaò	255d9247b8	Reduce locking on MySQL_Logger writes to disk	7 years ago
René Cannaò	0831740e45	Several enhancements on Query logger Enhancements: - added metrics rows_affected and rows_sent - added global variable mysql-eventslog_default_log : if 1 , logging is enabled for every query unless explicitly disabled in mysql_query_rules.log . Default is 0 - added global variable mysql-eventslog_format : default is 1 (legacy format). A value of 2 enables logging in JSON format. Issue #871 Changing value at runtime causes the current file to be closed and a new one created - fixed logging for prepared statements: till 2.0.5 only some percentage of prepared statements was correctly logged Extended tables stats_mysql_query_digest and stats_mysql_query_digest_reset to also include sum_rows_affected and sum_rows_sent Extended `eventslog_reader_sample.cpp` to support the new enhancements	7 years ago
René Cannaò	37d79ad23a	Fixed compiler warning: ISO C++ forbids converting a string constant to char pointer	7 years ago
René Cannaò	7ea33a1312	Completing initial implementation of Audit module	7 years ago
René Cannaò	9e2c1fdda1	Draft implementation of audit log	7 years ago
Nick Vyzas	28a9e3f984	Fixes - "Event log file is overwritten on ProxySQL service restart" #1201	8 years ago
René Cannaò	5e05348b22	Remove spinlock from MySQL Logger #977	9 years ago
René Cannaò	a6c53d0d12	Revert "Refactor header includes."	9 years ago
Maxim Bublis	30dbcc21a8	Refactor header includes. * reduce build time by 25%; * remove circular dependency on `cpp.h` and other headers. * fixes issue with `my_global.h` from mariadbclient which redefines `__attribute__`, that leads to broken STL and GCC intrinsics if `<algorithm>` wasn't included prior including this header; * fixes another issue with redefined `__attribute__` which leads to `_conn_exchange_t` not being aligned to cache line size; * removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize` to be `pthread_dummy(0)` which led to stack size being never adjusted; * add missing include guards to some headers; * remove unused proxysql_hash.h header.	9 years ago
Maxim Bublis	043a321e55	Cleanup legacy codebase.	9 years ago
René Cannaò	fb12719ba7	Quick return from MySQL_Logger::flush_log() #644	10 years ago
René Cannaò	8e81495bee	Improved logging Added logging information for: - client - server - hostgroup_id - digest - duration	10 years ago
René Cannaò	31ac6625f6	Logging improvements, related to #561 - added new function realtime_time() to get real time - added support for multiple events, default PROXYSQL_QUERY - MySQL_Event::write() will perfom a different action depending from log_event_type - rewrote part of eventslog_reader_sample.cpp: - it had few bugs - MySQL_Event::read() will perform a different action depending from log_event_type - time are printed in real time and not monotonic time - all info about a query are printed in one line	10 years ago
René Cannaò	cfd0cf32e8	QueryParserArgs is moved into Query_Info This is a further attempt to reduce calls to malloc() QueryParserArgs is no longer a pointer, but a struct inside Query_info	10 years ago
René Cannaò	ba7993ee69	Removed any reference to protobuf	10 years ago
René Cannaò	7830689dd9	New event logger Implemented new event logger that will replace protobuf It uses the same encoding of mysql protocol Added also a reader in ./tools/	10 years ago
René Cannaò	a63b7cf951	MySQL_Logger configuration - added global variable eventslog_filename - added global variable eventslog_filesize	10 years ago
René Cannaò	7a4a2092a5	MySQL_Logger implementation - Re-enabled MySQL_Logger - handling special case in which mysql-commands_stats and mysql-query_digests are disabled - specifies the length of the query - Query_Info is always initialized	10 years ago
René Cannaò	b420501d85	Temporary disable MySQL_Logger #434	10 years ago
René Cannaò	5ad97666f2	Improvement on MySQL_Logger added flush_log_unlocked() hardcoded max size for testing	10 years ago
René Cannaò	337c22aeab	PROXYSQL FLUSH LOGS now flushes also mysql logs (#434 )	10 years ago
René Cannaò	535030d546	MySQL_Logger starts logging using protobuf	10 years ago
René Cannaò	dbb9c111bb	Extending and testing MySQL_Logger()	10 years ago
René Cannaò	125cfa2c3b	First commit with MySQL_Logger	10 years ago

46 Commits (71cd99f40899bf607d6440ebc6b72954ccbb0388)