169 Commits (2ddc8186b775eaf8dab374bfcac7cd5a32f9fe3f)

Author	SHA1	Message	Date
Kir Shatrov	0922949b8c	Support min_gtid comment	6 years ago
René Cannaò	b8ecdaa708	Removing some comments	6 years ago
René Cannaò	e9dc7a9b60	Implementation of SQLI fingerprints whitelist ... libsqlinjection generates a lot of false positives. This commit introduces a new table: mysql_firewall_whitelist_sqli_fingerprints This table can list fingerprints generated by libsqlinjection: if the fingerprint is listed in this table, proxysql will consider it as a false positive. This commit also enables SQLi algorithm only if the query is not already explicitly whitelisted.	6 years ago
René Cannaò	485d313514	Further development of firewall whitelist. ... * introduced 2 new variables: - mysql-firewall_whitelist_enabled : defines if the firewall with whitelist algorithm is enabled or not - firewall_whitelist_errormsg : error message returned to the client if an error message is not already set * Query_Processor returns firewall_whitelist_mode to the calling MySQL_Session, but the logging on error log currently happens within the Query_Processor itself * if mysql-firewall_whitelist_enabled is turned on and a user it is not found in mysql_firewall_whitelist_users , the default behavior is that query is rejected * lookup on mysql_firewall_whitelist_users happens for username@ip (specific IP) and username@'' (all IPs) . Match on network subnet is currently not implemented * implemented SAVE MYSQL FIREWALL FROM RUNTIME * fixed implementation of SAVE MYSQL FIREWALL TO DISK * runtime configuration can be viewed in runtime_mysql_firewall_whitelist_rules and runtime_mysql_firewall_whitelist_users	6 years ago
René Cannaò	94f27716b7	Preparing SAVE MYSQL FIREWALL FROM RUNTIME	6 years ago
René Cannaò	6ee2a0cb19	Load mysql firewall whitelist rules to runtime	6 years ago
René Cannaò	ce4a4a2464	Code cleanup and speeding compile time	6 years ago
René Cannaò	80e92b144d	[WIP] Second commit for mysql firewall whitelist ... Improved also compiling time	6 years ago
René Cannaò	6bed334db1	stats_history.history_mysql_query_digest #2368 ... Implementation of new table stats_history.history_mysql_query_digest #2368 Implemented command `SAVE MYSQL DIGEST TO DISK`. Save to disk with really minimal memory footprint compared to querying `stats_mysql_query_digest` or `stats_mysql_query_digest_reset` . Implemented new admin variable `admin-stats_mysql_query_digest_to_disk` to automatically save to disk.	6 years ago
René Cannaò	f829ebcba0	Allows max_lag_ms to be 0	6 years ago
René Cannaò	a0ea0c118c	V2.0.8 fast routing (#2309) ... * Implementation of unit tests related to mysql_query_rules_fast_routing * Implementation of rules_fast_routing with khash * Adding memory usage for new fast routing algo	6 years ago
René Cannaò	5054998f7f	hostgroup sign in stats_mysql_query_digest #2285	7 years ago
René Cannaò	2fec51c0a9	A lot of changes to make query digest map faster ... Extended SQLite3_result to support a mutex for concurrent rows insert. Extensively rewritten get_query_digests() and get_query_digests_reset() to support multiple threads, and concurrently generate a resultset. Function get_query_digests_reset() defer the deletion of objects from query digest map after generating the resultset for Admin. This drastically reduce the amount of time that query digest map is locked. Rewritten get_query_digests_total_size() to make it multi-threaded. Added 3 new functions for purging: - purge_query_digests() : wrapper - purge_query_digests_sync() : synchronous purge of query digest map, either single-threaded or multi-threaded - purge_query_digests_async() : asynchronous purge of query digest map, single-threaded only If table `stats_mysql_query_digest_reset` is queried but not `stats_mysql_query_digest`, only the first one is populated. If table `stats_mysql_query_digest_reset` is queried and also `stats_mysql_query_digest`, the second one is populated as a copy from the first. Implemented new SQL `TRUNCATE [TABLE]` for `[stats.]stats_mysql_query_digest[_reset]`: this will purge the query digest map as fast as possible without creating any table in SQLite. For testing/benchmark: - modified ProxySQL_Test___GenerateRandomQueryInDigestTable() to generate more random data and randomly switch `mysql_thread___query_digests_normalize_digest_text` Extended command `PROXYSQLTEST` with the following tests: - 1 : generates entries in digest map - 2 : gets all the entries from the digest map, but without writing to DB, with multiple threads - 3 : gets all the entries from the digest map and reset, but without writing to DB, with multiple threads - 4 : purges the digest map, synchronously, in single thread - 5 : purges the digest map, synchronously, in multiple threads - 6 : purges the digest map, asynchronously, in single thread (this seems the faster) Replaced some sprintf() with new function my_itoa() . Modified class QP_query_digest_stats() to include username, schemaname and client_address if no longer than 24 bytes. This makes storage requirement for QP_query_digest_stats() bigger, but reduces a lot of calls to memory management. Modified QP_query_digest_stats::get_row() to drastically reduce the number of malloc/free and string copy. Overall, this commit has the following effects: - it increases the memory usage for query digest map. On test workload, this results in 10% more memory usage - drastically reduces the calls to memory management - populating `stats_mysql_query_digest_reset` is 3 times faster - the amount of time in which query digest map is locked is reduced by 95%	7 years ago
René Cannaò	46e11afab5	Add min_epoch_ms as a modifier for max_lag_ms ... Instead of specifying max_lag_ms, it is possible to specify min_epoch_ms	7 years ago
René Cannaò	ce757db70d	Merge pull request #2156 from sysown/v2.0.6-hg_locked ... V2.0.6 hg locked	7 years ago
René Cannaò	f2f3935fcd	drafting locked hostgroup	7 years ago
René Cannaò	0831740e45	Several enhancements on Query logger ... Enhancements: - added metrics rows_affected and rows_sent - added global variable mysql-eventslog_default_log : if 1 , logging is enabled for every query unless explicitly disabled in mysql_query_rules.log . Default is 0 - added global variable mysql-eventslog_format : default is 1 (legacy format). A value of 2 enables logging in JSON format. Issue #871 Changing value at runtime causes the current file to be closed and a new one created - fixed logging for prepared statements: till 2.0.5 only some percentage of prepared statements was correctly logged Extended tables stats_mysql_query_digest and stats_mysql_query_digest_reset to also include sum_rows_affected and sum_rows_sent Extended `eventslog_reader_sample.cpp` to support the new enhancements	7 years ago
René Cannaò	01023d6856	Merge branch 'v2.0.4' into v2.0.1-aurora	7 years ago
René Cannaò	af0a7100e2	Adding support for max_lag_ms in query	7 years ago
Carson Ip	1232a6382e	Fix minor typos	7 years ago
René Cannaò	9b14103236	Collect memory statistics about query rules	7 years ago
René Cannaò	4d3e45fc1b	Fixed bug in get_current_query_rules	7 years ago
René Cannaò	0a60cfda11	Merge branch 'v2.0.0-tb' into v2.0.0-merge-tb	7 years ago
René Cannaò	6f8c37eec9	Store digest_text on a different hash table ... This should drastically reduce memory usage where identical queries are executed an several schemas or by different users.	7 years ago
René Cannaò	c5cff7a921	Added client_address in stats_mysql_query_digest	7 years ago
René Cannaò	72724b22e0	Preparing for support of cache_timeout ... Adding column mysql_query_rules.cache_timeout . Functionality is not implemented yet, only the column is added.	7 years ago
René Cannaò	f666d40db2	Do not cache empty resultset, rule setting #1696	8 years ago
René Cannaò	b63df5aff2	Fix bug introduce in 6eeeb5c	8 years ago
René Cannaò	c15ee49797	Add support for wildcard in mysql_query_rules.client_addr #1450 ... Input validation: - client_addr not longer than INET6_ADDRSTRLEN - % allowed only at the end of client_addr Query rule itself remembers if there is a wildcard or not. If there is a wildcard compares string till the wildcard. If client_addr=='%' , it is a match all.	8 years ago
René Cannaò	13a57285b1	Trigger new routing algorithm only if hash table is not empty	8 years ago
René Cannaò	ce7d32421f	Reduce the size of delimiter	8 years ago
René Cannaò	97c81b4ef6	Initialize qr in Query_Processor	8 years ago
René Cannaò	b5d8c4d3ba	Added table mysql_query_rules_fast_routing ... This table is responsible for performing routing for very specific use case scenarios, like having tens of thousands of schemas	8 years ago
René Cannaò	46d7f9a14b	Add support for GTID consistent reads ... This patch includes a new column mysql_query_rules.gtid_from_hostgroup If defined, it is used for GTID consistent read	8 years ago
everpcpc	2ac60b0dbc	log before free	8 years ago
René Cannaò	78fce2b0d4	Improvement on stats_mysql_query_digest #1100 ... * record also unknown queries * remove parenthesis from the beginning of the query to understand the type of query	9 years ago
René Cannaò	fbd2f15efe	Memory Optimizations ... Try to allocate buffers in tokenizer() Optimized MySQL_ResultSet::buffer_to_PSarrayOut()	9 years ago
René Cannaò	a8846cc543	Incorrect number of columns in Query_Processor::get_current_query_rules() ... Crashing bug affecting only 1.4	9 years ago
René Cannaò	708f14a6e6	More accurate memory metrics	9 years ago
René Cannaò	1414d4430f	Introduced new table stats_memory_metrics ... Here an example: ``` Admin> SELECT * FROM stats_memory_metrics ORDER BY variable_name; +----------------------+----------------+ | Variable_Name | Variable_Value | +----------------------+----------------+ | Auth_memory | 836 | | SQLite3_memory_bytes | 1636648 | | jemalloc_active | 3469312 | | jemalloc_allocated | 3055248 | | jemalloc_mapped | 14680064 | | jemalloc_metadata | 1480576 | | jemalloc_resident | 4542464 | | jemalloc_retained | 0 | | query_digest_memory | 459 | +----------------------+----------------+ 9 rows in set (0.00 sec) ```	9 years ago
René Cannaò	4188d2009c	mysql_query_rules.OK_msg #1045 ... Extending mysql_query_rules with a new field: OK_msg If OK_msg is not NULL, an OK packet is sent to the client with an optional message as specified in OK_msg itself. If OK_msg is an not NULL empty string, an OK packet is sent to the client without any message. If both error_msg and OK_msg are present, error_msg is returned. This commit includes also few minor bugs, mostly related to typo that would prevent online upgrade of mysql_query_rules from early release of 1.4.0 .	9 years ago
René Cannaò	281e9cd619	Remove spinlock from Query Processor #977	9 years ago
René Cannaò	fcfa5682bc	Fix bug #970 ... Crash on mysql-query_digests=false, mysql-commands_stats=true, and short query	9 years ago
René Cannaò	a096164dd2	Various memory optimization	9 years ago
René Cannaò	790cc78acd	Return next_query_flagIN in qpo ... This is a minor improvement on pull request #825	9 years ago
René Cannaò	f35e853ecb	Merge pull request #825 from klinometr/1.4.0-sticky-flag ... Added sticky flag realization	9 years ago
René Cannaò	45a5acd947	New variables to reduce memory footprint #766 #729 ... The 2 new variables are: * mysql-query_digests_max_digest_length : defines the maximum length of digest_text as reported in stats_mysql_query_digest * mysql-query_digests_max_query_length : defines the maximum query length processed when computing query's digest and digext_text	9 years ago
René Cannaò	a6c53d0d12	Revert "Refactor header includes."	9 years ago
Maxim binopt	5ac7c3cc33	Merge branch 'v1.4.0' into 1.4.0-sticky-flag ... # Conflicts: # include/query_processor.h # lib/Query_Processor.cpp	9 years ago
Maxim Bublis	30dbcc21a8	Refactor header includes. ... * reduce build time by 25%; * remove circular dependency on `cpp.h` and other headers. * fixes issue with `my_global.h` from mariadbclient which redefines `__attribute__`, that leads to broken STL and GCC intrinsics if `<algorithm>` wasn't included prior including this header; * fixes another issue with redefined `__attribute__` which leads to `_conn_exchange_t` not being aligned to cache line size; * removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize` to be `pthread_dummy(0)` which led to stack size being never adjusted; * add missing include guards to some headers; * remove unused proxysql_hash.h header.	9 years ago
Maxim Bublis	043a321e55	Cleanup legacy codebase.	9 years ago
René Cannaò	21d5854e88	Removed a lot of valgrind warnings	9 years ago
Maxim binopt	d4d17584f3	Change sticky_flag realization to next_query_flagIN realization.	9 years ago
Maxim binopt	408c279ef2	Added sticky flag realization	9 years ago
René Cannaò	0b78ade053	Allow rules on proxy_port without proxy_addr #712	9 years ago
René Cannaò	7f8f0fc0df	Implementation of mysql_query_rules.multiplex #594	9 years ago
René Cannaò	c27a8099e1	Added runtime support for new RE modifiers ... Added support for runtime_mysql_query_rules and SAVE MYSQL QUERY RULES FROM RUNTIME	10 years ago
René Cannaò	453d52201b	2 bugs in re_modifiers argument2 bugs in re_modifiers argument	10 years ago
René Cannaò	331603aa91	Various bug fixes releated to RE	10 years ago
René Cannaò	311db6a681	Improving support for RE ... Added re_modifiers in mysql_query_rules	10 years ago
René Cannaò	dbfc0f5395	Merge branch 'v1.3.0' into pcre2	10 years ago
René Cannaò	faa3752e4c	Queries were not logged #723 ... mysql_query_rules.log was ignored	10 years ago
René Cannaò	fdf22675b6	Adding support for PCRE ... Added pcre-8.39 Introduced new variable mysql-query_processor_regex: - 1 : PCRE - 2 : RE2 (default)	10 years ago
René Cannaò	b05318acc7	Bug fix #719 ... runtime_mysql_query_rules.digest is now reported as NULL instead of 0x0000000000000000	10 years ago
René Cannaò	9b892ccd1d	Fix bug #717 related to digest ... mysql_query_rules.digest was not checked, and therefore unable to filter	10 years ago
René Cannaò	af07b5e249	Merge branch 'v1.2.4' into 1.3.0-alpha	10 years ago
René Cannaò	3380f6df48	Adding parsing for several SQL commands ... ALTER VIEW CALL CHANGE MASTER CREATE DATABASE CREATE INDEX CREATE SCHEMA CREATE TABLE CREATE TEMPORARY CREATE TRIGGER CREATE USER CREATE VIEW DESCRIBE DROP TABLE DROP TRIGGER DROP USER DROP VIEW FLUSH GRANT KILL LOCK TABLE LOAD OPTIMIZE RENAME TABLE RESET MASTER RESET SLAVE REVOKE SAVEPOINT TRUNCATE TABLE UNLOCK	10 years ago
René Cannaò	21fdb12e0b	Support for PREPARE (text protocol) #684 ... Added queries EXECUTE and DEALLOCATE PREPARE in Query Processor PREPARE automatically disables multiplexing	10 years ago
René Cannaò	5b5d924dc1	Merge branch 'v1.2.3' into 1.3.0-alpha-merge123 ... Conflicts: lib/MySQL_Session.cpp lib/ProxySQL_Admin.cpp src/Makefile	10 years ago
René Cannaò	e0c7ca209a	Implemented PROCESSLIST with Prepared Statements	10 years ago
René Cannaò	b9d0a9861b	Revert of last commit	10 years ago
René Cannaò	a53b8c2631	Loop only if flagOUT != 0	10 years ago
René Cannaò	d9b93b3937	Added new var mysql-query_processor_iterations ... When query rules are processed, even if flagOUT is set rules are processed in numerical order. This new variable allows to jump back (reset rule_id to 0) N amount of times. Using this variable is possible to implement loops. Futhermore, a loop is possible with only one rule setting flagIN=flagOUT . The default is 0. The maximum value is 1000000 .	10 years ago
René Cannaò	068456c11e	Added REPLACE in __query_parser_command_type	10 years ago
René Cannaò	07a0e6bab6	Adding comment field per issue #643 ... Added: * mysql_query_rules.comment VARCHAR * mysql_replication_hostgroups.comment VARCHAR * mysql_servers.comment VARCHAR NOT NULL DEFAULT '' All these tables support online upgrade from previous version (without such fields) at startup. Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping	10 years ago
René Cannaò	a9395cd087	Adding comment field per issue #643 ... Added: * mysql_query_rules.comment VARCHAR * mysql_replication_hostgroups.comment VARCHAR * mysql_servers.comment VARCHAR NOT NULL DEFAULT '' All these tables support online upgrade from previous version (without such fields) at startup. Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping	10 years ago
René Cannaò	d51bed1718	Reduce frequent alloc of Query_Processor_Output ... A new Query_Processor_Output (qpo) is allocated for every query. Allocation is now done once per session, thus speeding up execution time under memory profiler	10 years ago
René Cannaò	d885d4a737	Implementation of max retries ... Added global variables mysql-query_retries_on_failure (default 1) Added new field mysql_query_rules.retries (default NULL)	10 years ago
René Cannaò	9318d1a815	More matching criteria ... Issue #537 mysql_query_rules now matches also on : - client_addr - proxy_addr - proxy_port - digest	10 years ago
René Cannaò	f02fabeec9	Match on client_addr	10 years ago
René Cannaò	65f6dbc22d	Extended mysql_query_rules with new fields ... Fields present but not supported yet - client_addr - proxy_addr - proxy_port - digest - log	10 years ago
René Cannaò	c827048466	Removed obsolete code from Query_Processor.cpp	10 years ago
René Cannaò	4194efb33c	Added commands in Query_Processor::__query_parser_command_type() ... Added support for: - START TRANSACTION - SET - SHOW - SHOW TABLE STATUS	10 years ago
René Cannaò	67cb0627c4	Minor optimizations in Query_Processor	10 years ago
René Cannaò	f7140179a9	Minor optimization in Query_Processor::update_query_processor_stats()	10 years ago
René Cannaò	639cb0d1e5	Optimizations ... - attempt to remove libinjection - wrote a custom strcasecmp	10 years ago
René Cannaò	bc8c5e175a	Added column mirror_flagOUT in mysql_query_rules ... Mirroring starts when either mirror_hostgroup or mirror_flagOUT is set	10 years ago
René Cannaò	2e276f06f4	Second commit to implement mirroring ... Expanded table `mysql_query_rules` to add `mirror_hostgroup` Expanded struct `_Query_Processor_rule_t` and class `Query_Processor_Output` to support `mirror_hostgroup` Added new variable `mirror_hostgroup` in `MySQL_Session` The following logic ensure that a mirror_hostgroup set when processing the query in the original session is preserved when reprocessing the query in the mirror session: * when a new mirror session is *created* `mirror_hostgroup` is set according to the result of Query_Processor * when a mirror session is *executed* , `mirror_hostgroup` is copied into `default_hostgroup` Current limitations: * queries larger than 15MB aren't mirrorred * upgrade from 1.1 is still not possible it will wipe table mysql_query_rules	10 years ago
René Cannaò	cfd0cf32e8	QueryParserArgs is moved into Query_Info ... This is a further attempt to reduce calls to malloc() QueryParserArgs is no longer a pointer, but a struct inside Query_info	10 years ago
René Cannaò	4b1155e6c5	Reduce calls to malloc() ... myhash in Query_Processor::query_parser_update_counters() is now in the stack	10 years ago
René Cannaò	995203b6b3	Allocated buffer for first_comment only if needed ... Minor performance improvement to reduce the number of calls to malloc()	10 years ago
René Cannaò	a285b2ce94	Moving query digest table to std::unordered_map ... This is an attempt to track a possible memory leak or fragmentation	10 years ago
René Cannaò	f015cb7176	Added support for ROLLBACK in libinjection	10 years ago
René Cannaò	e43c7899c4	Track hostgroup in stats_mysql_query_digest #446	10 years ago
René Cannaò	535030d546	MySQL_Logger starts logging using protobuf	10 years ago
René Cannaò	a884e2d000	Build fixes	11 years ago
René Cannaò	0449d8f38e	Fixing typos	11 years ago
René Cannaò	43656f781e	Query filtering #418 ... Minor fixes	11 years ago
René Cannaò	ea15fd4681	Less aggressive multiplexing , #394	11 years ago
René Cannaò	e551213bf7	Clean up in Query_Processor	11 years ago