190 Commits (cf8cbfd89af57656a4a3253c7d0832ce25728fad)

Author	SHA1	Message	Date
René Cannaò	d23a531e00	Minor sanity checks	4 years ago
René Cannaò	8d83bd3562	Merge branch 'v2.x' into v2.1.2-cluster	4 years ago
Javier Jaramago Fernández	0c7934aab8	Replaced 'mysql_query_digest_and_first_comment' with new implementation 'mysql_query_digest_and_first_comment_2'	4 years ago
René Cannaò	f7bb4d201b	Add get_current_query_rules_fast_routing_count() ... Query_Processor::get_current_query_rules_fast_routing_count() now allows to quickly execute this query is ran on Admin: SELECT COUNT(*) FROM runtime_mysql_query_rules_fast_routing	5 years ago
René Cannaò	194243503a	Adding LCOV_EXCL_START around assert(0) ... LCOV_EXCL_START and LCOV_EXCL_STOP around assert(0) will improve code coverage	5 years ago
Javier Jaramago Fernández	028623c06c	Added 'first_comment' to 'MySQL_STMT_Global_info' to allow processing it during 'STMT_EXECUTE' #3427	5 years ago
Javier Jaramago Fernández	65e8caf9b6	Fixed memory corruption due to invalid free of 'qi->stmt_info->query' #3427	5 years ago
Javier Jaramago Fernández	fd5c21156c	Fixed memory leak in 'Query_Processor::new_query_rule' due to field 'match_digest' never being freed	5 years ago
Javier Jaramago Fernández	622a763920	Added new column 'attributes' to 'mysql_query_rules' table	5 years ago
Javier Jaramago Fernández	66478b89ee	Added new stat 'new_req_conns_count' counting the num of 'create_new_connection' recieved	6 years ago
Javier Jaramago Fernández	4409044398	Added support for annotation 'create_new_connection' that prevents reusing backend connections	6 years ago
René Cannaò	00ce76e57e	Add support for SAVEPOINT #948	6 years ago
René Cannaò	7b021e027a	Do not cache routing metadata for COM_STMT_EXECUTE ... From 1.3.0 till 2.0.x , when COM_STMT_PREPARE is executed and Query Processor is executed, metadata are stored in MySQL_STMT_Global_info(). These metadata include routing information. Is a prepared statement is prepared on a specific hostgroup, it will always be executed on the same hostgroup. This causes negative consequences, like the one reported in issue #2691 . This commit: * remove hostgroup_id from MySQL_STMT_Global_info() and other metadata * Query Processor is called also during COM_STMT_EXECUTE * removes hostgroup from table stats_mysql_prepared_statements_info * Query Processor do not perform any query rewrite ruding COM_STMT_EXECUTE	6 years ago
René Cannaò	f9fdde3b99	Fast routing lookup without username	6 years ago
René Cannaò	9c2f0bf3ac	Adding tests to benchmark multiple lookups in mysql_query_rules_fast_routing	6 years ago
René Cannaò	d15f055b1b	Remove define PROXYSQL_QPRO_PTHREAD_MUTEX ... Removed also all old code that didn't use pthread mutex	6 years ago
René Cannaò	c4a0ea9d4d	Remove all code not related to FAST_ROUTING_NEW208 ... Removed also #define FAST_ROUTING_NEW208 This was new code introduced in 2.0.8 , and now is the default	6 years ago
val	a095ef00d3	use DIGEST_STATS_FAST_1 always	6 years ago
René Cannaò	b0af4d272d	Fix 2 bugs related to firewall ... * Query_Processor::find_firewall_whitelist_rule() wasn't correctly finding rules * SQL injection algorithm should only be triggered if the query isn't blocked already	6 years ago
René Cannaò	e40246f62a	Fixed a bug in firewall whitelist	6 years ago
Kir Shatrov	2f7bb1a1cf	Validate gtid input	6 years ago
Kir Shatrov	0922949b8c	Support min_gtid comment	6 years ago
René Cannaò	b8ecdaa708	Removing some comments	6 years ago
René Cannaò	e9dc7a9b60	Implementation of SQLI fingerprints whitelist ... libsqlinjection generates a lot of false positives. This commit introduces a new table: mysql_firewall_whitelist_sqli_fingerprints This table can list fingerprints generated by libsqlinjection: if the fingerprint is listed in this table, proxysql will consider it as a false positive. This commit also enables SQLi algorithm only if the query is not already explicitly whitelisted.	6 years ago
René Cannaò	485d313514	Further development of firewall whitelist. ... * introduced 2 new variables: - mysql-firewall_whitelist_enabled : defines if the firewall with whitelist algorithm is enabled or not - firewall_whitelist_errormsg : error message returned to the client if an error message is not already set * Query_Processor returns firewall_whitelist_mode to the calling MySQL_Session, but the logging on error log currently happens within the Query_Processor itself * if mysql-firewall_whitelist_enabled is turned on and a user it is not found in mysql_firewall_whitelist_users , the default behavior is that query is rejected * lookup on mysql_firewall_whitelist_users happens for username@ip (specific IP) and username@'' (all IPs) . Match on network subnet is currently not implemented * implemented SAVE MYSQL FIREWALL FROM RUNTIME * fixed implementation of SAVE MYSQL FIREWALL TO DISK * runtime configuration can be viewed in runtime_mysql_firewall_whitelist_rules and runtime_mysql_firewall_whitelist_users	6 years ago
René Cannaò	94f27716b7	Preparing SAVE MYSQL FIREWALL FROM RUNTIME	6 years ago
René Cannaò	6ee2a0cb19	Load mysql firewall whitelist rules to runtime	6 years ago
René Cannaò	ce4a4a2464	Code cleanup and speeding compile time	6 years ago
René Cannaò	80e92b144d	[WIP] Second commit for mysql firewall whitelist ... Improved also compiling time	6 years ago
René Cannaò	6bed334db1	stats_history.history_mysql_query_digest #2368 ... Implementation of new table stats_history.history_mysql_query_digest #2368 Implemented command `SAVE MYSQL DIGEST TO DISK`. Save to disk with really minimal memory footprint compared to querying `stats_mysql_query_digest` or `stats_mysql_query_digest_reset` . Implemented new admin variable `admin-stats_mysql_query_digest_to_disk` to automatically save to disk.	6 years ago
René Cannaò	f829ebcba0	Allows max_lag_ms to be 0	6 years ago
René Cannaò	a0ea0c118c	V2.0.8 fast routing (#2309) ... * Implementation of unit tests related to mysql_query_rules_fast_routing * Implementation of rules_fast_routing with khash * Adding memory usage for new fast routing algo	6 years ago
René Cannaò	5054998f7f	hostgroup sign in stats_mysql_query_digest #2285	6 years ago
René Cannaò	2fec51c0a9	A lot of changes to make query digest map faster ... Extended SQLite3_result to support a mutex for concurrent rows insert. Extensively rewritten get_query_digests() and get_query_digests_reset() to support multiple threads, and concurrently generate a resultset. Function get_query_digests_reset() defer the deletion of objects from query digest map after generating the resultset for Admin. This drastically reduce the amount of time that query digest map is locked. Rewritten get_query_digests_total_size() to make it multi-threaded. Added 3 new functions for purging: - purge_query_digests() : wrapper - purge_query_digests_sync() : synchronous purge of query digest map, either single-threaded or multi-threaded - purge_query_digests_async() : asynchronous purge of query digest map, single-threaded only If table `stats_mysql_query_digest_reset` is queried but not `stats_mysql_query_digest`, only the first one is populated. If table `stats_mysql_query_digest_reset` is queried and also `stats_mysql_query_digest`, the second one is populated as a copy from the first. Implemented new SQL `TRUNCATE [TABLE]` for `[stats.]stats_mysql_query_digest[_reset]`: this will purge the query digest map as fast as possible without creating any table in SQLite. For testing/benchmark: - modified ProxySQL_Test___GenerateRandomQueryInDigestTable() to generate more random data and randomly switch `mysql_thread___query_digests_normalize_digest_text` Extended command `PROXYSQLTEST` with the following tests: - 1 : generates entries in digest map - 2 : gets all the entries from the digest map, but without writing to DB, with multiple threads - 3 : gets all the entries from the digest map and reset, but without writing to DB, with multiple threads - 4 : purges the digest map, synchronously, in single thread - 5 : purges the digest map, synchronously, in multiple threads - 6 : purges the digest map, asynchronously, in single thread (this seems the faster) Replaced some sprintf() with new function my_itoa() . Modified class QP_query_digest_stats() to include username, schemaname and client_address if no longer than 24 bytes. This makes storage requirement for QP_query_digest_stats() bigger, but reduces a lot of calls to memory management. Modified QP_query_digest_stats::get_row() to drastically reduce the number of malloc/free and string copy. Overall, this commit has the following effects: - it increases the memory usage for query digest map. On test workload, this results in 10% more memory usage - drastically reduces the calls to memory management - populating `stats_mysql_query_digest_reset` is 3 times faster - the amount of time in which query digest map is locked is reduced by 95%	7 years ago
René Cannaò	46e11afab5	Add min_epoch_ms as a modifier for max_lag_ms ... Instead of specifying max_lag_ms, it is possible to specify min_epoch_ms	7 years ago
René Cannaò	ce757db70d	Merge pull request #2156 from sysown/v2.0.6-hg_locked ... V2.0.6 hg locked	7 years ago
René Cannaò	f2f3935fcd	drafting locked hostgroup	7 years ago
René Cannaò	0831740e45	Several enhancements on Query logger ... Enhancements: - added metrics rows_affected and rows_sent - added global variable mysql-eventslog_default_log : if 1 , logging is enabled for every query unless explicitly disabled in mysql_query_rules.log . Default is 0 - added global variable mysql-eventslog_format : default is 1 (legacy format). A value of 2 enables logging in JSON format. Issue #871 Changing value at runtime causes the current file to be closed and a new one created - fixed logging for prepared statements: till 2.0.5 only some percentage of prepared statements was correctly logged Extended tables stats_mysql_query_digest and stats_mysql_query_digest_reset to also include sum_rows_affected and sum_rows_sent Extended `eventslog_reader_sample.cpp` to support the new enhancements	7 years ago
René Cannaò	01023d6856	Merge branch 'v2.0.4' into v2.0.1-aurora	7 years ago
René Cannaò	af0a7100e2	Adding support for max_lag_ms in query	7 years ago
Carson Ip	1232a6382e	Fix minor typos	7 years ago
René Cannaò	9b14103236	Collect memory statistics about query rules	7 years ago
René Cannaò	4d3e45fc1b	Fixed bug in get_current_query_rules	7 years ago
René Cannaò	0a60cfda11	Merge branch 'v2.0.0-tb' into v2.0.0-merge-tb	7 years ago
René Cannaò	6f8c37eec9	Store digest_text on a different hash table ... This should drastically reduce memory usage where identical queries are executed an several schemas or by different users.	7 years ago
René Cannaò	c5cff7a921	Added client_address in stats_mysql_query_digest	7 years ago
René Cannaò	72724b22e0	Preparing for support of cache_timeout ... Adding column mysql_query_rules.cache_timeout . Functionality is not implemented yet, only the column is added.	7 years ago
René Cannaò	f666d40db2	Do not cache empty resultset, rule setting #1696	8 years ago
René Cannaò	b63df5aff2	Fix bug introduce in 6eeeb5c	8 years ago
René Cannaò	c15ee49797	Add support for wildcard in mysql_query_rules.client_addr #1450 ... Input validation: - client_addr not longer than INET6_ADDRSTRLEN - % allowed only at the end of client_addr Query rule itself remembers if there is a wildcard or not. If there is a wildcard compares string till the wildcard. If client_addr=='%' , it is a match all.	8 years ago
René Cannaò	13a57285b1	Trigger new routing algorithm only if hash table is not empty	8 years ago
René Cannaò	ce7d32421f	Reduce the size of delimiter	8 years ago
René Cannaò	97c81b4ef6	Initialize qr in Query_Processor	8 years ago
René Cannaò	b5d8c4d3ba	Added table mysql_query_rules_fast_routing ... This table is responsible for performing routing for very specific use case scenarios, like having tens of thousands of schemas	8 years ago
René Cannaò	46d7f9a14b	Add support for GTID consistent reads ... This patch includes a new column mysql_query_rules.gtid_from_hostgroup If defined, it is used for GTID consistent read	8 years ago
everpcpc	2ac60b0dbc	log before free	8 years ago
René Cannaò	78fce2b0d4	Improvement on stats_mysql_query_digest #1100 ... * record also unknown queries * remove parenthesis from the beginning of the query to understand the type of query	8 years ago
René Cannaò	fbd2f15efe	Memory Optimizations ... Try to allocate buffers in tokenizer() Optimized MySQL_ResultSet::buffer_to_PSarrayOut()	9 years ago
René Cannaò	a8846cc543	Incorrect number of columns in Query_Processor::get_current_query_rules() ... Crashing bug affecting only 1.4	9 years ago
René Cannaò	708f14a6e6	More accurate memory metrics	9 years ago
René Cannaò	1414d4430f	Introduced new table stats_memory_metrics ... Here an example: ``` Admin> SELECT * FROM stats_memory_metrics ORDER BY variable_name; +----------------------+----------------+ | Variable_Name | Variable_Value | +----------------------+----------------+ | Auth_memory | 836 | | SQLite3_memory_bytes | 1636648 | | jemalloc_active | 3469312 | | jemalloc_allocated | 3055248 | | jemalloc_mapped | 14680064 | | jemalloc_metadata | 1480576 | | jemalloc_resident | 4542464 | | jemalloc_retained | 0 | | query_digest_memory | 459 | +----------------------+----------------+ 9 rows in set (0.00 sec) ```	9 years ago
René Cannaò	4188d2009c	mysql_query_rules.OK_msg #1045 ... Extending mysql_query_rules with a new field: OK_msg If OK_msg is not NULL, an OK packet is sent to the client with an optional message as specified in OK_msg itself. If OK_msg is an not NULL empty string, an OK packet is sent to the client without any message. If both error_msg and OK_msg are present, error_msg is returned. This commit includes also few minor bugs, mostly related to typo that would prevent online upgrade of mysql_query_rules from early release of 1.4.0 .	9 years ago
René Cannaò	281e9cd619	Remove spinlock from Query Processor #977	9 years ago
René Cannaò	fcfa5682bc	Fix bug #970 ... Crash on mysql-query_digests=false, mysql-commands_stats=true, and short query	9 years ago
René Cannaò	a096164dd2	Various memory optimization	9 years ago
René Cannaò	790cc78acd	Return next_query_flagIN in qpo ... This is a minor improvement on pull request #825	9 years ago
René Cannaò	f35e853ecb	Merge pull request #825 from klinometr/1.4.0-sticky-flag ... Added sticky flag realization	9 years ago
René Cannaò	45a5acd947	New variables to reduce memory footprint #766 #729 ... The 2 new variables are: * mysql-query_digests_max_digest_length : defines the maximum length of digest_text as reported in stats_mysql_query_digest * mysql-query_digests_max_query_length : defines the maximum query length processed when computing query's digest and digext_text	9 years ago
René Cannaò	a6c53d0d12	Revert "Refactor header includes."	9 years ago
Maxim binopt	5ac7c3cc33	Merge branch 'v1.4.0' into 1.4.0-sticky-flag ... # Conflicts: # include/query_processor.h # lib/Query_Processor.cpp	9 years ago
Maxim Bublis	30dbcc21a8	Refactor header includes. ... * reduce build time by 25%; * remove circular dependency on `cpp.h` and other headers. * fixes issue with `my_global.h` from mariadbclient which redefines `__attribute__`, that leads to broken STL and GCC intrinsics if `<algorithm>` wasn't included prior including this header; * fixes another issue with redefined `__attribute__` which leads to `_conn_exchange_t` not being aligned to cache line size; * removes `pthread_setstacksize` calls as `my_global.h` redefines `pthread_setstacksize` to be `pthread_dummy(0)` which led to stack size being never adjusted; * add missing include guards to some headers; * remove unused proxysql_hash.h header.	9 years ago
Maxim Bublis	043a321e55	Cleanup legacy codebase.	9 years ago
René Cannaò	21d5854e88	Removed a lot of valgrind warnings	9 years ago
Maxim binopt	d4d17584f3	Change sticky_flag realization to next_query_flagIN realization.	9 years ago
Maxim binopt	408c279ef2	Added sticky flag realization	9 years ago
René Cannaò	0b78ade053	Allow rules on proxy_port without proxy_addr #712	9 years ago
René Cannaò	7f8f0fc0df	Implementation of mysql_query_rules.multiplex #594	9 years ago
René Cannaò	c27a8099e1	Added runtime support for new RE modifiers ... Added support for runtime_mysql_query_rules and SAVE MYSQL QUERY RULES FROM RUNTIME	9 years ago
René Cannaò	453d52201b	2 bugs in re_modifiers argument2 bugs in re_modifiers argument	9 years ago
René Cannaò	331603aa91	Various bug fixes releated to RE	9 years ago
René Cannaò	311db6a681	Improving support for RE ... Added re_modifiers in mysql_query_rules	9 years ago
René Cannaò	dbfc0f5395	Merge branch 'v1.3.0' into pcre2	9 years ago
René Cannaò	faa3752e4c	Queries were not logged #723 ... mysql_query_rules.log was ignored	9 years ago
René Cannaò	fdf22675b6	Adding support for PCRE ... Added pcre-8.39 Introduced new variable mysql-query_processor_regex: - 1 : PCRE - 2 : RE2 (default)	9 years ago
René Cannaò	b05318acc7	Bug fix #719 ... runtime_mysql_query_rules.digest is now reported as NULL instead of 0x0000000000000000	10 years ago
René Cannaò	9b892ccd1d	Fix bug #717 related to digest ... mysql_query_rules.digest was not checked, and therefore unable to filter	10 years ago
René Cannaò	af07b5e249	Merge branch 'v1.2.4' into 1.3.0-alpha	10 years ago
René Cannaò	3380f6df48	Adding parsing for several SQL commands ... ALTER VIEW CALL CHANGE MASTER CREATE DATABASE CREATE INDEX CREATE SCHEMA CREATE TABLE CREATE TEMPORARY CREATE TRIGGER CREATE USER CREATE VIEW DESCRIBE DROP TABLE DROP TRIGGER DROP USER DROP VIEW FLUSH GRANT KILL LOCK TABLE LOAD OPTIMIZE RENAME TABLE RESET MASTER RESET SLAVE REVOKE SAVEPOINT TRUNCATE TABLE UNLOCK	10 years ago
René Cannaò	21fdb12e0b	Support for PREPARE (text protocol) #684 ... Added queries EXECUTE and DEALLOCATE PREPARE in Query Processor PREPARE automatically disables multiplexing	10 years ago
René Cannaò	5b5d924dc1	Merge branch 'v1.2.3' into 1.3.0-alpha-merge123 ... Conflicts: lib/MySQL_Session.cpp lib/ProxySQL_Admin.cpp src/Makefile	10 years ago
René Cannaò	e0c7ca209a	Implemented PROCESSLIST with Prepared Statements	10 years ago
René Cannaò	b9d0a9861b	Revert of last commit	10 years ago
René Cannaò	a53b8c2631	Loop only if flagOUT != 0	10 years ago
René Cannaò	d9b93b3937	Added new var mysql-query_processor_iterations ... When query rules are processed, even if flagOUT is set rules are processed in numerical order. This new variable allows to jump back (reset rule_id to 0) N amount of times. Using this variable is possible to implement loops. Futhermore, a loop is possible with only one rule setting flagIN=flagOUT . The default is 0. The maximum value is 1000000 .	10 years ago
René Cannaò	068456c11e	Added REPLACE in __query_parser_command_type	10 years ago
René Cannaò	07a0e6bab6	Adding comment field per issue #643 ... Added: * mysql_query_rules.comment VARCHAR * mysql_replication_hostgroups.comment VARCHAR * mysql_servers.comment VARCHAR NOT NULL DEFAULT '' All these tables support online upgrade from previous version (without such fields) at startup. Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping	10 years ago
René Cannaò	a9395cd087	Adding comment field per issue #643 ... Added: * mysql_query_rules.comment VARCHAR * mysql_replication_hostgroups.comment VARCHAR * mysql_servers.comment VARCHAR NOT NULL DEFAULT '' All these tables support online upgrade from previous version (without such fields) at startup. Introduced also a parser for escaping strings with single quotes: double quotes are allowed without extra escaping	10 years ago
René Cannaò	d51bed1718	Reduce frequent alloc of Query_Processor_Output ... A new Query_Processor_Output (qpo) is allocated for every query. Allocation is now done once per session, thus speeding up execution time under memory profiler	10 years ago
René Cannaò	d885d4a737	Implementation of max retries ... Added global variables mysql-query_retries_on_failure (default 1) Added new field mysql_query_rules.retries (default NULL)	10 years ago
René Cannaò	9318d1a815	More matching criteria ... Issue #537 mysql_query_rules now matches also on : - client_addr - proxy_addr - proxy_port - digest	10 years ago