83 Commits (5e68075edfea82ab4d3740b5db02e9edd30cf768)

Author	SHA1	Message	Date
René Cannaò	5e68075edf	Cluster: Speed up processing of MySQL Query Rules ... Multiple changes: - Query_Processor stored the resultset of query rules loaded to runtime (previously this was only for query rules fast routing) - Admin returns the stored resultset (query rules and fast routing) when queries by Cluster - In khash replaced the hashing function from the built-in __ac_X31_hash_string to CityHash32 - When Cluster is used, it calls load_mysql_query_rules_to_runtime() passing the resultsets retrieved by the remote peer - Increased SQLite cache_size to ~50MB: this seems to be a very small optimization and probably it will be reverted - pull_mysql_query_rules_from_peer() uses transactions to write to mysql_query_rules_fast_routing - (important change) pull_mysql_query_rules_from_peer() verifies the checksum of MySQL Query ules before loading them to runtime	4 years ago
René Cannaò	43c6acb218	Merge branch 'v2.x' into v2.1.2-cluster	4 years ago
André Cabaça	977bf0f1da	Removed dead code ... As reported in #3740 there was duplicated code. Originated from this merge #739	4 years ago
René Cannaò	f7bb4d201b	Add get_current_query_rules_fast_routing_count() ... Query_Processor::get_current_query_rules_fast_routing_count() now allows to quickly execute this query is ran on Admin: SELECT COUNT(*) FROM runtime_mysql_query_rules_fast_routing	5 years ago
Javier Jaramago Fernández	622a763920	Added new column 'attributes' to 'mysql_query_rules' table	5 years ago
Javier Jaramago Fernández	66478b89ee	Added new stat 'new_req_conns_count' counting the num of 'create_new_connection' recieved	6 years ago
Javier Jaramago Fernández	4409044398	Added support for annotation 'create_new_connection' that prevents reusing backend connections	6 years ago
René Cannaò	9c2f0bf3ac	Adding tests to benchmark multiple lookups in mysql_query_rules_fast_routing	6 years ago
René Cannaò	d15f055b1b	Remove define PROXYSQL_QPRO_PTHREAD_MUTEX ... Removed also all old code that didn't use pthread mutex	6 years ago
René Cannaò	c4a0ea9d4d	Remove all code not related to FAST_ROUTING_NEW208 ... Removed also #define FAST_ROUTING_NEW208 This was new code introduced in 2.0.8 , and now is the default	6 years ago
val	a095ef00d3	use DIGEST_STATS_FAST_1 always	6 years ago
René Cannaò	c4607d94fe	class Web_Interface_plugin friends Query_Processor	6 years ago
Kir Shatrov	3b031f3518	Prevent crash on index() returning NULL ... in handler___client_DSS_QUERY_SENT___server_DSS_NOT_INITIALIZED__get_connection	6 years ago
Kir Shatrov	2f7bb1a1cf	Validate gtid input	6 years ago
Kir Shatrov	0922949b8c	Support min_gtid comment	6 years ago
René Cannaò	e9dc7a9b60	Implementation of SQLI fingerprints whitelist ... libsqlinjection generates a lot of false positives. This commit introduces a new table: mysql_firewall_whitelist_sqli_fingerprints This table can list fingerprints generated by libsqlinjection: if the fingerprint is listed in this table, proxysql will consider it as a false positive. This commit also enables SQLi algorithm only if the query is not already explicitly whitelisted.	6 years ago
René Cannaò	485d313514	Further development of firewall whitelist. ... * introduced 2 new variables: - mysql-firewall_whitelist_enabled : defines if the firewall with whitelist algorithm is enabled or not - firewall_whitelist_errormsg : error message returned to the client if an error message is not already set * Query_Processor returns firewall_whitelist_mode to the calling MySQL_Session, but the logging on error log currently happens within the Query_Processor itself * if mysql-firewall_whitelist_enabled is turned on and a user it is not found in mysql_firewall_whitelist_users , the default behavior is that query is rejected * lookup on mysql_firewall_whitelist_users happens for username@ip (specific IP) and username@'' (all IPs) . Match on network subnet is currently not implemented * implemented SAVE MYSQL FIREWALL FROM RUNTIME * fixed implementation of SAVE MYSQL FIREWALL TO DISK * runtime configuration can be viewed in runtime_mysql_firewall_whitelist_rules and runtime_mysql_firewall_whitelist_users	6 years ago
René Cannaò	94f27716b7	Preparing SAVE MYSQL FIREWALL FROM RUNTIME	6 years ago
René Cannaò	6ee2a0cb19	Load mysql firewall whitelist rules to runtime	6 years ago
René Cannaò	4005746b1a	More on firewall whitelist	6 years ago
René Cannaò	ce4a4a2464	Code cleanup and speeding compile time	6 years ago
René Cannaò	80e92b144d	[WIP] Second commit for mysql firewall whitelist ... Improved also compiling time	6 years ago
René Cannaò	6bed334db1	stats_history.history_mysql_query_digest #2368 ... Implementation of new table stats_history.history_mysql_query_digest #2368 Implemented command `SAVE MYSQL DIGEST TO DISK`. Save to disk with really minimal memory footprint compared to querying `stats_mysql_query_digest` or `stats_mysql_query_digest_reset` . Implemented new admin variable `admin-stats_mysql_query_digest_to_disk` to automatically save to disk.	6 years ago
René Cannaò	a0ea0c118c	V2.0.8 fast routing (#2309) ... * Implementation of unit tests related to mysql_query_rules_fast_routing * Implementation of rules_fast_routing with khash * Adding memory usage for new fast routing algo	6 years ago
René Cannaò	2fec51c0a9	A lot of changes to make query digest map faster ... Extended SQLite3_result to support a mutex for concurrent rows insert. Extensively rewritten get_query_digests() and get_query_digests_reset() to support multiple threads, and concurrently generate a resultset. Function get_query_digests_reset() defer the deletion of objects from query digest map after generating the resultset for Admin. This drastically reduce the amount of time that query digest map is locked. Rewritten get_query_digests_total_size() to make it multi-threaded. Added 3 new functions for purging: - purge_query_digests() : wrapper - purge_query_digests_sync() : synchronous purge of query digest map, either single-threaded or multi-threaded - purge_query_digests_async() : asynchronous purge of query digest map, single-threaded only If table `stats_mysql_query_digest_reset` is queried but not `stats_mysql_query_digest`, only the first one is populated. If table `stats_mysql_query_digest_reset` is queried and also `stats_mysql_query_digest`, the second one is populated as a copy from the first. Implemented new SQL `TRUNCATE [TABLE]` for `[stats.]stats_mysql_query_digest[_reset]`: this will purge the query digest map as fast as possible without creating any table in SQLite. For testing/benchmark: - modified ProxySQL_Test___GenerateRandomQueryInDigestTable() to generate more random data and randomly switch `mysql_thread___query_digests_normalize_digest_text` Extended command `PROXYSQLTEST` with the following tests: - 1 : generates entries in digest map - 2 : gets all the entries from the digest map, but without writing to DB, with multiple threads - 3 : gets all the entries from the digest map and reset, but without writing to DB, with multiple threads - 4 : purges the digest map, synchronously, in single thread - 5 : purges the digest map, synchronously, in multiple threads - 6 : purges the digest map, asynchronously, in single thread (this seems the faster) Replaced some sprintf() with new function my_itoa() . Modified class QP_query_digest_stats() to include username, schemaname and client_address if no longer than 24 bytes. This makes storage requirement for QP_query_digest_stats() bigger, but reduces a lot of calls to memory management. Modified QP_query_digest_stats::get_row() to drastically reduce the number of malloc/free and string copy. Overall, this commit has the following effects: - it increases the memory usage for query digest map. On test workload, this results in 10% more memory usage - drastically reduces the calls to memory management - populating `stats_mysql_query_digest_reset` is 3 times faster - the amount of time in which query digest map is locked is reduced by 95%	7 years ago
René Cannaò	6aac86bebc	Changed the way max_lag_ms in query is computed ... It now remembers if the query is waiting for long time	7 years ago
René Cannaò	af0a7100e2	Adding support for max_lag_ms in query	7 years ago
René Cannaò	9b14103236	Collect memory statistics about query rules	7 years ago
René Cannaò	0a60cfda11	Merge branch 'v2.0.0-tb' into v2.0.0-merge-tb	7 years ago
René Cannaò	6f8c37eec9	Store digest_text on a different hash table ... This should drastically reduce memory usage where identical queries are executed an several schemas or by different users.	7 years ago
René Cannaò	c5cff7a921	Added client_address in stats_mysql_query_digest	7 years ago
René Cannaò	72724b22e0	Preparing for support of cache_timeout ... Adding column mysql_query_rules.cache_timeout . Functionality is not implemented yet, only the column is added.	7 years ago
René Cannaò	f666d40db2	Do not cache empty resultset, rule setting #1696	8 years ago
René Cannaò	c15ee49797	Add support for wildcard in mysql_query_rules.client_addr #1450 ... Input validation: - client_addr not longer than INET6_ADDRSTRLEN - % allowed only at the end of client_addr Query rule itself remembers if there is a wildcard or not. If there is a wildcard compares string till the wildcard. If client_addr=='%' , it is a match all.	8 years ago
René Cannaò	b5d8c4d3ba	Added table mysql_query_rules_fast_routing ... This table is responsible for performing routing for very specific use case scenarios, like having tens of thousands of schemas	8 years ago
Nikolaos Vyzas	9f3c265fd9	Resolving conflicts for "Add support for GTID consistent reads"	8 years ago
Nick Vyzas	bfcf57e9d0	Updated 'hits' to uint64 to prevent overflow	8 years ago
René Cannaò	1414d4430f	Introduced new table stats_memory_metrics ... Here an example: ``` Admin> SELECT * FROM stats_memory_metrics ORDER BY variable_name; +----------------------+----------------+ | Variable_Name | Variable_Value | +----------------------+----------------+ | Auth_memory | 836 | | SQLite3_memory_bytes | 1636648 | | jemalloc_active | 3469312 | | jemalloc_allocated | 3055248 | | jemalloc_mapped | 14680064 | | jemalloc_metadata | 1480576 | | jemalloc_resident | 4542464 | | jemalloc_retained | 0 | | query_digest_memory | 459 | +----------------------+----------------+ 9 rows in set (0.00 sec) ```	9 years ago
René Cannaò	4188d2009c	mysql_query_rules.OK_msg #1045 ... Extending mysql_query_rules with a new field: OK_msg If OK_msg is not NULL, an OK packet is sent to the client with an optional message as specified in OK_msg itself. If OK_msg is an not NULL empty string, an OK packet is sent to the client without any message. If both error_msg and OK_msg are present, error_msg is returned. This commit includes also few minor bugs, mostly related to typo that would prevent online upgrade of mysql_query_rules from early release of 1.4.0 .	9 years ago
René Cannaò	281e9cd619	Remove spinlock from Query Processor #977	9 years ago
René Cannaò	790cc78acd	Return next_query_flagIN in qpo ... This is a minor improvement on pull request #825	9 years ago
Maxim binopt	5ac7c3cc33	Merge branch 'v1.4.0' into 1.4.0-sticky-flag ... # Conflicts: # include/query_processor.h # lib/Query_Processor.cpp	9 years ago
Maxim Bublis	043a321e55	Cleanup legacy codebase.	9 years ago
Maxim binopt	d4d17584f3	Change sticky_flag realization to next_query_flagIN realization.	9 years ago
Maxim binopt	408c279ef2	Added sticky flag realization	9 years ago
René Cannaò	7f8f0fc0df	Implementation of mysql_query_rules.multiplex #594	9 years ago
René Cannaò	311db6a681	Improving support for RE ... Added re_modifiers in mysql_query_rules	9 years ago
René Cannaò	af07b5e249	Merge branch 'v1.2.4' into 1.3.0-alpha	10 years ago
René Cannaò	3380f6df48	Adding parsing for several SQL commands ... ALTER VIEW CALL CHANGE MASTER CREATE DATABASE CREATE INDEX CREATE SCHEMA CREATE TABLE CREATE TEMPORARY CREATE TRIGGER CREATE USER CREATE VIEW DESCRIBE DROP TABLE DROP TRIGGER DROP USER DROP VIEW FLUSH GRANT KILL LOCK TABLE LOAD OPTIMIZE RENAME TABLE RESET MASTER RESET SLAVE REVOKE SAVEPOINT TRUNCATE TABLE UNLOCK	10 years ago
René Cannaò	21fdb12e0b	Support for PREPARE (text protocol) #684 ... Added queries EXECUTE and DEALLOCATE PREPARE in Query Processor PREPARE automatically disables multiplexing	10 years ago